Welcome to the Cannabis Data Science Meetup group.
Fun group of cannabis enthusiasts, data scientists, anyone who's interested in wrangling data
into the cannabis space.
So see a bunch of friendly faces and then Prince introduced herself.
So good to see you Grant, Hector, Tammy, Candice.
Just to give everybody a chance to speak if you want.
Before Candice and I spill the beans about some of the nifty things that we've been cooking
up.
Does anyone else want to speak?
I'll just go around and just see if anyone has any topics that you're curious in tackling
in the coming months because we're actually ready to employ our talents onto some new
projects.
So I'll maybe hint at some of those because as all we'll demonstrate today, one of these
tools that we're cooking up is coming along nicely.
So before we get to this Hector, anything on your plate or agenda?
No, I'm still looking for work and I'm doing some projects on the side with data analysis
just to keep my skills fresh.
I'm excited to see what you have for us today.
Too cool.
And one of the highest in demand things is so we've done natural language processing
and one thing that we may attempt to do, I don't think, yeah we haven't done it yet,
is natural language generation.
So that takes it one step further so that's almost given a prompt, create text, and there's
actually a nifty application for this.
So technically that's data science.
So yeah, so we can actually maybe move on to this before too, too long.
So that's another teaser.
So but Tammy, it's going to be fine.
So Tammy, anything on your plate that you're interested in working on?
Well it's been a while since I've attended but since then I have graduated from my data
analytics boot camp and am actively looking for a job.
So I would like to, there's my camera, now I'm back in my office, I'd like to work on
some things and maybe improve my portfolio.
Too cool Tammy.
Remind me if you don't mind what state you may be in just so I kind of have a, just knowledge,
you don't have to share, I guess that's kind of personal information, but if you want to
do that way I can kind of just understand the cannabis regulations in your space.
Well I am in the mountains of Virginia.
Virginia.
Yep, yep, it's not as big of a mountains as what's behind Hector.
So this is, it'll be interesting to see how Virginia pans out.
So just to be frank, there were, it seems like there were maybe some people that saw
a lot of the ball coming before others.
Right.
Right, you know, you're in the right channels, you may have just kind of heard about this.
So you know, some people have been able to kind of get a good running at this, in particular
hemp companies and there's no problem there.
In fact, I've always been going on about, oh, hey, companies, you know, you may want
to think about positioning yourself to enter, right.
So it worked out well for the hemp growers in New York, big time.
Well I know there was North Carolina is where I'm from.
And the farmers all around, like right beside my in-laws house, they were growing hemp and
they stopped after one year because of, there were vehicles running through their yard and
through their fence to get to the field.
It's crazy.
Actually, that's an interesting complication that you raise is, you know, you've got to
have security dialed in and it's maybe that's something that maybe people don't take into
consideration with the outdoor, right.
Because if you've got your indoor facility, you can put locks on your doors, some would
have cameras, but you're right.
I don't know if you could just put a hemp field right beside a highway and you're right,
people may be romping through.
But anywho, like I said, it'll be interesting to see how things pan out.
I'm optimistic.
And then in Virginia, you're actually not that far from Maryland as well as the Northeast.
So it's kind of getting up there, but you know, New Jersey, there's a lot of action
going on there.
And, you know, with data analytics, you know, you can do a lot online.
So it wouldn't take.
So long story short, as we'll show you some opportunities today that primarily have to
do with lab data.
And you may want to think about reaching out to some of the laboratories in Maryland.
I know a laboratory in New Jersey, but you may want to get in contact with more there.
New York, I don't know who's testing in New York, but probably somebody is.
So I mean, so that's a start.
So that's what I started to do.
The laboratories have a lot on their plate, though.
So there's a lot of people that need help with laboratory data.
So in fact, the retailers, the retailers are probably going to be the easiest to get in
contact with, but they're probably going to have the, you know, the highest guards up,
right?
They don't.
Right.
But it doesn't hurt.
So my recommendation is send some of these find there if they have the contact information
on their website.
You may want to reach out to them that way because someone's checking the inbox, right?
And so long story short is we're going to be doing some certificate of analysis parsing
and it's all hands on deck and we're doing some in California.
I was going to mention, going to see what we can't do in Washington state and Prince,
Colorado, so we can be in touch there.
And then like I said, the whole Eastern seaboard, right?
You know, it's all hands on deck.
So there's probably people over there that need certificate of analysis parsing as well.
Right.
So anywho, Prince, I know you're listening in, so feel free to chime in at any point.
And then Grant, is there anything on your plate that you want answered?
Any data science topics on the brain?
Not particularly.
I've actually, I've just been struggling to actually find the time to join this meeting.
I joined sort of right before, you know, you had set up the new time and everything.
And yeah, I've just always had work meetings overlapping except for today.
So yeah, I'm just stoked to just learn more, honestly.
Happy to have you Grant and happy the stars aligned.
In fact, this is not really the lesson of the day, but it's something that's kind of
a recurring theme, stars aligning.
But anywho, candidates, anything on your mind?
I know we've been doing a lot of certificate parsing.
So anything jumping out at you that you'd like to share or thoughts?
Thank you, Keegan.
Let's see.
Yeah, we've been spending a lot of time web scraping URLs and parsing PDFs.
And it's been pretty cool.
We sure do have a lot of data.
And so what I have not completed is I want to do some plots on terpene ratios.
And but I haven't completed that yet.
I just started that today.
But we sure do have a lot of data.
So we'll we'll look at some terpene ratios today.
And one thing that's fun about them is it's as we were talking about with John is it's
sort of a good sort of like I like to call these sanity checks.
Right.
So you do so much work and it gets a little abstract that sometimes you just kind of need
to put things down on paper and do a sanity check.
And so it's like, you know, we've done so much data wrangling.
Like, it's just time to visualize it a little bit just as a sanity check, just see like,
okay, you know, are these percentages?
Are they between zero and 100?
What does the distribution look like?
So we'll do some of those things today.
It's just, you know, just important.
It's all like, you know, as a good data scientist, rule number one, you know, look at the data.
It sounds great.
Anywho, George, welcome to the group today.
We've got John, anything on your mind today, George, about data science with the merger
of the cannabis industry and anything you'd like to answer?
I have no idea.
I'm here to learn.
I'm a student, I'm a master's of data science student at University of Colorado Boulder.
Too cool.
Well, I'm actually here for an assignment.
I have to attend a data science meetup and do a writeup about it.
And this one seemed interesting, seemed to be one of the more interesting ones.
Well, I love it.
So is there anything to get done, you know, the chemiletics name out there?
And so, you know, the University of Colorado points you in our direction, then that's superb.
And in fact, I think you'll walk away with something out of it.
Hopefully lots of cool things going on.
So John, good to see you.
We're just about to kick off with some of the showcasing of the new tools that we've
built.
So before we really dive into that, anything on your mind, cannabis data science related
that you want to talk about?
Check the mic.
You may have your own.
We can't hear you, John.
We can hear you now.
Yeah, sorry.
It helps to unmute.
I'm real interested in what Boulder has in terms of data science curriculum and how you
would be steered to something like this, George.
That's very cool.
Right.
They have this new master's program specifically for data science.
I think it just it's new as of last year.
It's about a year old now.
And what credentials do you need to get into it?
I mean, what kind of undergraduate degree are they needing to get so that you can get
credential?
It's pretty interdisciplinary.
This program is actually they have performance based admissions.
So you take a few classes unenrolled.
But my bachelor's is in computer science.
I think bachelor's in any kind of math, applied math statistics would be that would be good
to that would work.
And I think there are a lot of interdisciplinary people from that are coming from fields like
biology.
Excellent.
Well, I'd be happy to learn more.
I would just say you've got a good background, George.
Computer science is one of the best backgrounds you could have because you'll need to touch
up.
You'll need to, of course, learn some statistics.
Right.
A lot of the statistics that I knew the hood, but I always emphasize it's important to,
you know, understand what's going on under the hood in case something breaks.
But for the most part, once you understand the statistics, then you can use it.
Then that's the easy part.
The hard part is getting your model or your statistical model, your algorithm into production.
And so that's wide open.
So that could be, you know, are you going to be running it in the cloud?
Are you going to have your own server?
I mean, you all you know all that jazz.
So that's sort of the fun part, right.
So since you already have all those skills in your tool belt, as you start learning,
you know, you may want to.
In fact, I would recommend while you're at the University of Colorado, you may want to
go get acquainted with some of the statisticians there.
Right.
There's probably some brilliant statisticians there that you could learn a lot from in just
30 minutes or more, you know, sit in on some of their lectures.
Because if you can basically stack these skills, so you got your computer science, you can
ship your model and you know the statistics, so you're not making rookie mistakes and you
actually know what you're doing, then you're just going to, you know, what someone in UC
Irvine told me, you're going to be selling like hotcakes, right.
Because you're actually going to be able to build and implement statistics that people
need.
So, too cool.
But enough of hammering that home.
But let's just go ahead and kick off today, right.
And show you, you know, what you can actually do, you know, with statistics.
And computer science too, right.
Today's actually going to be a computer science heavy day.
So just to go back, so I always like to look at the history of some of these technologies,
because you know, John likes to remind me and I always end up discovering a lot of these
technologies aren't new.
They're just better implementations of old technology.
And so, you know, if you go, you know, hit Wikipedia, and so I didn't do anything fancy,
so you know, you know, look into this too.
So it looks like there was a character, Emanuel Goldberg, who started, interestingly, it looks
like, parsing text from what I believe were early movie film.
So maybe there were titles or credits that needed to be parsed.
So this, what's the right word, innovative character came up with a way to programmatically,
or, you know, through a machine, get characters.
And I guess he was turning them into telegraph code.
So really, really early stages of what we'd call optical character recognition.
And this is where it's like, you know, it's such a small, funny, intertwined world, right.
Because this first device, right, the very first device to use optical character, optical
character recognition, used my component Selenium photo sensors.
And just, I got a laugh out of this because we're actually using a Python package, Selenium.
So we're also using Selenium in optical character recognition, just a much different type of
Selenium.
So I don't know if you get a laugh out of that, those are the types of jokes that Courtney
data scientists make me.
Don't put that in your write up, Hector.
It'll be a discouraging note.
And then this character, so I recognize this name, so Ray Kurzweil, a technologist, I
guess one would say.
Just interestingly enough, Ray Kurzweil started a company back in 1974 that essentially scanned
documents and I guess I assume digitized them by recognizing the printed font.
So once again, you know, here I'm thinking like, you know, I'm bragging to you that,
you know, oh, you know, we're using this cutting edge technology.
But really, this is technology that Ray Kurzweil has been using for almost 50 years now.
So you know, so we're just so I get inspiration out of this in that it makes me think that,
okay, you know, now we can just kind of use OCR just as sort of like a passive tool, almost
like one would just use like a screwdriver.
We're not quite there yet.
It was still kind of a little complicated to use.
But you know, it'd be nice to think that this is going to be one of the standard tools.
And you know, just to show you that, you know, you can make a buck from parsing documents,
right, to the LexisNexis, parsing legal documents.
And then once we hit the 2000s, there's tons of web-based applications.
But I don't think there's that much open source optical character recognition.
At least, I mean, I know the PIE, the Tesseract, which we're using was open sourced in 2018.
So there may have been some open source technology before then.
But so you'll have to look into that yourself.
But that may have been a hurdle, because maybe a lot of I know, like this first technology
was patented.
So it could have been some of this technology was patented for a while.
But that's where my research ends and my questions begin.
But here's sort of the cannabis data science application is okay.
In certain states, it's mandated.
In most states, they're there for the asking.
But if you go and buy a cannabis product, they're in almost every state that I know
of, quality assurance is mandated.
So the producer has to send a sample to the laboratory to get tested for cannabinoids
and contaminants.
This is interesting data, in my opinion, right?
So for example, I'm working with the cannabis diary.
It's a mobile app that consumers can use to track their consumption.
Right.
And so the idea is, you know, you can trend your cannabinoid consumption over time.
There's a lot more to the app than that.
But but this is, you know, in use for the data.
But you'll have to get the data in a manner.
And so the way you would do such is, I mean, there's lots of ways, right?
And this is why I was saying reach out to laboratories.
So for example, I reached out to all the laboratories in Washington state to see if I, you know,
anywhere interested in, you know, trying to make this data easily accessible to consumers
in a similar way that say SC labs or MCR labs in SC labs in California, MCR labs in Massachusetts
respect.
So I was just seeing, OK, are any of you interested in one one was.
And so there may be a further information there.
I'll kind of tease that and maybe talk about that more next week.
But that's just the start.
So for example, this is why I was saying you may want to think and think about reaching
out to the labs in Maryland, New Jersey and New York and just say, hey, you know, are
you interested in maybe sharing a template of your COA so that way we can make sure that
we can parse it.
But then what I'm interested in is, is, is how well does this work in the wild?
And so that's what we may start testing out here in the.
The near future.
Right.
Is now we've and we'll demo it today.
You know, we have the ability to extract the data from the actual PDF.
We can do it from an image of the PDF.
Can we do it from a wild image?
So if you know, because if somebody snaps a picture on their phone of a piece of paper,
how good is the resolution going to be with phones these days?
It's shockingly good.
How lined up is the paper going to be?
So you may have to do some fancy stuff there.
But, you know, where there is a will, there's a way.
So that's sort of, you know, on the road map is, you know, because how cool would that
be, right, if you could just take a picture of a certificate and then all of a sudden
all that data is now yours, which it should be.
Right.
Because these are tests mandated for your safety.
Right.
Or just your your knowledge.
So then you can, you know, take that data and trend it.
I mean, this is the thing that George and everyone else is, you know, once you've
learned, say, time series statistics, you just want to look at everything as a time
series.
So, so anywho, and then in particular.
So that's just a computer science exercise.
And then we actually need some statistics.
Right.
And so the idea is, OK, what do we actually do with this data?
So once we've gotten this data, which would basically be X, so we'll just have all the
data so you can think about X is just a vector of lab result data.
So once you have all of that lab result data, could you actually use that to predict products
that a consumer may enjoy?
And then before I open up to questions, I would just like to say to my background in
economics and so enjoy has a pretty specific meaning to me.
And so, you know, to me, this is basically like, OK, you know, the, you know, the consumers
trying to maximize their utility from this product.
So actually, the enjoy is a super loose word.
Right.
So that could mean, right.
Why are the, you know, we're basically I'm just assuming, OK, this is a utility maximizing
consumer.
So enjoy is actually kind of a little subjective, I guess.
So is that their, you know, well-being, you know, are they a medical consumer or are they
a recreational adult use consumer?
So there anywho.
But any thoughts, comments, questions about the questions at hand before open it up to
show you how we can solve this?
Well, before I bore you to death, let's just go ahead and talk about the algorithm and
then we can go ahead and read in the data.
So long story short is, you know, last week, we were basically just recommending products
off of the average concentration.
So we were just saying, OK, can we find products that are chemically similar to products that
a consumer's purchased in the past?
And there was a typo in the code last week.
I'll show you how we can fix that.
In the meetup, we didn't get to it, but the code's there.
But yes, you know, we can find the most chemically similar products.
But a question was raised, well, what should it be taken into consideration how much the
consumer actually likes a product?
And so that's where, does anybody remember sentiment analysis?
So long story short is, we used reviews in the past to basically rank the consumer's
sentiment of a product from zero to one.
So instead of just taking an average of the cannabinoid concentrations, you could basically
take a weighted average.
So if you think about the edge case, so the binary case, zero or one, zero, the consumer
doesn't like the product, one, the consumer likes the product.
Then you would just take the average of the products that the consumer likes.
So that's the easiest case.
And that was sort of what I just kind of conjectured last week was a technique that you could do.
And now we've formally put it into math.
And so this is just a weighted average.
George, you know, this is nothing fancy, but this is one of the reasons why it's good to
get intimate with statistics, because yes, you'll go way above and beyond, but you'll
also become intimately, probably too intimately familiar with weighted averages.
And just, you know, you can go deep with statistics.
So like I said, I would encourage you to pick up some of those skills because I think they're
useful.
But anywho, you know, that's the edge case, zero or one.
And then what you can do is just let theta, so this parameter, vary between zero and one.
And then you'd get your continuous moving average, where the closer it is to zero, the
less the consumer like the product with 0.5 being neutral.
And then everything, you know, greater than 0.5 to one, the consumer generally viewed
as positive.
Okay.
Well, that's the math behind everything.
And so now let's just, any questions before we dive in?
Is there survey data for the consumer sentiment or are you going to try to infer that?
I'm not quite clear.
We're going to use Leafly review data, so these are fictitious consumers.
So we're just saying, oh, given this, well, there is, I guess, a real person behind the
username, but to us, they're just usernames.
But yes, so we'll basically be making product recommendations for, I suppose, real users
on Leafly.
So I suppose if any Leafly users are watching this and we happen to use your namespace,
then these may be product recommendations for you.
So this is...
So are you, I'm sorry, are you accessing a Leafly scale where it's like or don't like,
or is it one of the response category scales or something?
There's a number of them in the Leafly scales.
Excellent question, John.
So essentially we're just working with the free form review.
Oh, the text.
Exactly.
So given the text, we're using natural language processing to estimate a score between zero
and one of how positive we think this consumer was with their review.
Okay.
But lots to showcase real quick.
So just going to start reading in a bunch of helpful packages here, and then we'll just
start walking through some of the code because we've got some cool things here today.
So really just reading in a bunch of helpful Python packages.
If you're interested, I'll post the code to GitHub.
So I'll recommend that you may want to read through it if you're interested in the code.
But otherwise I'm just going to start working through here so that way we can get to some
of the interesting work we're doing here.
Okay, cool.
So in this repository, we've got a PDF.
And so this is a certificate of analysis that you may receive from a laboratory or from,
say, if you're a consumer and you purchased Gaviota Mist at a retailer in California.
Perhaps you asked the retailer, hey, can I have their certificate?
And maybe they gave you this PDF.
So as I said, the next step is can we do this if somebody just took free form pictures of
each page?
That's sort of going to be the next hurdle.
But I'll show you something cool that we can do.
So first things first, we'll want to just create just some temporary...
Oops, I just must not have made it where we want it to be.
Hopefully we've got the images folder.
We must have made this...
It's okay, we've made this over in the consumer recs folder by accident.
Oh well.
So anyway, actually, let me just...
If you don't mind, let me just kind of restart this a little quick.
Let's be a bit more straightforward this way.
That way we're kind of saving things where we should be.
All right, so anywho, we're going to need to parse this PDF.
And so we've been doing that in the past, no big issue.
But let's just have a look at this PDF real quick.
So in particular, we're interested in this missed PDF.
So missed.
What we can do is we can read this in with PDF plumber.
So as I always say, someone call a plumber.
And so if we read in the document, then we can see the text of the document.
So we can just look at the first page, get all the text.
This is what we've been doing.
Here I'll just print out the first 30 characters.
Uh-oh, if we look at all the text, so this is all the text on the first page.
We just get a bunch of CIDs.
So that's my text.
And so I've kind of looked into this, and this is just the standard that PDF plumber
returns when it doesn't recognize a character.
So we're not recognizing any of the characters on the page.
So instead of, so you could do a font lookup at this point, but going down that rabbit
hole was unsuccessful for us.
So what we decided to do was to implement optical character recognition.
So at first it seemed like a daunting task, but actually I'll show you, it works out pretty
well.
So this is just a quick test just to see, oh, yes, we need optical character recognition.
Cool.
So how does one go about doing such?
Well, interestingly enough, you actually first, you just start by converting all of these
pages, this may throw an error actually, but we should start generating these images.
So basically the first step, and I've got the algorithm up here, is one, I think, may
not have created these directories.
Sorry for the, hold up here.
First thing first, turn them into indicates.
I've got a feeling I just forgot the code up top to create that directory.
And one thing is I'm currently cranking the resolution up pretty high.
And so what I found is if the resolution's any less than 300, here we are, then it's
no good.
So here I'm just going to copy one of these images because they're temporary.
So if you look at one of these, this is basically just an image of the first page of the PDF.
So this is where I realized that the consumer could potentially take these pictures if we
can get the orientation correct, right?
Because we're basically now working with an image, right?
So one could think if you lined your phone up perfectly, you could take such an image.
And I'm curious if anyone else, if you want to look at this, can we work with a wild image
of a COA?
So that's the first step is basically create all the images.
And then this is the fun part.
And so we're standing on the shoulders of giants.
So we're just using Tesseract to turn this image into a PDF.
So Tesseract is an open source OCR library that you can install.
Please get in touch with me if you need any help getting it installed.
And then once it's installed, then you're essentially off to the races.
So we brought all these images here.
So the six images of the six pages.
And what file format are those images at this point?
These are in PNG.
.png, okay.
So it was recommended.
So we did a couple things here if anyone's interested.
So this was a function I found on Stack Overflow.
But technically, we're basically removing the alpha channel.
And a computer scientist may need to help me with this.
Perhaps this has to do something with transparency or?
Yeah, the alpha channel is transparency.
That's right.
So as I said, I'm not 100% sure if this is necessary.
But this was sort of the recommended way to turn a PDF to a PNG.
So in fact, George or anyone else, that's sort of the benefit of having this code open
sourced is feel free to look through this function and see if this is appropriate.
Is a PNG in fact appropriate?
Or would a different file format be appropriate?
But I think what this is doing is if the PDF has a transparent background, I think we're
just giving it a white background so we can recognize the text better.
So once again, it may be overkill, but that's part of the algorithm.
So important.
It could be important.
And then essentially, just one by one, this is where we're actually using the optical
character recognition.
And like I said, this is what I was talking about, like OCR as a almost like a screwdriver.
So we're just an OCR company.
It's just a tool that we use.
So it's like when you're building a house, you may use a screwdriver.
So when you're building an app, you may use OCR.
So now you can actually recognize this text here.
So I'll prove that we can do that here in Python.
So now we've got missed zero.
So now let's see if we can't.
So this is what I call a sanity check.
So as you're programming, it's worth just saying, oh, can we just read and miss zero
before we go too much further?
Can we actually get the text on this page?
And drum roll.
We don't have a bunch of CIDs.
And so now the idea is now you can just go about parsing as usual.
And so it's not necessarily an easy feat.
So PDF Plumber has methods for extracting tables.
If any of you think of clever algorithms for how to parse the actual data, that's kind
of where I think the main value is added, because we're basically using tools to recognize
all the text.
And then basically, the value added part is writing a custom algorithm to essentially
be able to handle all the nuances of the formatting choices that people chose.
And like I said, the grand scheme of this is to create a generalized parsing algorithm.
So that way, the algorithm is simply smart enough to recognize that this 86.04% corresponds
to total can of annoyance.
Easier said than done.
But if you're clever, capitalize on this opportunity.
But this is sort of what we've been working on.
So if you're interested, we basically have a specific algorithm to parse the COA.
It's not glamorous.
It just uses the way the text is laid out.
But this is the important part, is just proving that you can in fact read the text.
So as I said before, it was just all CID.
That's useless to us.
And now we actually have useful data.
And as I said, it came from an image.
So that's just kind of cool.
And then just to kind of show you this in the wild, technically, what you can do now
is right now you have six individual PDFs.
Well, just merge them back together.
And then here I'm just removing the unused PDFs just because I just, right, George, this
is something that you'll pick up.
Right.
And so this is why it's useful to have the computer science background is you just kind
of have to have that foresight that, yeah, we can't have thousands of thousands of high
resolution PDFs just redundantly laying around.
That's just going to be a lot of storage.
So I'm just doing a little bit of data cleanup.
But what did we, we did save this, right?
We just saved it as the test PDF.
And so here is just the PDF.
And so now, you know, we should be able to say, OK, let's say this doc is now the test
PDF.
You now can just say, OK, you know, for, well, first you have to read it in.
So you open the doc and then you can just say, OK, you know, for page, you know, and
report dot pages, just extract the text from each page and handle that accordingly.
And so just bear yourself a bunch of texts.
So wall of text.
So you've heard of the wall of sound.
Well, now we give you the wall of text.
So anywho, now you have, you know, all the data from the COA.
And you know, it could be worse.
You know, to be honest, we did some patent scraping and legal documents are honestly
much worse right there.
The formatting is great.
Like every single legal document may be different here.
You know, at least every every certificate from canalysis labs is the same.
So just to go ahead to just to quit teasing you to there's all the data.
Well, we want that data.
And so, like I said, if you're interested in the algorithm, check it out.
But we should just be able to use one of these tools we built, COA doc, and we just say,
hey, you know, let's check this out.
Starting with.
Let's start with the original mist that we couldn't identify.
So this is the doc that we couldn't identify.
Confirm.
Now from role.
So it's remember this is going to take a hot second.
And I think we can hopefully watch it happen as it happens.
But, you know, essentially just packaged this same functionality, the convert to images,
convert to PDF.
See, there are the images being generated.
Then we don't need the images anymore.
So I'll remove those once the PDFs are created.
So here you see PDF 01.
And, you know, so we'll convert all the pages.
And then at that point, you know, COA doc has a bunch of built in algorithms.
I wonder.
Sorry, I built kind of over.
Kind of over blowing the memory here.
But well.
Like I said, this is the important part.
So this is this is the value that I think analytics added is now given this PDF, which,
you know, originally was just as good as an image.
We can now get all the data from it, in particular, all these results.
So for example, you've got your Delta 9 THC, 83.27.
So that should be in this table.
So 83.27 for Delta 9 THC.
So now you don't have to do any manual data entry.
So now, you know, see, you want to trend the milligrams of THC that you're consuming, which
is something that I'm interested in doing, because I heard a saying, you know, if you're
not measuring it, you're not managing it.
So I'm curious about this.
And then, as I said, you know, basically, the kid with the cannabis diary app is doing
is recording consumption in tandem with, say, your activities of the day.
And, you know, maybe your reviews as well.
So that way you can see, you know, or these compounds, you know, having a positive or
a negative effect on you.
Right.
So was it a bad day?
How many milligrams of THC did you consume?
Or was it a good day?
How many milligrams did you consume?
And like I said, you know, you may be over consuming, you may be under consuming, say,
if this is something you need.
This is 100% up to you.
But like I said, this is just something that I personally was interested in.
And that's George, you may have already figured this out.
But that's kind of how you start with the, that's how you can start with a good app.
Because if you just find a problem in your own life and say, hey, you know, how can I
fix this?
Or, you know, you can find a problem in someone else's life or a problem with someone else's
business and solve that, right?
That's what business models are all about.
But this is a method for like a consumer app.
So what's the problem in my life?
How can I solve it?
Can I build an app to solve that problem?
Would other people be interested in using that app?
So to long story short is, you know, now we can not only just get the THC and CBD, which
maybe, you know, you could enter that in by hand.
But now we get alpha-cedrine, alpha-pinene, alpha-terpinial.
You know, it's kind of miraculous, the amount of compounds that are in a cannabis plant.
And good folks like John at the CESC are currently researching what in the world makes some of
these compounds, like what effect may they have on you?
And different people, right?
Everyone's got different biochemistries.
And, you know, this is where I mean, my knowledge is already separate.
So it's like, you know, first things first, I'd be interested in trending THC and CBD.
But then what's going on with all of these analytes?
You know, maybe they have meaningful, meaningful effects.
And so we're essentially using these, you know, this is what I talked about, our X bar.
So all of these, this is basically our, we're basically going to create a vector of all
of these results.
So to kind of, so that's the OCR.
I mean, you know, I promised you a data science exercise.
So if you can hang out for 10 more minutes, then I'll go ahead and show you here how you
could actually use this data.
So I think if I just change this data directory, then everything should be okay.
All right.
So now we're just going to use a big data set of these lab results.
And so the idea is, okay, awesome.
You know, we now have a mechanism to parse all of this data.
You know, and like I said, you can look at contaminants too.
Because here, look at this.
So this is actually an important thing that I kind of want to take a tangent on, just
to kind of help, right, it's Canvas data science after all.
So just to kind of help people understand what's going on in the cannabis industry.
So this here is a concentrate.
So this is concentrated cannabis.
So what people do is they use some sort of solvent to basically concentrate down the
cannabinoids, and then typically these solvents aren't something that you want to be consuming.
They're typically purged, well, not typically, they are, you know, they try to purge as much
of this as possible, ideally 100% from the product.
But, you know, as we've kind of found out with chemistry, you know, dealing with 100%
are difficult.
And so the idea here is it looks like this is an ethanol concentrate, because basically
it looks like this laboratory has a detection limit on their residual solvents.
I'm not sure if it says anywhere right off the bat, but here it says.
So they can detect down to 10, please correct me on this if I'm wrong, but I think this
is, this may just be micrograms.
Yeah, that's micrograms.
Okay, so they can detect down to 10 micrograms.
So they can only quantify up to 50 micrograms.
So technically this product here has between 10 and 50 micrograms of ethanol in it.
Per gram, you always have to normalize it to an M-gram.
You may even have a, oh, it's a half gram concentrate.
No, but the unit is micrograms per gram.
I know, I was just wondering if I accidentally spoke correctly, even though I misspoke.
John's correct.
This is between 40 micrograms per gram.
And that it's a half a gram concentrate, then you would expect around, you would write,
your expectation would be 20 micrograms per gram of ethanol.
And like I said, that's up to the consumer to decide that may be a negligible amount
of ethanol.
Right, so for example, somebody once told me that, I mean, if you're using a butane
lighter, you're maybe consuming a non-negligible amount of butane.
That's a possibility.
That's actually some research that I would love to see somebody do with one of those
inhalation machines.
But those are at laboratories.
But here, let me stay on track here.
Anywho, this is kind of, this was one of the reasons I was thinking that it was important
to parse these is this one less than the LOQ, you know, this one, this one may be negligible.
But remember, you know, the action limit is 5,000.
So technically, you could have a concentrate pass at 4,999 micrograms per gram.
Probably a laboratory probably wouldn't pass that.
I mean, they may.
But right, I don't know, it'd be interesting to ask laboratories what their policy on
that would be.
But this is something that we looked at in Washington state is you do see high levels
that do get passed, right, like 4,200 or 4,800 would probably pass.
And as I said, that's up to the consumer to decide.
100% up to the consumer.
They may say, hey, consuming tons of butane, I don't really, I don't care about this small
amount.
Maybe they do care about the small amount.
You know, I, I, for one, I primarily just care about detections of pesticides.
So that's what I look at.
I just, I love to see the full clean panel.
Whether there's any logical sense to that, I'm not 100% certain yet.
It's just one of those things where, you know, even if I see a hit, I'm just a little skeptical
of, you know, are there hot spots in the flower?
And maybe we just didn't test one of the hot spots or I don't know.
But I guess every consumer has their own preferences.
Those are just my own preferences.
But the main point is, if you actually bought this concentrate at the store, they may have
the total cannabinoids and the total THC label.
But, you know, the chances of them having, you know, all 20 to 40 turkeys plus, you know,
they're, you know, the, the retailer's not going to put this, say, technically, this
is a detect, right?
So they wouldn't put a, this product with, be detected ethanol in it.
And, you know, if it's an ethanol concentrate, you know, like I said, it's chemistry, you
know, you may not ever really be able to purge 100%.
But anywho, I just kind of wanted to go down that rabbit hole, just because these are things
that I kind of take for granted because I spent a little time at the laboratory.
So I'm just kind of familiar with these abbreviations.
But if these abbreviations are new to you, then maybe you've gained some insight there.
But enough of that nonsense.
Let's actually look at the data, right?
That's what we, that's what we came here to do.
Here, let's just work with the data file we were working with last week because it's behaving
better.
So the idea is these are just a bunch of values that we've already parsed.
So if you look at, say, Raw Gardens website, you can see that they have, they actually
have hundreds of these canalysis COAs.
So that's why we originally wanted to parse these, is now we can parse hundreds of their
COAs.
And I think today we're just working with a sample of 100,000.
So you know, if you plot the, so we're just going to start doing some plots.
And I'll try to get through everything quickly.
So thank you all for staying so long.
So long story short is, you've got the data, let's look at it.
So we're just going to care about two terpenes today.
And as I said, you know, you could use 40.
We're just going to use two today just as a proof of concept.
And then we're just going to use a bunch of consumer reviews.
I just realized the weighting average part may have an error.
So I can at least walk you through the sentiment analysis.
And then if we run into a, because I think the problem on this weighted average here
is I would say we're working with a vector and not a one dimensional scalar.
If that's the correct terminology, it's been a hot minute.
In fact, I can maybe even try to maybe be able to change this to a scalar really, really
quick while the data loads.
So long story short is I'm going to be using the log of beta pinin to beta limenine as
my predicting factor because we've talked about it in the past.
So as I said, you can build your own prediction model.
And that's why I was saying the fun part once you get the statistics down is actually just
thinking of good variables to use and wrangling them.
So the reviews, we have maybe 42,000 reviews.
So that takes a hot minute to always kind of push my computer to its limit with the
streaming at all.
So just checking to make sure everybody's still here.
See if we can't free up some bandwidth.
So while the reviews are loading, I'll just go ahead and start wrapping up for today.
And then we'll maybe look at one last visualization and then get out of here.
I'll let you get out of here and get on with your day.
But basically last week we said, OK, start somewhere and then iterate.
And then I just wanted to build on that today and just say the place where you begin doesn't
have to be perfect.
And so this is, we're basically actually just using the lessons of gradient descent and
actually putting those into application sort of in our real life.
So this is sort of going to be a real life gradient descent function where it's, you
just start somewhere and iterate and just, if you go somewhere and it's not the right
direction, go backwards.
And then if you go somewhere and it is in the right direction, just keep going more
in that direction.
That's gradient descent and in that shape.
OK, so enough of that nonsense.
OK, real quick, just to show you some cool statistics and then let you maybe go explore
the data on your own.
So we said, OK, we've got a whole bunch of reviews and I'll just say, oh, let's just
look at this first one here.
And basically it just would be a block of text.
And so given a block of text, can we rank this 0 to 1?
And so if you're particularly interested in this, we actually did a whole talk on sentiment
analysis.
I'm not sure if it's uploaded yet.
If not, then hopefully soon.
Just kind of want to put up these last couple of visualizations.
We did cover a lot of ground with the optical character recognition, so we may want to see
some of these consumer recommendations.
But I've been droning on for a lot while these figures are rendering.
Does anyone have any questions, comments or thoughts about some of the material we covered
today?
Basically here's the review and you could basically get a ranking of this review.
And that would be, I think, minus 1 to 1.
So this should be a positive review, basically 1 plus.
So that would be between on a scale of 0 to 1.
This review we would give a 0.65.
So not the most favored cannabis product out there, but not the least.
And so the idea is, OK, you can kind of combine these.
So ask your consumers how they like the product and you could just ask them to write a preform
review.
And so that's what the cannabis diary is doing.
There's essentially saying, hey, journal about your day.
What did you consume today?
Journal.
Just journal.
And so then we could actually do the sentiment analysis where, OK, here's your block of
text.
Was it a good day, bad day?
And then you could correlate this to the actual, right?
So for example, this review right here has, this has chemicals related to it.
So this has 0.11% beta-pinene, so on and so forth, if you're interested in the THC.
So that one's primarily delta 9 THC.
That could just be how the lab measured that one.
But anywho, let's see if this runs into an error here.
If it does, we may need to save this.
We may need to save the product recommendations for next week.
My apologies.
Just still have one last error to sort out.
So it's been a long time coming, so bear with me for the setup.
But that's sort of the idea in the nutshell is, you know, now that we have all this data
here, so you have all of the cannabinoid concentrations and you have somebody's review, well, you
know, now we have all of the pieces of the puzzle that we need to do product recommendations
incorporating the consumer sentiment of each product.
And so this, I think, would be a pro-consumer algorithm, right?
Algorithms have just been getting a bad rap in the news lately.
But you know, it's just a word, right?
Algorithm, it's just a step of instructions.
That's what it is in my book.
And so this is a step of instructions that I think would benefit consumers, right?
And I think this incorporates a critical piece that a cannabis data science member brought
up in that not only do you just want to recommend consumers based on their history, but you
also want to base it off of how well they reacted to each product.
So just because they bought a product doesn't necessarily mean you want to recommend that
product again.
And in fact, if they didn't like that product, you definitely don't want to recommend that
one again.
So if they truly got a zero for a product, that wouldn't even be included in your algorithm.
So the idea is theoretically, right, they should just keep recommending consumers products
that they like more and more and more until I guess it would stabilize, right?
So ideally, right, you'd hit your long term X bar.
That's just your favorite strain, right?
You finally figured it out, right?
So you just tried a bunch of different strains in the store, wrote your review, got your
product recommendation for the next one.
And then you sort of gradually find the one you like the best.
And then there you go.
Then that would be your maxima.
So we're basically using that great gradient descent, right?
So then you reach your optimal point.
So yeah, I just kind of thought about that now.
But yeah, I think this would help consumers find their optimal product in the long run.
But I'll have to think more about that.
I would like to make two points as we close.
There's one gaping hole still that we certainly at CESC are trying to fill.
And we have an app that has been launched in proof of concept.
And we're currently seeking funding to release version two.
It's called the dosing project.
And it's called dosing project for a reason.
Because we're trying to come up with dose and amounts that are used.
And unless you specifically query or figure out a mechanism to get that into this data
series, you are missing a crucial piece of information.
Maybe this was too strong and they overdosed or maybe they underdosed or whatever.
So absent the dosing piece, you've got a hole.
We're certainly working to try to fill that.
The other major point I might make is in all of this, there's an inherent survey bias.
If you don't like a product, you're not necessarily going to take the time to review it.
Also it is possible we are seeing hints of this in the data that we analyze that certain
strains may be associated with couch locked.
You're not going to write a review because you're too couch locked.
So interestingly, there may be a strain bias that causes this too.
So one of the fixes may be not to have people rate, but to look for repeat purchase.
And especially if in your COAs, you put unit coding into it.
In other words, it's not just the lot numbers, not just what the batch is, but there's an
actual merchandise ID sequential code.
And we've done this at one point on a test launch of a product a couple of years ago.
We put actual batch, actual item number counting.
So if you're reporting or you review something or you put it in your diary or wherever you
report it, but if you're looking for a repeat measure with a different lot number, that's
a pretty good clue or a different item number.
It's a pretty good clue that you're in favor of this product because you bought it again
or you got it again.
And that may be at the end of the day be the most potent indicator of product recommendation.
Can I jump in?
Because you basically have raised three good points and I'll address them in reverse order.
That's how my memory works.
The last point you picked up is basically what I was talking about is this is what economists
like to talk about at lunch or statisticians or data scientists is what variables should
you use?
And so essentially I would say, okay, repeat purchase, that's a variable.
So you could almost use that as a dummy variable, zero, one.
If somebody's purchased it before, it gets a value of one.
If they've never purchased it before, you get a value of zero.
Include that variable in your model.
And as you said, it may be quite an effective predictor.
Like I said, it can enter your model in more sophisticated ways than that.
Maybe it enters in the averaging somehow.
So that was that one point, which was brilliant.
The second point was me just kind of defending the work is the reviews we have are quite
flawed.
And as you said, we actually encounter sampling bias because not everybody loves to review.
And so that's where that kind of hits on the third point is that's really where the value
added comes from the dosing project.
I can't even disagree, maybe an app you're working on is marrying the product data with
the test results and the review too.
And so, John, you're doing that in a clinical study.
And so that way you're getting everybody.
So that way you don't have sampling bias.
So to your solving problem two by number one, which is what you're doing.
So that gets to the third point, your first point, which was the dosing project.
And that's where I think you're adding a lot of value.
You're marrying the or contextualizing, if you want the more formal word, contextualizing
the product with the lab result with the consumers outcome, so to speak.
So their effects, sentiment, well-being, utility, whatever word, enjoyment, whatever word you
really want to use or what you're measuring.
Maybe this is a medical patient and they've got a very quantifiable outcome.
So as I said, the sky's the limit.
And that's what's fun about this.
We're just unlocking data.
And you're just using tools, right?
So that's why I was saying everything came together.
Next week I'll finally wrap it up and actually give you some product recommendations.
So maybe we'll just do that first thing next week and not do a bunch of fancy stuff.
But now we have all the fancy stuff in our tool belt, right?
If any of you need help with the optical character recognition, reach out to me because this
is something that you should be able to get installed on your computer and just be able
to regularly use, which is awesome because who knows what cool uses you'll think of for
it.
And then we just did our demonstration of how you can wrangle this data in the Canvas
industry.
So I'll let all of your minds chew on this.
I know I've taken up a lot of your time.
So I just want to thank you all for coming because it's you and your ears that help advance
all of this, right?
So it's us bouncing ideas off of each other, working with each other.
So I want to give a big thanks to Canvas who really tested all of this data parsing.
So the reason we have all this fun data to look at today is because of Canvas.
So big thanks to Canvas.
Thank you, Canvas.
And John, too, for really getting this ball rolling.
Because actually, right, I'm taking the data now and I'm going to start looking at the
ratios between the lemmonine and the beta-pinene.
And it's all really great.
It's very exciting what Canalytics is doing.
Be careful on that.
You're talking about for the raw gardens?
Yes.
Yeah.
Be careful because that comprises a number of different product types.
And it's a mishmash.
We have to subset by product to make sense out of this.
Don't go overboard on the analysis of that until we've done that.
Oh, right.
I was just putting it into a data frame and then selecting the columns and then x, y,
and then plot.
That's all I was kind of doing.
Yeah.
Unless we have subtypes, it's kind of meaningless.
It was more for a reality check just so that we could check all this new data and all this
OCR parsed PDFs.
It's just crazy amazing.
And so we have so much more data.
So I was just going to rerun the same plots, just ratio, not even using like K&N or anything
fancy.
May I encourage you to use the beta-karyophylline-alpha-uniline ratio instead of the beta-pinene-limonine
ratio?
Oh, I'm on the hook doing both.
Okay.
Excellent.
Because I think that's...
You've got the slack today.
Okay.
And just in general, are you guys comfortable now with the Tesseract and PNG pairing?
That's interesting news for me, if that's the case.
It's pretty well ironed out.
Like I said, we haven't tried any images in the wild.
So that's the next step is...
Images in the wild are not going to be all that current because most people don't get
access to paper copies of this.
Well, ask your retailer next time you're there.
And then like I said, it's good that you're doing a reality check, Candice, just making
sure we've got numbers in that.
I think the image is the more important one.
Well, actually Keegan, it was your suggestion to look for outliers.
True.
Give yourself a hand.
It is important.
And in fact, I'll leave you with two bits of nuggets.
The first is I'll make the code available online so that way you can continue exploring
and maybe fix this product recommendation and beat me to the punch.
And then the two is I know I've got about this on Saturday Morning Statistics is outliers
are surprisingly interesting.
We talked about those with patents.
So a lot of times you may exclude outliers, but if you're doing a patent hunt, then you
want to find the outliers.
So they're always worth looking at.
Maybe there's something going on there.
So anywho, we've taken up a lot of time, so I definitely kind of want to get you all out
of here today.
You're respectful of your time, but I love the enthusiasm.
So please keep it up and I'll share all the code with you so that way you can go get your
hands on it all explore and we can reconvene next week and we'll at least do the consumer
recommendations and then we'll have time to do some sort of fun other data science exercise.
So please let me know if you want something.
Otherwise, I'll think of the funnest thing I can.
And we can have a fun time crunching numbers again.