Welcome to cannabis data science, the 73rd one.
So we've been doing this for about a year and a half.
So thank you all for lending your ears and making this happen.
Long story short was working on an automation task because we actually found an awesome
source for lab results.
So we've been looking at lab results for a handful of samples and strains, and I figured,
well, let's just go straight to the source.
And so today we'll do that.
Go straight to the source, get the lab results.
Ended up spending a bunch of my time doing automation.
So we'll finally have an awesome data set for you.
But then, you know, you fine folk may have to help me in the coming weeks with some of
the statistics.
Essentially, the topics of the day are, you know, web scraping.
And then just to introduce some fun data science topics, I thought, OK, we could also talk
about margin of error.
That's actually pertinent to the data we'll be collecting in a couple of sessions.
And then we'll also look at secure hash algorithms just because let's just spice it up with
something interesting and fun and just, you know, fun fact.
These little things end up having so many uses.
So turns out I was just reading and looks like they were formally created by the NSA
in 2002.
And then once you get these new technologies, it's just amazing what people will think to
do with that.
And so ultimately, you know, these hashing algorithms are what's underlying Bitcoin.
And this is what the authentication system for the CanLytics website uses.
We use SHA-256 hashes.
So this is just pretty standard technology that people use.
But anyways, I'll just show you how you can use them today, too.
So what are we even going to be doing today?
There's a fantastic laboratory in Michigan, PSI Labs.
And through December, the end of December of 2021, they've been diligently posting their
laboratory results.
And as you can see, you can get quite detailed information for the various samples that have
been tested.
So this is a Marshmallow OG.
It's awesome that they've taken an image of this.
And I'll actually introduce to you some fun side projects you could potentially do.
But you've got some images.
They've got for various samples.
It depends on what people requested or what analyses were performed.
But you may have pesticides.
Looks like there was no undetect across the board.
Microdials, metals.
Lots of data.
From what I could tell, PSI Labs does its state that they don't want robots on their
website or this or that.
I was thinking from an accessibility point of view, I made this joke the other day that
I'm half man, half Python.
So I think you could almost make an accessibility claim that if your preferred way to browse
the web is through Python, then so be it.
And so what's cool about being half Python is that a human may look at this and you can
only do so much.
You can only remember so many photos.
You can only remember so many lab names or sample names.
But with Python, we've got a stunningly good memory and we can work incredibly quickly.
So let's dip into some of our extraordinary abilities here.
Because basically the idea is there's hundreds of pages here and if you just wanted to go
through them by one, it would be hard to review all of these samples.
And the idea is if you reviewed all these samples, you have a good idea about what's
the distribution of can of noise, flowers, concentrates, you name it.
Because every type of sample...
So basically the idea is we've been using requests a lot.
We just request a page and that's fine and dandy.
But I guess I won't go...
I'm still with you all, right?
So for sure it is.
If you basically look at everything it's sent to you, basically what ends up happening is
this whole body is rendered with JavaScript after it arrives in your browser.
So it's hard to just request this in your traditional manner.
So basically what we're going to do is we're just going to use Chrome through Python.
So that's why I said we're sort of half man or half woman, half Python here, because we're
still going to use Chrome.
We're just going to drive Chrome through our Python console here.
And this is going to be an interesting script to go through today since it is web based.
So hopefully we've got enough bandwidth to get through here.
But the idea is if you actually start looking at what's going on programmatically, you'll
see that it looks like the developer may have coded in 492 pages, but if you start looking
at how the website behaves, you'll see that there's actually thousands of pages of lab
results.
So here's lab results going back to 2019.
Just keeps going and going and going.
And so I realized that there's almost 5,000 pages of lab results.
So estimated 10 lab results per page.
That's about, and that's why I said you're going to get your hands on about 50,000 lab
results here, because we're going to get just shy of that, about 49,000 lab results.
And I will deliver all these to you as promised.
As you'll see, the data collection process is a little lengthy.
So I do have a lot.
And then I'll basically just start dripping these out to you.
But everyone who signed up will get every lab result that is collected.
So how do you know about doing this?
Well, what's cool is, so check this out.
We'll basically just open up a Chrome browser here.
And we'll want to define what page we want to point it at.
So we basically say, OK, Chrome, go to PSI Labs website, please, and go to page 4200.
And so now Chrome will diligently perform this task for us, albeit it may be a little
slow, a little slow out of the gate just because of all the tasks I'm asking my computer to
do at the moment, such as streaming to all of you fine folks.
So hopefully this will get there sooner rather than later.
So basically, I went through one of these pages here.
So if you look at the Marshmallow OG page and this page, and you just say, OK, what
are all the unique data points that we can collect?
Well, oh, so finally.
So awesome, this page loaded page 4200.
Fantastic.
So that's why I was saying I think you could make a good excuse, not even an excuse, I
think you could make a good case that why should a website say that you can't access
it through, quote unquote, automation?
Because it's basically just, what if that's just your preferred way to browse the web?
That's how you want to access the web.
Anywho, that's what we'll basically be doing.
The other reason is why should we even collect this data?
Well basically, I had this almost like a panic slash nightmare on Monday because I couldn't
find this website.
And then it started to make me realize that we could easily lose all this data to history.
You know, they were only getting the lab results just stopped at 2021.
So maybe they updated their software or maybe their software developer is no longer developing
on this application.
So there's many possibilities and long story short is I hate to see such phenomenal data
on this critical thing.
We all think that cannabis can do a lot of positive good in this world.
So if all this data just vanished, that would just be a disaster.
So I almost think future generations may look back on us and be like, you didn't even write
down all that data and you had all the tools.
It was just sitting there right before you and you just didn't take the time to do it.
So basically, I just sort of had this panic on Monday and realized that yes, PSI Labs
probably got this data nice and diligently stored in their database.
But it wouldn't hurt to just go ahead and create another archive of this data because
I think this is data that in my opinion people are going to be looking at for years to come.
So long story short, we can do our part and try to collect it.
So and also I also kind of just see this as meeting people to get all the lab results
online and accessible.
That's it.
Yeah, an API would be nice but we've got to test cannabis here.
We did our part.
And so basically, I'm saying, oh, fine, we'll meet you where you are.
You did your part.
You did a phenomenal job.
We can take this and run it home for you.
So long story short, we can basically just look at their HTML and start finding the various
elements that we need.
So basically, we can say, oh, on this first page, we can get these 10 samples just as
a sanity check.
We can look at this first card.
So these are still sort of just like HTML elements.
But I think we can print it out.
So basically, this is all the text on the card.
So we're looking at presidential by sensei.
Sure enough, that's the first card right here.
Presidential by sensei.
It's actually sensei seven.
Sure enough, sensei seven.
And so that's what's nice about the computer, right?
Us humans, we're real imperfect.
But the idea is, as a computer, I think we underestimate how good their memory is, especially
with these large data sets.
So for example, I figured it'd probably be worthwhile to go ahead and deal with images
in general with online images, these are PSI labs images.
So they belong to them.
So you can't really just download them and copy them around willy nilly.
But in general, a link on the website, links on the web can be readily copied and shared.
So basically, the idea is, you can readily link to their image.
So you can save the URL to their image.
And then if you ever want to go look at this image, say you're doing all your research
and there's something just kind of funny about this presidential sample, you can go check
it out.
You can go look at their image and displaying it on your website, you have to be super careful.
Typically, you'd want to give PSI labs credit.
But generally, that's how Google and Google images works, right?
They're just using links to images all over the internet.
So generally, like I said, I try not to be as ruthless as some of these companies that
you see.
But in general, right?
For example, in the presentation today, I think it's worthwhile showing you their image.
So I'm not copying it or saving it.
I'm just kind of showing it to you and talking about it.
But funny enough, does anybody recognize what this is?
Tri-combs?
One more time.
Tri-combs?
Well, exactly.
So all the little crystal-y things are tri-combs.
I was thinking that there's so many plant parts here, right?
The botanists could pick out so many.
But basically, this whole thing is a calyx.
So this is basically like one cannabis flower.
And that's actually the Canlytics logo.
It's just a calyx.
And so basically, like a bud is just hundreds, if not thousands, of these beautiful little
flowers.
And yeah, you've got the calyx.
I think the orange hairs are called stigma, perhaps.
And then exactly the tri-combs.
It's just covered in tri-combs.
And when you start to look at pictures like this, it actually makes the brain kind of
think.
And that's actually where my question for our research for today came from is, I mean,
look how covered in tri-combs this cannabis flower is.
And so it kind of makes one think that could a cannabis breeder eventually breed a cannabis
flower to basically just be one giant tri-comb?
How big can these tri-combs get?
So I was thinking from a biological point of view, it may actually be, we may be able
to parse it out in the data.
Are these tri-combs essentially getting bigger over time?
And so the way we could basically say is, is the average cannabinoid concentration increasing
over time?
And then I was just going to throw out two real cool, quick machine learning projects
you could do.
You've got all these images.
Well it's maybe sort of extraordinary how a computer could process an image.
And so it's sort of like if you look at a cannabis flower, you could say, oh, that's
high quality cannabis.
That's low quality cannabis.
And even when you start, say, for example, I worked at a cannabis testing laboratory
and you start to see so many samples that you could actually even start to get a good
approximation in your head.
Like, okay, what percentage THC would this flower test at?
You're not going to know spot on, but you can kind of just get an idea from just kind
of looking at how covered in tri-combs the flower is.
So what I was wondering is, would it be possible, and you've got a couple different images here,
right?
So you could maybe use the flower zoom now because this would be a bit more realistic
in your application.
But could you train a machine learning algorithm to predict the cannabinoid concentration by
just looking at the image?
Because the idea is for all of these samples, right, we have the concentration and we have
the image.
So you could actually, and like I said, I've never done any image processing, but I was
just thinking that this actually is an application that you may be able to pull off.
Is there any way that you can predict the concentration from the images?
Because remember, we've got almost 50,000 observations.
So long story short, as a diligent data scientist, I figured it would be worthwhile to just go
ahead and record the image URLs, whether or not I'll use them, I don't know.
But that's the cool thing about data is I always say never throw away data.
Save every piece of data you can because who knows when it will be useful by who in the
future.
So I may never use the images and maybe some clever data scientist will think of a use
for them.
So long story short is you can go ahead and collect all the sample details through this.
It's a little bit more complicated, but the code is code, so if you're interested in
that you're welcome to read through the script.
The idea is you can start to get all of the cool data points that you need.
And then I promised you hashing algorithms, so I'll just go ahead and introduce those
to you real quick.
So essentially the way the hash works is you basically have a secret and a message.
And so those are commonly called your private key and your public key.
So in this case we'll just say okay, the date that this was tested is a quote unquote secret.
It doesn't actually matter.
I'm just using this to create a random ID.
But I just kind of wanted to explain the concept to you.
You basically got this secret that only you know and then somebody else has another piece
of information.
So in this case sample name and then basically if you combine these together in a hash you'll
always get the same hash.
So it's just going to be a bit, and I'll show you here in a second, it's going to be a big
random string that it's unique and you can't go backwards.
So the idea is once you have the hash you can't back out this information here.
So it ends up being useful in many different scenarios.
This may have been a little much of a tangent, but I don't know, I threw it in there.
And then it's common to quote unquote salt the public key.
So basically just have just another random piece, not random, but another piece of data,
in this case the producer that you'd add to the public information.
So that way somebody basically says, oh, I want presidential OG.
I know it's coming from the producer, salt that, and then I'm the only one who knows
what date it was tested.
And then all of a sudden you can create the sample ID.
And then that's just going to be this long random string.
It's completely unique that no one else is going to accidentally create the same ID and
you can't back out any of the information.
So the idea is this random ID I can just readily share with a lot of people.
It has a lot of information to me, but it has no information to anyone else.
This has no information to anybody, but I can now, now if some producer wants a sample,
I can potentially look this up on my end.
So kind of a bit of a tangent, but I thought it would flow a bit smoother into the meetup
for today.
But if you're interested, feel free to talk with me more about that or what have you.
But long story short is we can go ahead and finish collecting all the cool data points.
So now we've got the analyses and I'm not worrying about cleaning the data much yet,
other than just removing like HTML stuff.
So I'm just getting the data as is and we'll clean it up later.
So basically, and now we'll go ahead and get the link to the actual details.
So we basically find all of this data here, except I skipped these compounds since we'll
just get those from the sample specific page.
So cool.
So we basically, we've got its name, company, date tested, sample type, analyses, and image.
So tons of information already.
And then we can basically, we can get its COA.
And once again, you can't just copy the PDF, but you could record the URL.
That way you could, if need be, you could find the COA in the future.
And here is where I want to really applaud PSI Labs and introduce the other topic that
we were going to talk about today, the margin of error.
And so PSI Labs has internally studied their methodology, right, so that you can see they're
even listing it here.
They're using GCFID, which is actually non-traditional, I would say, for cannabinoids.
A lot of, or not even non-traditional, but it's a, remember this is 2016.
So what would be actually interesting is to see, okay, in 2021, maybe they're using, the
common method used today is HPLC.
Back in the day, right, back in 2016, I think it was really common to do cannabinoid analysis
by GC.
This is gas chromatography versus liquid chromatography.
And so if you're a diligent data scientist, as we are here, so we're going to record the
method that was used, that could be applied statistically, right?
So we may want to see if the method that you use to test cannabis has any statistical effect
on the concentration, which it may.
So we may find out that the HPLC is the superior method, and we may only want to, or the GC
may be the superior method.
We don't know until we test it, right?
But I've got my hypothesis that the HPLC may be the superior method, since that's the one
that people have kind of adopted, but you can do pretty incredible things with gas chromatography
too.
Gas chromatography is what people normally test terpenes with.
Okay, so enough of those tangents.
Another cool thing, so right, they've done an internal test, and they've tested and they
found that, okay, we have about a 10% margin of error.
So you can think, and I want to say this is maybe like a Z distribution or something,
so I don't know where, how exactly they're getting these sample size numbers from, but
let's say that these are accurate, then that would mean that PSI Labs has internally tested
their method with a sample size of almost 100, right?
So that means they created an internal standard, right?
So what's an internal standard?
It's basically just maybe say somebody, they're getting a bunch of cannabis sample in, so
what they could do is they could grind up a bunch of flour or mix up a bunch of concentrate
or grind up a bunch of cookies together.
And the idea is you could create sort of a representative sample of what flour would
may be, and then you can just, you basically just test, well actually, that's an internal
standard.
They probably actually need to get a third party standards for this.
So they probably actually have to pay a third party company that says, oh, we sell 99% THC
solution.
Then you'd basically buy the 99% THC solution or what have you, and then you may dilute
it, right?
You may buy a 99% THC dilution, you may dilute it to the point where it should have a 50%
concentration.
You test it 100 times.
So that means they, right, they created this reference standard, then they ran 100 samples
on their GC, and then their margin of error had to be within 10%.
So that's actually a pretty wide margin of error when you think about it.
So this has a 50% concentration of THC.
That means it could have, yes, so it could have as low as, don't trust any of the statistics
or math I'm telling you today because as I said, we were off to a super rocky start.
But long story short is it's important to take into consideration your uncertainty.
It's useful to know that this sample has a minuscule amount of CBDA detected, but the
uncertainty is so great that statistically I don't think that's necessarily different
from zero.
CBG may be statistically different than zero.
And so, you know, and then just kind of keep that in mind.
Actually the margin of error is quite low on this sample.
But long story short is this is a useful statistic that I do believe most laboratories calculate,
but I think very few report on their COAs.
And I'll get to the juicy part here too.
But basically this is kind of typical, but maybe even on the low end.
So this is a 10% margin of error, which, you know, it's not, you can see higher in the
cannabis industry.
In fact, you know, we were looking at plant patents and it was kind of, it became pretty
apparent that the plant patent we were looking at, they were working with a 20% margin of
error.
And so, you know, cultivators, I think they just may need to get edgy.
Right?
I never fault anyone, right?
So people will get flustered and they'll say, oh, you know, we sent three different samples
to three different laboratories and got back three different results.
One, are those, you know, three different results, are those within the laboratory?
If you sent them to the same laboratory, so let's say you sent this presidential flower
to PSI labs three times, you know, they may send you back a 22%, a 25%, and then they
send this 23% here.
And so, numbers like that can fluster a cultivator, right, because they're like, oh, you know,
which is it?
Is it 25% or is it 21%?
Those numbers, whether we like it or not, could easily influence the price that that
would set you at retail.
So basically, it's just, it's an information game slash problem where it's just trying
to get the information out the best they can.
And that's why I applaud PSI labs, is they're doing their part to get that information out
there.
And so I don't think we have it for every compound here, but they're reporting it for
the Campanoids, which is awesome.
And so then like diligent data scientists will go ahead and collect the margin of error.
And like I said, I don't necessarily know how I'm going to use it right away, but we
can step one, right, step one, get the data.
So we'll just get the data.
So what's this even going to look like?
You know, I'm getting their QR code, which I may not even get.
So basically, I was kind of thinking about this and, right, it's always good to be transparent
about all the potential problems.
I was speaking with the laboratory in Las Vegas and they did not like the idea.
Actually I take that back.
Maybe it was a, actually, I think it was a cultivator in Oregon and they did not like
the idea of having their COAs out there because they said that people would copy their COAs
and you know, fraudulently use them.
And that's no good.
And so some labs, you're sort of implementing things like QR codes.
That way you scan the QR code.
It goes to PSI labs website.
You can kind of confirm.
So I don't necessarily know if I should use this QR code or not.
I was actually going to say that the QR code may be sort of a 20th century technology.
And the 21st century technology equivalent of a QR code would be, Candace?
Drum roll.
It's actually going to be essentially a data NFT.
That's correct with Ocean, yes.
And so that's why I was sort of introducing these hashes because I mean,
Ah, okay.
I get the logic for sure.
Well I'll kind of get, we can kind of maybe build this more up next week.
But the idea is, right, you can take any random, not random, but you can take any message and
encode it with a private key and you'll get back this hash.
And so the idea, I do believe, with the data NFTs is, right, and I may be mistaking this
entirely, but I do believe if you have a certain set of data and you hash it, you'll get a
unique hash, right?
And it'll just, that's your hash, right?
That data will always give you that hash.
If the data changes, you'll have a different hash.
And so that's sort of the idea behind traceability is, right, once the laboratory reports the
numbers, right, it's got a hash.
And the idea is the hash is sort of on the blockchain, so a bunch of people will agree
on the hash and you can't just go back in time and change the hash willy-nilly.
You know, you can change it for the future and then everything will be recorded, but
you know, this original hash will kind of exist.
Right, like a ledger.
Exactly.
So basically the idea is, you know, if a laboratory reports this set of results, they'll get hashed
and that will sort of just be this unique hash.
So that way, you know, everybody can basically agree that yes, you know, those were the results.
And like I said, I am not the smartest cookie in the jar, so don't try to go to me for an
explanation about how the blockchain works.
But basically that's my monkey brain understanding of what's going on is somehow the data is
getting hashed, the hash is unique, everyone agrees about the data going in on it, and
you know, as far as the data NFTs, there may be essentially some sort of ownership involved.
But like I said, I'm going to read up more on this, on the statistics and data, so that's
what I'll teach you and I'll try to learn more about some of these other cool technologies.
No, it's really cool.
And then with Ocean, with their training, they actually have a Rinkeby test network,
so you can play around with it.
You can actually send messages, transactions back and forth, get the different hashes and
kind of get a real feel for what's going on without having to hook in your bank account
or credit card, right?
It's all free.
They can actually give you a free coinage too.
It's pretty interesting, their training.
And I think we may just be sort of on the cutting edge of it, right?
It's basically like, remember the secure hashing algorithms came about in 2002.
Bitcoin came about in like 2008 or 2009 or so.
So it's not like, and these days, if you're talking to a software developer, they're going
to expect you to implement some sort of hashing algorithm.
If you're doing like authentic, right?
So like I was working with a developer and we were doing authentication for an API.
It's like, yeah, you need to implement some sort of hashing algorithm, at least the SHA-256,
right?
There's more complicated ones that exist today, but this is sort of an industry standard.
So it's basically like, once a new technology will eventually become the standard.
So I think QR codes are useful, but I think especially for laboratory testing, I think
things like data NFTs, essentially smart contracts, I think those are just going to end up being
standard place with laboratory testing.
So that's what I'm trying to, and not like overnight, it may be five or 10 years from
now, but that's what my hypothesis is.
So that's why I'm trying to learn about them.
Okay, so now we're kind of at the end.
I'll have to think about some way to make it up to you all for the rocky start today.
But basically, wrote all of this code into these reusable functions.
So now you can basically go through and I shared this script on Slack, but I'll make
sure to email it out to everybody as well.
And then I'll also post it all to GitHub.
And I will email you the data.
And so the idea is you can now run this script on all the pages and collect the data and
we can look at the data.
And keep in mind, I will deliver you the near 50,000 results that were promised.
I was only able to run this for two or three hours yesterday afternoon.
So in those three hours or so, I collected 2,000 lab results for you.
And then I'll go ahead and immediately share.
These are their most recent 2,000.
And I know I promised you 50,000.
So I will deliver.
If you will be so kind to wait.
So I'll go ahead and start collecting the rest.
And as I said, it takes about an hour and a half to collect 100 samples.
So it could literally take me two or three days of computing power to collect all 50,000.
So I'm sorry if I was a little misleading.
I hope I didn't mislead you.
But everyone who signed up will get all of this data in full.
And better than that is this is the unprocessed data.
And so basically, and I'll go ahead and wrap up here since we don't need to beat you to
death with this.
But basically the idea is, okay, now that we have it all collected, we can go through,
clean it, right?
We can remove the delta symbols, right?
We can parse out the units.
We can handle these positive, negative signs and margin of error.
We can do all that fine, dandy cleaning stuff after we've collected it.
So I'll do that for you.
And then basically next week, I was going to do this today, but it was just too jam-packed
of a day and we had such a – thanks to me, we had a rough start.
But for next week – and we can have a bit more of a conversation about the data next
week too.
But next week we can finally start to explore this.
And as I said, the main question – and I actually saw some cultivators and a business
magazine talking about this, and so I think this is what people are interested in is,
are cultivators able to essentially get better and better at breeding for these compounds?
And so I kind of wanted to do a time series analysis, right?
We've got results going back to 2016 all the way to 2021.
So in this five or six-year time span, have breeders been able to breed for statistically
higher levels of D-limonene, beta-pinene, mercine, curiophiline, linalool, THC, CBD?
Or have processors been able to get more efficient?
Are concentrates increasing over time?
So I think these are questions that people are interested in that I think the Cannabis
Data Science team is uniquely positioned to answer.
So first step, first we'll just get the data.
Then we'll clean it up and analyze it.
So my apologies.
It's a completely mess of a presentation today.
So hopefully once I get this data in your hands, that may make up for things.
But does anybody have any questions, comments, thoughts before we call it a day?
All right.
Too cool.
Until next time, have an awesome day.
Keep advancing Cannabis Science.