Good morning, Charles and Brooke and everyone else. So thank you for joining the Cannabis Data Science Meetup group.
My name is Keegan, the founder of Canlytics, a company to make cannabis analysis simple and easy.
Today we will be looking at lab results in Washington State, and Charles has prepared some work trying to predict if a sample may fail quality assurance testing.
And then I can follow up with some analysis on cannabinoids afterwards. So without further ado, Charles, would you like to present some of your work?
Sure. Let's see, I've actually never presented.
Where is that?
And can I even do it?
So it should look like a box with an up arrow.
Oh, that one. Okay.
Cool.
Okay, so previously I had done some work with trying to predict sample failures in the lab. And so I came up, and so most of the stuff at the beginning is just sort of reading in the files and cleaning them up.
And so, and then encoding them and splitting the data into test training and testing sets.
And then because there's a huge class imbalance.
There's way more passing samples and there are failing samples.
And so I try to manipulate these class weights to help the model compensate for the lack of failing data.
And this just, all the data is categorical, right? It's, you know, it's not numeric. So in order to use CatBoost, you tell it to which columns are categorical. And then CatBoost is really good for dealing with categorical data, and it tends to be a fairly good classifier.
So you train the model and what I had done is I had taken this data and used Optuna to come up with an optimal set of parameters for this particular data with CatBoost.
And then Optuna is a genetic algorithm that goes through and it'll try something and they'll try a set of parameters, and then it'll, you know, it'll come up with a, it'll actually come up with a population of parameters and then try those, take the best ones, and then use those to come up with new parameters.
And this takes about a day, day and a half to run.
So, last time, a big part of the problem was is I didn't really have a goal.
So Keegan had pointed out that the big goal was to not miss any failing samples.
Because last time I tried, I was kind of trying to optimize everything.
So I came up with this particular model and it only misses one of the failing samples.
So, right, it pretty much meets your goal.
But, oh, and so, and also CatBoost will tell you like what feature, what, how important each one of the features were.
So the lab ID turned out to be the most important for some reason.
I mean, maybe particular, either a certain lab or certain labs are more strict or certain labs just tend to get more samples that fail.
You know, we don't really know.
But this, the whole thing bothered me.
This is kind of really odd.
I mean, there's obviously a lot of false positives, you know, thousands.
So that's a lot.
So I was wondering, is this thing really learning anything?
And the answer is no.
It's basically a dummy classifier.
So what it does is it predicts that marijuana fails 99 percent of the time.
When in reality, it only fails 12 percent of the time.
That was the sneaking suspicion I started to get last time was maybe there's a certain product that tends to fail more than others.
And like you said, it's basically just pinpointed that sample type and just and you're basically saying it's just saying that that sample type fails every time.
Yeah, basically. Yeah. 99.5 percent of the time. So, yeah, it basically.
Well, that's what the classifier thinks or predicts that marijuana fails almost all the time.
So and but yeah, it does.
It does fail more than the than the other product types.
But in reality, it's only 12 percent.
And is this mixed marijuana by chance or it's just labeled as the type is labeled as marijuana.
I don't know what the the intermediate type by chance.
No, that's the the actual product type.
OK, we'll dive into some of this data here in a second.
So from my understanding, the intermediate type is really.
Is it is kind of the main classifier.
So. Like, so, for example, here, just like the type, right. So everything's going to be categorized essentially as an end product.
Well, everything that makes it to the store shelves.
And so I'm curious about the product type.
So for marijuana, the the main product type or.
Was it subtype or what did you say it was?
I can't. It's your immediate type, intermediate type.
It was intermediate type. And so that was flower.
But if you go back to the old notebook.
There is very few entries in the table that had intermediate type.
Most of them were missing.
The product type or just type was the main was the one that was filled in the most.
So intermediate type was kind of was missing most of the time.
Like 85 percent of the entries were missing intermediate type.
And I did try and go through and fill those in, but it just kind of didn't seem to.
There's kind of no clear way forward with that.
Just.
Looking at some of the variables here. So.
Were those just not end products by chance?
The ones without intermediate type.
No, in in product also was missing a large number of intermediate types.
I would have loved to have used intermediate type.
As an input, but it just.
There was kind of no clear way forward as to how to fill in those missing values.
So let's power on for now and then we'll start poking at this data here in just a second, because I've done some follow up work with Canterbury analysis.
And then we can dive back into the sample type discussion.
But please continue because no need to get hung up on this for the time being.
Yeah, no, I'd be interested to, you know, to find out. You know what features are really viable and and how we could fill in some of that missing data.
So the next thing was in product, which was predicted to fail one and a half percent of the time.
And it actually fails. In reality, it fails one percent of the time. So this was pretty close.
Can I pause you there? So this is where savvy cannabis licenses will.
Do some interesting analytics, predictive behavior, cost benefit analysis, because essentially what you can do is you can factor, say, a one percent chance that your products fail into your your cost, your estimated costs.
So that way you can budget correctly.
So, for example, if you don't take that one percent chance of failure into consideration to that's in economics terms, that's a cost.
Right. Because there's a one percent chance. And upon that chance, there's a cost.
Right. You're going to you're going to have to expend to destroy the product.
It's going to be a loss in inventory. So that's essentially an expected cost.
So. So, so long story short, I think this is something that businesses should take into consideration.
So when you're starting your cultivation, I think you should factor into your your cost, the estimated probability that you may fail.
Because what I what I've seen is cultivators, processors, they don't account for the probability of failure.
And so then when something does fail, it becomes a disaster because they say, oh, like we know they didn't they didn't budget that they didn't plan for that.
And now all of a sudden it's hard for them to make ends meet.
So so I just wanted to pause you there just to kind of drill that point home that although it's just a one percent chance of failure,
if you're producing large amounts of flour or concentrates, you know, one percent is not it's not negligible.
So something to pay attention to. Anyways, Charles just wanted to drive that point home real quick.
OK, no, that's that's a good point. That's.
Especially in startup businesses. Yeah, there's nobody.
Yeah, nobody. A lot of people don't factor in that something could go wrong.
And when something does go wrong, then it is a disaster because they're operating on such a tight budget.
And they haven't thought about these things.
And you raise an interesting dimension, another factor, the size of the company.
So this number may be biased down to say large cultivations, they have things under control, maybe for the most part.
That's an assumption. But, you know, maybe they they're have their, you know, their flower rooms well.
Quarantined and they're able to expend a bit more to keep their failure rate low. Plus, they're sending in tons of samples.
So it may look like the failure rates low. But if you do it conditional on business size,
I wouldn't be surprised if smaller businesses may even have a slightly higher failure rate.
That's interesting. And so is there a way to get the size of the cultivator out of out of the data or or maybe even the length of time they've been in business?
Well, there's a couple of ways you could proxy the size of the business.
So you could just do sales. So you could there's sales data so you can rank the companies by their sales.
There there is tiers. So technically there's tier one, tier two and tier three.
That data may be there. I suspect it is, but we'll want to I think that data is there.
So we'll want to make make certain so you could just look at the tiers. That would be the simplest.
Do tier ones have a higher failure rate than tier three?
And then next, I would run a regression of if you can do that. I'm thinking if you just coded zero one as fail,
you may be able to run a regression on like a like a logistic regression on failure rate on sales.
So do companies that have higher sales, is their failure rate lower?
And my hypothesis is it is it may be because they you know if they have higher revenue,
then they may be able to expend a bit more just on, you know, quality control, keeping the rooms clean.
OK, checking your HVAC system. You know, you know how it is.
You know, sometimes smaller businesses may try to cut corners, try to keep costs low.
And then that may or may not result in a higher failure rate. But it'd be interesting to see.
OK, well, that's good. That's yeah, that's something I can I can try out and try and move forward with.
So one thing I did want to point out was that in product made up 68 percent of the training data,
whereas marijuana only made up 12 percent. So the classifiers seeing in product more.
So it's able to learn more about it, learn it, you know, learn about it better.
So that I believe that also is a factor in how you know, and on the numbers that came out.
Because when you go down to.
Harvest material, it predicted that it was the failure was 58 percent when actually it was seven percent.
And that made up 13 percent of the data. So again, it didn't see as much of it.
And these like microbial failures or.
I guess we didn't really break that down, but I guess if it's harvest material, that must be microbes or it seems a little high.
I don't know, because I only took data that you knew before the testing.
Because if you started taking data that you knew about after the test happened, then you're peeking into the future.
So.
You know, you don't really know a lot, right? You know, the type, you know, the producer, you know, the lab.
And you could find some things out about the producer, but.
My thing is a predicting factor, but I'm saying.
Like, as far as how you coded up failures.
Was there like a variable that's used to mark overall failure status or are you going through each compound and comparing it to its limit?
Well, in the in the lab data frame, the lab results data frame, there's actually a pass fail column.
Interesting. So. This may get complicated, but.
It could be interesting to to start breaking the failure down by what what's failing.
So, for example.
Concentrates are the only type of sample being tested for residual solvents.
So it could be interesting to say, look at what's the chance of.
Concentrate failing residual solvents.
And then you could even break that down of what's the probability of it failing for each of the different solvent. So what's the probability that it fails for butane?
What's the probability that it fails for propane?
And so on and so forth.
And then with flour, you may want to break it down by essentially the microbes.
I don't remember them off the top of my head, but we've got the data set here.
So they're testing for E. coli and salmonella. So.
You could you could.
Look at the failure rate for each of those. And so this the way you would do this is you would essentially have to calculate those failure rates.
Manually by looking at the, for example, the microbial pathogenic E. coli variable and then compare that to the state mandated limit.
Which may be 10,000 CFUs. So that's a colony forming unit per gram.
Don't quote me. Don't quote me on that. So check the the whack.
But long story short, we could potentially get more granular on the failure rates. So look at sample type or for different things.
Okay.
Yeah, it's something I can work on.
There's so many variables there's that you have to kind of narrow it down. So, for example, what's the probability of a residual.
So what I think would be interesting is, okay, just look at concentrates.
So you may have to look at the different types of concentrates and then try to estimate the probability that they fail for mycotoxins.
Just broadly for any mycotoxin. I think that's ochratoxin and aflatoxin.
So you could just say, okay, what's the probability that residual solvent fails for a mycotoxin?
And then what's the probability that it fails for a residual solvent?
Just any of them. So whether that's butane or propane. So that's a bit more complicated analysis.
But once again, it's just going to provide more information to to processors so that way they can know, okay, what's your bigger risk?
Are mycotoxins your bigger risk? And then that would essentially come from biologically dirty. So just kind of dirty input.
So that would be like, okay, check your input if your mycotoxin risk is high.
And then if your residual solvent risk is high, then that means you want to address your process.
So you want to address how you are getting rid of your solvent.
How are you purging your solvent from your products, essentially?
So sorry to go down that rabbit hole. But there's in a way there's endless there's endless analysis there, right?
Because you can you can you can break that down a lot.
Right. Okay. Well, that's something to get into.
And I definitely have a lot of functions now to deal with this with this table and cleaning it up and pulling stuff out of it.
Good. Good. Yeah. Well, you want to bring us home here, Charles, and then we can potentially get our hands on the data real quick.
And I'll show you just you've done a lot more complex analysis than I have.
But I've just put together a couple histograms of some of the cannabinoids and we can just look at the variables and talk about talk about the data.
That's what we're here for. But anyways, would you bring us home here with with what you your main takeaways here and the rest of your analysis?
Okay. Yeah. So, you know, intermediate product failed like 31 percent of the time, but in reality, it only fails like two percent.
And but it only makes up seven percent of the data. So I think the you know,
the reason that the end product, you know, you get the best you get the best predictions from it is that it makes up the majority of the data.
So the classifier has enough data to learn something about it, and it doesn't have enough data about the other products to learn anything or to learn enough to make accurate predictions.
I did spend most of the week working on trying to come up with a balanced data set.
And I tried several different things with oversampling and even breaking the data up into passing and failing.
And then oversampling that. And but what I'm finding, you know, I'm kind of finding this.
I also found out that found the same kind of thing with the hemp data was that I'm not sure that there's anything there to predict.
Right. These could just be sort of random occurrences or.
Yeah, I mean, I don't know that there's actually enough if there's there's anything that's really predictable happening.
I can kind of keep going down this path a little bit further, but.
Yes, I'm not. Go ahead.
I think, well, two things. So you definitely could definitely could be right. So we may have to think about another dimension of the data that's worth.
That we could get some good insights from.
My question is, my recommendation would be maybe even put prediction aside, just look at some conditional averages for the time being, but I would drill down a little a little further.
So essentially what I would do is I would only look at end products.
So let's take a sample type. So, well, like a cluster of sample types. So I would say concentrates and I've got the list pulled up here.
So concentrates.
Could be hydrocarbon concentrates, concentrates for inhalation, non solvent based concentrates.
I would leave the mixes out.
And then you've got CO2 concentrate.
And then you may or may not want to include the food grade solvent concentrates.
Or and the non solvent based concentrates, right, because you would to the non solvent based concentrate. So there's going to be like what they call bubble hash, which you would use essentially water as your solvent.
So you wouldn't expect there to be butane in those concentrates.
So, long story short, you may want to pick a cluster of concentrates.
And then I would look or just just do one like just look at hydrocarbon concentrates. So just look at end product hydrocarbon concentrates.
What's the probability of sailing for residual solvents and then potentially for the different residual solvents.
And I think once you get that granular, I think that could actually provide useful insight as simply a conditional average to a process.
So then you would just tell processors, hey, historically, hydrocarbon concentrates have an X percent chance of sailing for residual solvents.
Similarly, you could do it with flour.
And there your principal variable would be your microbes.
So what's the chance if you grow flour, what's the probability your end product, your end flour, what's the probability that that flour will sail for microbial contaminants.
And it may be hard to predict, right, because those may be like one percent or less.
But I think just knowing what those percentages are could be useful for people planning their their cost structure, essentially.
Right. Okay.
You know, and I guess the other thing to take away from this is, you know, on this on the surface, this, you know, this classifier looked really good.
But after we dug into it, you know, it really wasn't much better than a dummy classifier.
So you should trust these things outright.
And that's how you could potentially do some robustness checks.
So just throw in a dummy classifier.
And so I think it's fruitful, interesting analysis, right, because.
It's better to do it than to not do it right. So I think we've learned a lot.
So it's.
You're OK, Charles. I'll go ahead and start presenting.
And that way we can see some of these variables that we've been talking about.
OK. So.
Just to show you what I've been looking at over here. So these are essentially the intermediate types that I've identified.
Just to kind of give you a bit of a background before we just dive straight into that.
So. Essentially, right, we're working with this Washington state data.
Primarily looking at the cannabinoids.
Well, the lab results in general, not necessarily the cannabinoids.
And so. I essentially wanted to do a quick analysis here.
And so. There's been a lot of talk about Delta 8 THC.
So I just wanted to take a quick look at Delta 8 THC.
And then I was just perusing the literature to see if what's the latest research being done.
Found an article here.
Let's see when was this published? Published not too long ago.
Oh, yeah. So it was just published July 10th.
And so essentially what they did was they just looked at edibles in Jamaica.
And so these.
Not necessarily.
From stores. So some of these were confiscated from from high school students.
So these were just. I think they only had around 50 or so.
So they basically had around 50 edibles that they came by.
And they were looking at the THC and CBD.
Found in the edibles. And so they're looking at the THC CBD ratio.
And so I thought, well, you know, they only have 50 edibles.
We probably have a lot more edibles in the lab results data set.
So why don't we try to to essentially replicate their analysis and see what we find with edibles here in Washington state.
So that's just a bit of background.
About where I'm coming from, just to build on Charles's analysis.
OK.
So let's see if we can't.
Get a terminal opened here.
So it'll give me about 30 seconds to read in the data.
So we're working with these lab results from Washington state.
So we've got about two gigabytes of.
Lab results here and just for a refresher, this is the same data set that we've been working with for some time now.
And then I'll be working to get an updated copy of this data set to the most recent time period.
So that way we can refresh our analysis.
OK, looks like we've read in the data here. So first things first.
Just want to look at the data.
So.
We have almost two million observations.
To go ahead and show you.
About the sample types here. So first off.
Just to show you the data points we're working with.
There's a lot of.
Analytes, so pesticides, solvents.
Microbes, terpenes, well there's not actual terpenes, cannabinoids.
OK, so the main identifiers are essentially type.
And intermediate type and to show you the guidebook links up here.
OK, so returning to our type discussion.
So essentially there's several broad types.
So.
The end products are what end up on the shelves.
Intermediate products could potentially be processed into other.
End products or become end products themselves.
So.
What I find most useful for determining what the sample is.
Is essentially its intermediate type.
And so this is technically the subcategory of inventory type.
And so it's conditional on the type.
So.
This is where we start to lose a little bit of the rhyme and reason.
So for example.
The concentrates are always your intermediate types or always your intermediate products.
Your edibles are always.
End products it looks like.
So it's a little bit of a mess, but essentially if you just look at.
Just the intermediate types.
It gives you a decent understanding of what the product actually is.
For example.
If you just look at the intermediate types for hydrocarbon concentrates, you'll just get hydrocarbons.
Or it's a little bit tricky. I was looking at the data this morning and it doesn't appear at first glance like there's a big distinction between flower and flowered lots.
I've got a sneaking suspicion that the flower is intended for processing and flowered lots is intended directly for sale.
In my in my analysis here, I just combined flower and flower lots and just called that flower data.
And so for now, I'll just be looking at flower data.
So flower lots is.
An intermediate type for end product and flower is an intermediate type for marijuana.
If you look at my version one of that notebook, it kind of it breaks that down and breaks it down by percentage.
And actually, flower is the only subtype for.
For marijuana. There's a bunch of others listed, but.
Flowers like at 100%.
Exactly. And so maybe even be worth asking the LCB themselves.
But essentially, I've got a sneaking suspicion.
One is like one is designated for stores and the other could potentially end up at stores or could potentially end up at a processor.
It looks a lot like they treat.
The two types the same. So.
Once again, just sort of deferring to what it brought up in the last meetup where this is not.
This is not a bulletproof analysis here. So if you were doing.
You know, research on your own or, you know, commissioned research, you're going to want to do a lot more research and dive in and answer these questions. So.
For example, wouldn't hurt to just email the Washington State LCB and ask, OK, what exactly is the difference between flower and flower lots?
They may or may not have.
An answer, but.
For now, we're just doing expedient analysis, but it's worth reading up a bit more because, like I said, I'm not 100% certain.
And I need to become certain. So.
So, so long story short, if you have any insights, definitely let everybody know.
Just keep powering on here.
I'll just be looking at flower and flower lots will worry about the distinction leader and.
Out of the samples.
Out of the almost 2 million, there is about 239,000 flowered lots. So I think Charles, this is may have been what you were hitting at where.
There's a lot of data there that may not necessarily.
Have lab results. Is that the case? Is that what you're finding or.
There's lab results, but like the intermediate type isn't filled in.
Yes, and so.
That's it. That's interesting. So we need to admit and find more. So I wonder if.
Those are observations that were just missing an intermediate type, or if those are.
Other types of samples, so I think I think there's more discovery to be had here.
But just to keep powering through dot to get bogged down and so essentially I'll run through this analysis.
Prefacing it with the fact that we still need to find out a bit more about the variables. So this analysis should be repeated once we've learned a bit more about the data.
But just to demonstrate some of the techniques we can use.
So first research question, I was just curious about Delta 8 THC because.
This was actually had a discussion the other day at the.
Washington State Liquor and Cannabis Board deliberative dialogue. They were talking about Delta 8 and synthetic cannabinoids.
I was curious. Okay, so what does the presence of Delta 8 THC look like in an ordinary flower?
So out of the almost 240,000 flower samples, a little less than 2000 had Delta 8 THC.
If you look at the actual percentage, it's less than 1%. So about 0.75% of all the flower samples tested had Delta 8 THC.
So it appears to be a rare compound.
And essentially.
I've restricted.
The sample to exclude outliers because so, for example, let's just take a quick description.
So.
You'll notice, okay, the mean is around 1%. However, we've got an observation in there that's coded at 64% Delta 8 THC and.
I don't think that's the case, so it's a flowers sample, so I think there may be either miscoding or some sort of outliers in the data.
So I gave it a generous.
5% exclusion of outliers.
So I'm restricting it to the bottom 95 percentile.
And just to look at that data.
Here would be your distribution or your density.
Of Delta 8 THC and flower.
So.
And then this is actually of the flower that contains Delta 8 THC. So this chart would have a lot more zeros. If we included all the flower that didn't have Delta 8 THC.
So.
I think it's interesting to observe. So it just is this this rare compound and there could be more work to be done there to, for example, you know, what are some of the.
This could be a huge data dump.
But.
You know, what are some of the strains.
I thought there was a straining variable.
But anyways, what are some of the strains that, you know, are more common.
Maybe it's product.
Now to get the strain, you have to merge it with the batch.
And then and then merge and then merge that with the strain.
That's right. So you have to do a bit of leg work to attach those data points.
I think that could be that could be fruitful legwork. So, for example, what are the sample types that have Delta 8 THC because as we saw, there's only about 1800 of them.
And there could be duplicates, right? So there could be one strain that's grown many, many times. So I'm curious. OK, so what are these strains that are growing that are producing Delta 8 THC and.
There's also a surprisingly large number of entries that are missing the strain ID.
Exactly. So this could be where it's just essentially.
Not the best data entry, so.
Different licensees may not be consistent about how they're entering in strains and tight, although.
You know, the system, I'm sure, would like like things to be nice and consistent.
So.
That's the reality of working with, you know, with real data is.
It can be a mess, right?
So we need to be careful with the data points because I was hearing somebody say, oh, there's an acronym, Geigo, where it's garbage in garbage out. So.
You have to be real careful about, you know, what data points we're looking at here and so.
You know, you may have to suffer in this case, I'm excluding the outliers because I just personally don't believe that there's a sample that actually has 64% Delta 8 THC.
So.
That's the reality of the situation.
But just to keep moving on, unless you've got some more observations Charles or Brooke.
So.
Just do. I don't have any questions. Okay.
So that was just a little look I wanted to do it Delta 8 THC just because there's been a bit of noise about that.
Next, just to keep doing some cannabinoid analysis. I thought, okay, we could reproduce this histogram. This is simply a histogram of the THC CBD ratio of the solid edibles in Jamaica.
A sample of solid edibles in Jamaica.
So we can do that in Washington State.
So.
Going back to the sample types.
You'll see that there are a handful of edible classifications.
There's topicals, there's capsules, there's solid edibles, there's tinctures, there's cooking mediums, there's liquid edibles.
There's transdermal patches. So there's, there's a handful of non-traditional cannabis products.
Just for the sake of simplicity.
I'm just doing analysis here on solid edibles.
Because I believe if you look at the breakdown of the edibles that they are looking at in Jamaica, you'll see they've got baked goods and candies.
They do have some beverages. So we may want to include liquid edibles.
However, for the most part, it looks like baked goods, chocolates.
So in our data, solid edible is 37% of the intermediate types for end product.
And liquid edible is only 10%.
So honestly, let's do this now. We've got time. So let's do this analysis real quick with solid edibles, liquid edibles, and then solid and liquid edibles.
So.
Just to do it real quick, just with solid edibles.
So I saw that there were about 9000 solid edible observations.
And so this is what's so cool about our analysis here in the Cannabis Data Science Meetup Group is this paper.
Right, so.
Right.
These authors.
Have their article published in the Journal of Cannabis Research.
And it is incredibly interesting because we want to know about the cannabis in Jamaica.
However, got to keep in mind that.
Right, actually they don't even have 50s. They've got 45.
Edibles collected over a four year period. And so.
I mean, for starters, you could argue that this data is already dated. I mean, it's 2021.
So is it even relevant to compare edibles in 2021 to edibles in 2015?
Maybe, you know, maybe not. And so I think that's what's cool about our analysis here is.
You know, a lot of the times your access to data is limited, right? So they're just looking at 45 samples and.
You know, they're good.
That's fruitful, right? Because you'll you still get a breakdown of the THC-CBD ratio.
We can collect data in Washington State. It's public. It's free. Anybody can collect it.
We've collected it and we have 9000 solid edible samples between.
And so we can even find our date range here. So.
I should have just done this on.
Wonder if I can.
Well, luckily I have this over here too.
OK.