Sorry about that.
Yeah.
Well, welcome everyone to the cannabis data science meet up group.
This is a group where we meet at least once a week to talk about cannabis data and how we can apply data analytics.
Here's a guest, Heather, who joins us regularly.
And so my name is Keegan.
I founded a company, Canlytics, and we're here to make cannabis testing simple and easy.
And while doing so, we can provide rich analytics to everyone in the industry.
So that's my comparative advantage.
So I'm always just here to share.
And I'll let Charles, would you mind introducing yourself for a quick?
Hi, I'm Charles.
I have 27 years of software engineering experience, and I've been transitioning into the data science area.
I have experience with TensorFlow and machine learning, genetic algorithms.
And this has been a great group to get some practice and work with some interesting data.
Definitely. And then if everyone, if they want to, if you want to go, I'll go around.
If you want to share a bit about yourself, feel free to share in, jump in.
So I guess next is Alan.
If you're interested, would you like to introduce yourself?
Yeah, hi. So I'm Alan, and I'm transitioning into the field of data science from being a science educator for 15 years and graduated from the Galvanized Data Science Bootcamp.
And yeah, working on improving my skills and trying to get a job.
Excellent. We have a pretty regular guest who is on vacation this week or be back next week, Paul.
And he's also getting a degree in data science.
And so what we found is the cannabis data is some of the, you know, some of the richest interesting data sets out there,
because, you know, especially with the traceability systems, you have such granular, high frequency data.
So it's fun to do analysis on regardless of your background.
Matt, would you, if you're interested, would you want to introduce yourself real quick?
Hi, everyone. I'm Matt. And I work with ERP system called Plex.
And I write reports using Intel Plex and SQL development.
So I'm really good at SQL and I'm interested in this group to access the APIs to write, I guess, RESTful services to access the RESTful data
and to manipulate it like in Tableau or something else.
That's it. Thank you. Awesome. Awesome.
You're in the right place. So we're we've touched on APIs.
So I put up a couple of videos on YouTube where we go through some API work.
But Canlytics has put together an API primarily for labs to use, but anyone can use it to store and access their lab results.
And then Canlytics also has a module to interface with metric and leaf data systems,
both of which have an API that serves the traceability systems in their respective states.
So you're in the right place. OK, good to hear. Thank you.
Awesome. Kelly, were you interested in introducing yourself real quick?
Yeah, I don't think I'm in the right place, actually.
I'm Kelly and I'm using CBD right now.
So I was curious. But you guys probably are going to roll right over my head.
Well, well, you may want to stick around and listen, because one of the things I put together today is so first,
we may look at some work Charles has done just predicting when a sample would fail quality assurance testing.
So that affects you. That's when products make it to the shelf.
And then the second thing is we're going to be looking at hemp testing data.
So people who are growing hemp, they need to get it tested to make sure that it doesn't have high levels of THC,
but then it predominantly has CBD. So they need to get that tested before it makes it to the shelf for you.
So we're sort of doing analysis upstream from from your products to make sure that what you get
is what it says it is on the label. That's the idea.
Yeah. OK.
And then, Heather, are you interested in introducing yourself or?
Yes, sure. I'm Heather. I've been joining Cannabis Data Science for a couple of weeks now.
I'm going to confess to you that I don't have data to share and no code to share.
I am currently looking for a job. My goal is at some point soon to be working in a cannabis lab, testing something.
I would I'm trying very hard to bring my QC experience to to light in the industry.
So anyway, I'm observing a lot and I have a lot of questions, but I'll reserve those after the presentation.
So thanks. Well, awesome, Heather. I may have to forward you some some job openings
because there's lots of laboratories that are looking for for people right now.
So thank you. You're in the right place.
And you're you're welcome to to learn from us.
Use it. Use data and use any snippets of code that you may find useful.
I question data, too. So you're talking about the quality assurance and the levels of anything that's tested.
I'm finding that at least with some of the products. So I live in Maryland, by the way.
So some of the products that I've been using over the last year, their tests, the terpene levels have changed.
Yet it's hitting the same. So and there's a lot of discussion on Reddit.
People complaining about the turp levels not reaching above three percent.
That's a very local issue. But so anyway, testing is an issue for somebody that makes homemade oils and everything.
We're interested in what we're putting out to our even ourselves, our local market.
What if you want to think about it that way? So numbers do interest me.
And why they they change without a apparent physical change is something of concern.
So I like to question data when I can, but that's not just for, you know, for entertainment value.
It's for the sake of the product, the quality and what we're using it for.
So, you know, therapy, pain management, anything that you need.
We do want to know what's in it and and why. So thank you.
You've got an inquisitive mind. So you're in the right place.
Thank you. I feel welcome. Thank you.
Awesome. Welcome, David. Real quick, we were just doing a round of introductions.
You would actually happen to be the last. If you're interested, are you would you mind introducing yourself real quick or we can kick it off either way?
Yeah, no problem. Sorry, I'm actually at the gym, but I wanted to definitely not miss this.
Yeah. So I recently got into data science, have a background in mainly business, project management.
And I also have a project upcoming that I want to do a capstone project on just finding different data sets and then showing the skills that we've kind of recently went over, which is Python, Tableau, SQL.
And so this is very appealing to me. So I wanted to hear a little bit more about it.
I've also been interested in part time investing the stock market.
So I figured it would be a great way to just get a little bit more solid foundation, get enough, you know, just become more educated in the space.
Awesome. Well, welcome, David. And so welcome, everybody.
And so let's just go ahead and kick it off. And so for starters, I think I can present unless you want to present Charles.
But I think if you're open, we can start showing some of the work that Charles has done predicting the probability that a sample will fail quality assurance testing.
So when the sample sent in, depending on its attributes, you know, what's the chance that it's going to fail?
So can you can you start to predict that ahead of time, essentially?
Do you want to present Charles or should I?
You can present. OK, awesome.
OK, so I, you know, I looked at like, how could you tell using the Washington data if a sample would fail?
And so in order to do this, you have to you know, you have you know, you have data that's from the past, you know, like the type and the date it was test or well,
the type and the lab ID and the producer ID.
You have also data from the future from after it was tested.
So you can't use any of that because you can't peek into the future to make a prediction.
So so I just read those things and, you know, and tried to use those.
So, yeah, I mean, there's a there's a lot of data there from the future.
But so, yeah, so I read that stuff in and then you can keep scrolling down to.
Yeah, and I went through and I kind of checked and got rid of data that was missing data or tried to fill in data.
And so the one thing that you have in that data is the product type, which you know ahead of time.
And I looked at each type of each of the product types and there's actually less than one percent failure for each one.
I mean, the really, really small value.
So this is going to be kind of, you know, so kind of difficult to to find.
And then also, there's also it's around a one percent failure rate.
So you have a lot of passing values, but not a lot of failures.
Yeah. So so that makes it makes it difficult.
And I went through so there's a type and there's an intermediate type.
And I tried to go through intermediate types to see if that would be useful.
So, sorry, but there's a lot of missing values and it was very, you know, there wasn't it wasn't a straightforward way to fill in the values.
And for some of and for some of the types, there was only one value.
So they would just be, you know, one to one correlation and it wouldn't really add the information.
So I ended up dropping the the the intermediate types.
Yes. And so it looks like the ones you want to focus on here are like the.
I think the flower lots are what's getting tested.
The flower may actually just be like wet flower that may be getting transferred for processing.
And need to double check on that. And then, of course, you want the breakdown of the concentrates.
So. But some of the, you know, it may not be worthwhile looking at some things like capsules and whatnot, but it depends on the data.
But it seems our biggest challenge here is just the prevalence of of zeros.
Right. So the prevalence of samples that pass make it hard to predict.
The black swan. And these numbers are percentages.
So they're really, you know, really small.
Or no, I guess it's no, it's normalized. But yeah, so but it is.
Are these failure rates right here or that's no.
Those are the number of samples that there are among the total sample.
So is this saying there's 50 percent of the samples are flowered lots?
Yeah. But 84 percent of the of the rows had no information.
There were no there was no intermediate type listed.
So it was kind of, you know, how do you fill that in?
And I couldn't come up with a good way to fill in that missing data.
OK, that's a that's a big that's a large percentage.
So did they those have lab results that 84 percent? Yes.
Did they have a product type?
Like they were sure they were just all in products, I'm thinking.
Yeah, there was some sort of there was a product type there, but there was no intermediate type.
Well, it's a chance they just weren't in products, but.
But it could be just bad.
It could be missing data. So.
Right. And there was just no way to figure out, you know, a good way to fill it in without just.
You know, without sort of corrupting it more.
So I just didn't use the intermediate type.
I think the intermediate type would have really been helpful.
But it was there was, you know, there wasn't a clear path forward on how to fill in that missing data.
So instead, did you go with.
So I just use the product type because that I had, you know, that was all there was a product type for filled in for almost all the rows.
And so I didn't have to drop that much data for that.
Right. And so for like marijuana, there was flour was the only intermediate type.
So if you had marijuana and flour, right, the flour didn't didn't add any more information to the marijuana.
Right. It was just it was one to one.
So you it was, you know, it's just it was just going to be an extra feature to use.
And it just would have made it more complicated for the learner because it had this extra feature, but the feature didn't add anything.
Yes. So I think we may just need to narrow our analysis down.
And that may be what we get to today with the hemp analysis is so.
It may be worthwhile to just look at the failure rate for flour, right, because.
Well, the other ones matter as well.
But ultimately, you need the clean flour before you process it.
Of course, I think you can, you know, just just keep distilling out the contaminants.
However. We just may need to narrow our analysis just to.
It gets a meaningful. Conclusions, I think.
But keep keep watching me do this. So, OK.
When you do a classifier, you know, you you start out with a dummy classifier.
And the dummy classifier doesn't care about what the inputs are, it just outputs something.
So I came up with a dummy classifier and then because there are only a limited number of product types, I one hot and coded them.
So because you can't put you can't you have to use a numeric value, you can't use the text value.
So what that does is it it takes the value and it adds a column for each for each type and then puts either a zero or one in that column, depending on if it's that type.
So you can see, like the first one, it's harvest material. So there's a one and all the other values are zero.
And so it's the next one's intermediate type.
And so this is a way for the classifier to you to be able to use this categorical data.
So and then I label encoded the status because, again, the status was pass and fail,
but it actually has to be there's your it has to be zero and one in order for the classifier to understand it.
So and then you split the data into a training set, the testing set.
So I held out 20 percent of the data for testing and then I trained a dummy classifier that output the most frequent value, which is pass.
So it just no matter what you give it, it it it you know, it outputs pass.
So then when you look at it, you have to figure out, like, what is a good metric to use to to judge your classifier, if your classifier is good or not.
So if you look at the recall, you know, it's it's ninety nine point two five percent. You know, I mean, that's that looks like it's a great classifier.
But, you know, it it and the confusion matrix down here, you can see it got all the passes right.
But it got all the failures wrong. So that's not really good.
So some of these macro recall and balanced accuracy are 50 percent, which is kind of really what it is.
Right. It got 50 percent of what you're trying to determine. Correct.
So basically, the training data is almost all passes.
And so it basically the algorithm just predicts pass every time, essentially.
Well, that's what that's what it's supposed to do. Right.
If you have. Yeah, this is just this is a baseline.
Like, so you should be able to train a classifier that outperforms the dummy classifier.
Right. The dummy classifier just it only outputs pass or this particular version of it.
And so this is just, you know, this is just to see if, you know, what your if your classifier is the one that the real classifier actually works.
And so I did a second dummy classifier that output distribution is stratified based on like.
You know, it it does a percentage based on what the actual the actual training data, what the what it outputs.
So it doesn't it didn't really do much better.
Right. It comes up with an again, it comes up with a 98 percent recall.
It's just a little bit, but it still does really poorly.
And so is there sort of a problem where.
It's going to start predict well, and then this is actually one of the things that you need to keep in mind.
So one of the rules of forecasting is know the forecasting error.
So, you know, what's what's the bigger problem is predicting a pass when it's actually a fail or predicting a fail and it's actually a pass.
So what's the cost of being wrong?
So. Right.
But again, this is this is a baseline.
This is what we're trying to outperform.
But, yeah, so and again, it's you know, it's about 50 percent right.
So because this is an imbalance, you know, in balanced data set, you can you can you can calculate these weights to help influence the output.
And so it'll sort of weight the failures, the weight, or it'll it'll boost the weights that predict the failures above the ones that predict passing.
So I calculated those and then I use a logistic regressor to come up with it and to do to do an analysis again, only using the type, the product type.
And you can see that it actually does much better at least at predicting failures.
It gets 96 percent of them right.
But then the passing ones drop down to 76 percent and you get 24 percent of the of values that should have passed predicted failures.
OK, so. OK, so with this model, it's only getting like.
Hold on, walk me through this real quick.
So it's getting 75 percent of the path like it's calling a pass a pass correctly, 75 percent.
So that's a true pass.
Yes. And it's getting 90, actually really 96 percent of the failures.
Correct. Wow.
So into those are it's accurate.
So that's saying a fail when it is a true fail.
Right. And so this balanced accuracy, right.
It's about 86 percent.
So right, because that's that takes the average of the path of the pass and the failures, the correct passes and correct failures.
And, you know, and that's about where it should be.
If you look at if you take those two values and divide Adam and divide by two, you that's it's about 86 percent correct.
So. And then what's the percentage here where it's it's labeling it a pass when it should when it's actually a fail?
That's that sort of purplish dark.
OK, that's what I thought. Yeah. And so that's.
Is that four percent or? Yeah.
OK, so that's so that's what that's the square you're really worried about, where the algorithm predicts a pass when it's actually a fail,
because, you know, if the algorithm predicts a fail and it actually passes, you know, it's not.
Basically, so the idea is you want to test the failures like as quick as possible.
That way you can do all your quality assurance tests to make you know that the laboratory does to make sure that it is, in fact, a fail.
So. Predicting that it's going to be a fail and it's actually a pass may kind of increase their costs a little bit.
But what you really want to avoid is saying it's a pass when it's actually a fail.
So that's so to me, this looks like an outstanding fit.
Is this way it looks like to you?
It does on the surface and and if you go down, I actually like I'm actually able to improve on this.
So I took in the the lab and the producer.
And added those as features. Interesting.
Because, again, those are things that you know ahead of time. Right.
There aren't a lot of things that you know ahead of time.
So then I used the Cat Boost classifier, which is a better quality classifier.
And I do that only on I but I used only product type.
OK, so can I ask what are all the factors we're using in prediction at this at this moment?
So are we just using these three variables to predict failure?
Up to this point, we've only been using the product type.
Which I find really weird that that like somehow is that accurate, but.
Because it seems like it would just be kind of.
It seems like it would be random. Well, so once we'll have to test it in practice, I think.
So what's what what happens when you take into consideration lab plus producer?
OK, well, OK, so first I use the Cat Boost classifier with just a product type so we could compare it to the previous classifier.
So if you scroll down some more, it. Let's see.
So, yeah, that. So, yeah, it it's slightly.
It's actually about the same. It's very close to the same.
I think it gets more of the passes correct.
And then when I use the lab type in the producer.
Or the lab in the in the the type the product at the lab in the producer, I got this.
I got 99% of the failures correct.
79% of the passing correct. It's down to what point zero zero five.
For predicting incorrectly predicting a failure or predicting failure is a pass.
So this seems really good.
But if you scroll down a little bit, you know, this is this is.
These are the actual numbers. So it, you know, it correct.
It gets almost all the failures correct. It only it only predicts 17.
Failures is passing, but it predicts eighty three thousand five hundred and fifty.
Passing values is failing. Which is huge.
So. So I just had a question to help orient myself and the the total amount of failures is
the two thousand nine hundred and fifty one and the 17, right?
Like those are the actual true failures. Yes. OK, cool. Thank you.
So I guess the amount of.
I guess what do you call it?
It is false. False positives, I guess that would be
depending on if you want to call it failure or positive.
So, yeah, I think in this case we are.
So I guess if you just put all of those ones that you predict as failure,
but they're actually passes on the assembly line, it may still.
It may I think it would still increase efficiencies, but you're right.
It would still kind of. You know, basically the lower you you can decrease that,
the more efficient you are. And then, you know, of course, the lower you are,
you have this number in this third quadrant, sort of the safer you are.
So that's that's the way I look at it. So if you lower this number, you're going to be safer.
And if you lower this number, you're going to be more efficient.
And of course, you want to maximize these two quadrants.
You're when you're predicting fail, fail predicting pass, pass.
Right. Yeah, I mean, there's a limited amount of information there,
and it seems it seems really odd to me that those combination of features,
you know, is able to predict this accurately. But, you know, it.
That's what it seems to be doing now, what it's actually doing underneath, I don't know.
So and the and the interest, the really interesting thing about this was that you can
you can ask the model what features were the most important in it, making its decision.
And product type is like what about 24 percent.
The lab ID was 74 percent and the producer was like two percent.
So. Well, I think that may be a problem.
Because the lab ID would tell it if it fails.
So I think you need to exclude the lab ID from the analysis, right?
Because that's that's why it's almost like perfect, because it's just.
Certain lab IDs just should always fail, right?
No, the laboratory doing the testing.
Oh, like the I see what you're saying, like the the laboratory license ID. Yeah.
So, I mean, and this sort of brings up the point, you know, do certain labs fail,
you know, are they more stringent in their methodologies and they fail more product
or do they just tend to get product, you know, from, you know, that that fails more often?
I mean, it's kind of and it's interesting that it's actually not.
You know, it's not really closely correlated to the producer.
So, I don't know, this brings up a lot of questions and it's this is kind of as good as I could get.
I actually ran the last classifier through Optuna, which is a.
This is like a genetic algorithm and it goes through and it keeps running tests.
It keeps it keeps trying the classifier with different parameters to optimize it.
It ran for like about 12 hours and I got that tiny little bit of improvements.
So I think this is fantastic.
And so I'm curious.
It makes me curious because it seems to me like an exceptional fit with very limited predicting factors.
So. So we could potentially you could potentially try to cut your teeth on with the same.
Same models is so just a if you're OK with slightly changing gears, but we're we're actually going to keep this whole analysis in mind here for a second.
Because basically what I want to show you is.
There's this hemp database, so still cannabis.
However, hemp, you know, they're actually shooting for low THC levels.
And so there's an interesting database here.
But out by the University of Illinois, where they basically have cultivators submit.
Some information about the strains they're growing.
So essentially how they planted it, when they harvested it.
And then they are getting it tested.
And, you know, several of these laboratories, ACT Laboratories, Pride, Rock River and perhaps a couple more here.
And so. We actually have nice data on.
The. So here we have just the essentially the strain, the cultivar.
The source, I'm not 100 percent certain if this is the producer or perhaps the seed company.
I've got a feeling it's the producer. You also have the state.
So this is for the Midwest. So you primarily have.
Illinois, Wisconsin, Indiana and Michigan, I believe.
And so then you also have the county. And then what I think is going to be an interesting predicting factor is the sample date.
So here they're measuring total THC and total CBD.
Well, they're doing CBG and CBD, it looks like.
So hemp producers, you know, they want a high level of CBD rate like this 13 percent CBD.
That's outstanding. And so that's what producers are trying to produce.
Right. Because they. For example, Kelly.
They need to process this into hemp for for people to buy on the shelves.
So the higher percentage the plant is, the more efficient your extraction is going to be.
However, watch out, because look, the average unit, this actually has a zero point four percent THC.
And so they would actually have to destroy this lot of hemp.
So even though that lot of hemp has 13 percent CBD, it's it breaks the federal.
Limit for hemp, which is zero point three percent.
And so that would actually be a failure rate for hemp.
And so what you see is the failure rate for hemp.
You know, that's going to be a lot higher than the failure rate we observed with quality assurance of recreational cannabis in Washington.
So, Charles, I was thinking your model may be useful for using these some of these factors, such as cultivar source.
I was going to perhaps look at state today with you to see if.
What state the cannabis is produced in may have an effect on its failure rate.
And basically, we want to see, OK, what's the chance of it?
The cannabis, the hemp being less than zero point three percent.
So just to go ahead and show you this data, I've gone ahead and scraped it into.
Just an Excel sheet.
You're welcome to as well.
Basically, I just copied and pasted this 100 observations at a time.
If you think of a better way to extract this data, then definitely let me know.
But just sort of brute force collected it.
And so this will actually be the first time that I'll have done analysis on this.
So we can actually do do it live.
And essentially, we may need to conclude next week.
But basically, the idea is we want to first calculate the if the sample passed or failed
based on the total THC levels.
And then we essentially want to try to predict, say, the failure rate given some of the factors.
So for today, I was going to start looking at state.
So we're just going to do this live real quick.
So well, I suppose I can run it here in VS code.
And so first things first, let's just read in the data.
And just start doing a little exploration here with some of the time we have.
But any thoughts so far, Charles, if this is data that you may be able to work with?
Yeah, actually, there's actually more features that are known in advance with this.
So I can definitely try this.
OK, so it looks like we have our.
So I may move over to Spider here in a second, but we'll start with VS code.
OK, so we've got our variables.
We'll actually here first, we actually want to read in the correct sheet.
OK, so.
So I went ahead and collected several of the little cut that all of the data points here,
really.
So you may want to take a look at this, Charles and everyone else as well.
But I didn't see if there was a quite a one to one relationship here.
So basically, you see there's two hundred and fifty three.
What they call cultivar entries.
And so this says the harvest date.
And you see, for example, there's three and then when you go and look.
At like the summary table.
You see there were OK, there were three abbeys.
And then this just has their average.
So this is trying to uncover like, OK, do specific strains have sort of expected CBD
and THC levels to essentially hemp producers are trying to.
Sort of narrow it down and sort of settle on some stock strains that they can rely on,
because as you can see, people are having a problem with failing for THC levels.
Here is the more granular data.
You can see the three abbeys and you can see the three different tests.
However, there's does not there's seven hundred and fifty three test data points.
So I'm not certain how these map to.
The the cultivar entries.
I've got a suspicion that perhaps.
Say a farmer planted a field of.
Berry blossoms.
Or what common strain you see is this cherry wine, cherry blossom.
So if a farmer grew a field of that, they may they may sample it multiple times.
So we still need to get to the bottom of of this data.
And because it would be nice to be able to combine.
The cannabinoid data with the the harvest date.
Second the next best thing to the harvest date we have is just the sample date.
And basically to show you some of the figures that they've put together that will be recreating
is essentially here is. The plot of total THC to total CBD.
So as we mentioned earlier, the producers, they want to maximize their CBD while staying
below the federal zero point three percent threshold.
And so. Would I notice that's interesting that I actually just noticed this looking
at this this time is it looks like there is a slight positive correlation here.
So that would make sense.
Generally the plants that are producing higher cannabinoids, higher CBD also produce higher
THC. And so you're sort of playing a dance here
where you're trying to grow the highest CBD plant you can.
To the point where you stay below the threshold. Because as you can see, I mean, it's looking
like a non negligible portion of the sample is is failing.
The one thing I've read is that and and the data here to do this analysis is here.
The time between the harvest and the testing. Apparently after you harvest hemp.
The THC level increases. And after a certain like people have have
tested their hemp and it's passed and then they're like transporting it and they get
pulled over and they test it and they fail because an amount of time has passed where
the THC level has risen. So there's sort of this race against the
clock. And you know what's happening is is it's these
CBG levels. So THC is a I'm not going to use the correct scientific
word here. It's a it's a derivative of CBG.
So CBG is a precursor element. So from or maybe CBGA.
I may be getting it wrong. But essentially they all all of the cannabinoids
essentially have a common precursor and depending on various factors such as you mentioned how
it's stored determines how the chemicals you know break down and oxidize into a different
you know how they break down and turn into different cannabinoids.
So so that's an interesting observation Charles. So
we may not be able to incorporate the CBG levels but
but what we could potentially even try to do is say say you're I don't know how we
would work at this into our analysis but say we could try to predict so say you grew in
abacus strain. Okay well given that you've grown an abacus
strain in Wisconsin and that the you know historic abacus has a you know is expected
CBG level of 0.24 you know given its you know historic averages you know you could almost
try to predict you know the probability of you know a sample failing.
So I think this would be incredibly helpful analytics to hemp producers because essentially
if you could give them like a chart because what I think would be cool would be to plot
a figure so like you were saying like to on the x-axis you would have number of days so
you could have like number of days between harvest and sample sample date.
So it would be interesting to see if the number of days in between there would affect the
probability of you failing of failing for high THC.
So that's the analysis we're going to start doing so like I said I'm just now starting
so you're going to see me basically just start hacking at this live right now but then
I think we can continue this next week and actually actually build a good model here
and actually make some some worthwhile predictions and produce some good statistics here.
So first let's just like I said the you know rule number one look at the data so let's
see if we can't do just that.
Okay so we've got 752 samples here.
The average is 5% CBD you've got the average THC is 0.23 so about 0.24% which is just squeaking
under the cutoff but as you see there's a standard deviation of 0.2 you know 0.2 so
you're going to have a large amount failing and so instead of just saying a large amount
let's actually calculate that real quick.
So let's just say okay why don't we just calculate the length of the data where and I may rename
these columns here in a second for ease of use.
So we want to locate these where the THC is greater than or equal to 0.3% and it looks
like we've got a column naming issue going on here.
Okay so for ease of use let's just rename some of these columns real quick
I'm just going to do it here just.
Thanks for bearing with me.
Now it's just going to be a bit easier to work with the data.
Now we can do this a bit quicker.
So let's find out all of the data where it's we've got 184 and so what's the percentage
well that's just going to be out of the total data set.
So Charles this may give you a bit better of a failure rate for your models because
here we've got you know almost 25% of the samples failing for high THC and that actually
sounds about right from my sort of from my historic experience and so that's it's actually
a major concern for hemp producers because historically I don't think every hemp producer
was getting their product tested for THC but it's because you now have to get your lots
tested and so you know cultivators are you know it's a major bummer right because you've
gone through your whole harvest you've done all of your work you've you've you've harvested
you've dried you've cured and you send it in and you you fail for high THC and the rules
say you now have to destroy your product so that could be a devastating hit to farmers
to cultivators who aren't expecting that so that's why it's important to to look at this
data so that way farmers can know ahead of time depending on what strain they're growing
or when they're harvesting you know what what's their probability of failing because as we've
noted you're sort of they're playing they're playing this this is a careful dance here
where they have to wait as long as they can to kind of get their CBD levels up but then
you know you need to harvest before your your THC becomes a problem okay so now we essentially
just want to to code here and if it's a pass or a fail so we'll just say fail
so how can we assign zero one here you may know off the top of your head Charles you
can use is it the label encoder scikit-learns label encoder it'll or you could do just like
a you could do a map you could do like a panda's map well then I think this may so I think
this may do the trick here but you know you're you're seeing essentially how about this but
basically we just want this is a zero or one where it's a one for failure and so we want
this to be a failure where if it's greater than or equal to 0.3 percent there we have
it okay okay so we're running near the end here so essentially what I'm going to do next
is essentially run a logistic regression of the failure rate because this is you know
zero or one yes and so this is where you can do some real interesting analysis what by
you know with some binary models and so we're going to use the list logistics regression
so basically like I said we may need to wait till next week so that way I can actually
write the model and have it ready ready to go for us but essentially I'll be using the
logit and then so that's our dependent variable and our independent variables they are we'll
want to think about how we can work with these and so right off the bat I was thinking okay
we can basically use dummy variables depending on what state they're in so real quick in
the last minute we'll talk about some of the factors that we'll use for next week's analysis
but basically we'll I'll be using state to start with so we essentially have Illinois's
we'll have to keep in to consideration that it looks like it's spelled in two different
ways here or it's got a space so we'll want to make sure to strip out that space but we
basically have Illinois Wisconsin Indiana and Michigan and so right off the bat I essentially
want to you know calculate the failure rates in these states right so for example you know
weights so so for example data
so you can start to you know right off the bat just look at conditional averages so if
you've been here before you'll know that you'll know that I'll say that you can find extraordinary
insights by just taking conditional averages so we'll start doing formal regressions next
week so next week I'll show you a logistic regression where we start to predict failure
rates but for now I'll let you start looking over this data I've published this to to
get hub so you should be able to find this data set here on get hub in the cannabis data
science repository so I'll let you start looking at this data yourself as good Bayesians until
next week and then that way you can establish your priors and then we can do our analysis
next week and like I said start looking at some conditional averages so for example just
this simple one right you've got a four percent higher failure rate in Indiana than in Michigan
interesting maybe that's insignificant maybe that's significant so next week we'll start
looking at some of the factors such as state and potentially try to use harvest date and
sample date we'll have to be creative on how we measure that so maybe we measure how many
days into the year it took them to harvest so think everybody think about that and we've
reached the end of the hour here so I'll let people get on with their day but are there
any questions comments concerns here at the end did you learn something David was it worthwhile
yeah absolutely very it's a shame I'm not in front of my computer because I would like
to try to duplicate some of the stuff but uh but yeah absolutely it's quite fascinating
I would like to find out this is just on my own grab some of that data and compare it
if our soil what's in the soil within each individual state or each individual area or
it's all good I'd like how you think because that's exactly how you get these brilliant
insights you take one data set and you combine it with another and so you had a brilliant
idea look at the soil so if you can get soil data based on Michigan based in Indiana Wisconsin
Illinois perhaps there is a factor in the soil that's affecting failure rates in the
different states so that's exactly how you make these brilliant insights is combining
data sets so I like how you think thank you awesome awesome guys and gal yeah Heather
so crew so it was awesome meeting with you all and talking about cannabis data and we've
got a good amount to dive into next week so next week I'll have the logistic regression
prepared Charles if you want to apply your prediction model to the hemp data I think
we've got some good some good insights we can uncover so thank you very much definitely
well it was awesome speaking with you all today and until next week have a productive
week thanks a lot have an awesome day