So is this your new place?
Well I am actually visiting Oklahoma City.
CannaCon is this week.
So tomorrow, Thursday, May 27th and then Friday May 28th there's CannaCon in Oklahoma.
So I thought it was a good opportunity to see how this market is developing, meet some
people in Oklahoma and try to talk with some laboratories here.
Good morning Jake.
Hey how's it going?
Welcome to the Canva Data Science Meetup Group.
Nice to meet you both.
So it's great to be here.
Well just to introduce myself, so my name is Keegan.
My background, I started working as an analyst at a cannabis testing laboratory and then
I found there were a lot of rooms for improvement.
So for example there's data entry that needed to be automated, certificates needed to be
created easy and then just a website and client portal.
So as a software developer I realized there's a lot of laboratories that need software solutions
so I launched CannaLytics, a company to help provide software solutions to laboratories
to make cannabis testing simple and easy.
Because I've got a lot of knowledge so I may as well share with you.
I guess I'll let Charles and yourself introduce yourselves.
Charles I think you were here first, go ahead if you want to.
So yeah I have like 27 years of software development experience and I'm moving into like data science
and machine learning now.
And this group has some really good data to explore and do some interesting things with.
And so I've been working on that, I've been working on some Kaggle competitions.
Oh cool, that's awesome.
How about yourself Jake, how did you hear about the group and...
So I guess it's kind of two factors.
So I've been interested in cannabis, I do like stock analytics data, like investing analytics.
And cannabis is not an industry I'm current with, it's one of two industries I'm really
looking at.
But I'm also just interested in data science generally, so like that piece.
And then the second thing is what I do most of my time is I'm building this data science
tool and I'm sort of trying to explore what industries it could be interesting in.
And we've had like one or two people that I've just met with recently who were doing
like analytics in the cannabis or at cannabis companies I believe.
And so I thought you know maybe this would be cool people to meet here in that space
as well.
That's awesome, if you're open to talk about it I would love to hear about the data science
tool you're building.
But if you want, if you guys want to try it out I could...
It's a...
If you want to...
Well honestly today was going to be a bit more of an open day, so I'm traveling to Oklahoma
City to visit Canacon.
So I haven't had as much time to prepare things this week as I normally do, so the whole point
of the group is to let everybody have a chance to share.
So yes, if you want, it would be...
Yeah why don't I... what I'll do here, I'll... this is a...
That'd be really cool actually.
Why don't I...
I'll send you both a link.
So I have a hosted version and a local version, but as I'm sure you're aware like downloading...
It's a Python tool, so like downloading Python packages is a brutal, brutal process.
So I'll send you both this posted version.
And if you want to...
You can just log in there.
Oh do you...
Are you still there?
I think you might have froze.
Sorry, I just had to get reconnected real quick.
Oh no worries.
So if you go to that link that I just sent, it's a Jupiter Hub link.
So if you essentially...
It's like a hosted Jupiter notebook essentially, or Jupiter lab really.
You have to put your email in to access it, but you're welcome to put in like a fake email
if you don't want to put in a real email.
I understand.
Oh sweet, yeah I can sort of guide you through.
Yes, so I'm just going to share my screen that way.
So just a heads up, so we were recording to that way people...
Oh yeah, no worries at all.
Totally cool.
If anything, if more people get to see it, that's cool.
How long have you been running this group?
So the data science group kicked off in February this year, so I think our first meetup was
February 24th or 26th.
Okay cool.
And...
Oh interesting, so...
So I'll give a little background.
So what it is, it's a visual...
It's supposed to be like a visual Python representation and sort of visualizing Python as a spreadsheet.
So we'd imagine everything you'll do in the spreadsheet will generate the equivalent Python
for you.
So if you run this cell right here, that first cell...
Yeah you can go from there.
That should do it.
You're going to have to put your email in again.
But I sort of just knew I could just do this to hopefully be able to get in touch with
people.
Just formally?
Yeah, yeah, you're welcome.
And we're not taking anything.
You can just put like NA here.
This is...
I kind of have this question here for like self-service users, but obviously since we're
on a call, you're welcome to do that.
Cool.
Oh sweet.
I didn't know you could do that.
So this is the tool.
So essentially...
So are you a Python user at all?
Oh definitely.
Both Charles and I love Python.
So I fell in love with Python five, six years ago.
And it's my weapon of choice.
And so I'm fascinated by this.
So I've done a little work.
So this looks almost like something you would build like with PyQt, but obviously it's running
here in the browser.
So honestly, I'm kind of curious how you built this, if that's something you're willing to
share, but it also looks like a cool tool.
Yeah I have a co-creator, who I made this with, especially my twin brother.
And so essentially it's a Python backend.
So let me actually, it's easier to explain once I show you a little bit of the functionality.
So let's import a data set really quick.
You can click the import button and then you can add files from your folder.
Because you're on the hosted version, you have to upload your file to the system first
and then you can import it into the sheet with that button we just clicked.
If you're on a local version, then you can just sort of, you're in your local files,
you can connect to whatever files you want.
But if you want to go back and click that button again, so it is add files from current
folder, that's just going to put in our sort of the demo data airport pets.
And I'll just show you, so let's do like, you can close that little window on the right
there.
The one that you're, the one you had the import window, you can close that.
This one?
No, the, if you scroll up.
Yeah, it works as import CSV.
You can close that.
And then now we, so essentially the gist of it is it's a spreadsheet and everything you
do in the spreadsheet is going to generate the equivalent code for you.
So if you filter, maybe, so apply, try applying a filter to a column.
Like is it possible to import our own data set here real quick?
Yeah, yeah, totally.
For sure.
Also all you have to, so one of two ways to do it.
One is you could, if you have it as like a CSV file, you could just upload it in here.
Okay, sweet.
So yeah.
Because essentially, you know, just put together some data here.
Yeah, totally, totally.
So I think the best way, did you click the upload button yet or are you just in your
local file?
Let me just get some data for you real quick.
Yeah, for sure.
No.
And then how do we upload data?
So yeah, so you want, yeah, exactly.
Click that button there, select your data.
And then now, so if we, let's rerun that top.
Oh, was that a, did you upload it as a CSV?
It was an Excel, does it need to be a CSV?
Yes.
I mean, it doesn't have to be, but if you want to do it sort of through the UI, it has
to be a CSV.
You can, the other thing you can do is you can import any data frame into the Mito just
by putting it into the argument to the Mito sheet dot sheet call.
So for example, if you're working in a notebook and you have a data frame you're working with
and you want to quickly call those in, you can do that.
Cool.
So now that top cell in the notebook where we call the Mito sheet, if you want to run
that again and then click the import button, you should be able to select your file.
But I think this is.
That's yeah, that's the wrong data.
But if you click, there you go.
You can close that.
It's a little tutorial.
Interesting.
So.
And if you want to build that original data.
Yeah, you can see this little drop down next to where it says airport pets at the bottom.
You can just click delete.
Because yes, we're all about, you know, working with actual data here.
So we just have a licensee data here.
I'm tired of doing the same demo data.
So this is great.
But so yeah, let me, so let me show you maybe like a few things.
And I just love to hear your thoughts or if you want to, if you have thoughts of how you
might want to play with that, that'd be cool.
For example, so let's do like a filter, for example.
So if we go to the top or if we go to the actual the sheet, and then if you click, so
each it's, you know, it's the same feel as a spreadsheet or it's, you know, aspiring
to be that.
So for every column, there's a little filter icon next to it.
So if you want to filter.
So like a good example here, yeah, we can add a filter.
See we have that NaN value.
This is kind of useful.
If you want one of the filter conditions is you can do, yeah, you can filter to a city
or you could filter like out null values by doing is not empty.
The last option.
You got rid of your null values.
And then if you scroll down to below the, yeah, let's see.
So this is code we're generating.
So that final step, step four, that is the equivalent code for dropping null values from
the column.
Yeah.
So we generate the equivalent code for each edit.
And it kind of goes back to why I made the tool, which is like, I think you mentioned
like if you was a five or six years ago, you started learning Python, like I started doing
Python maybe like two years ago.
And I just found that like memorizing the syntax and getting the syntax right all the time
was just like was such a maybe it just wasn't my skill set, but it was just such a learning
curve for me.
So I kind of wanted with my brother like make a way to make Python more accessible essentially.
Question, do we need mydosheader or is this just all on pandas?
So that's a good question.
So everything you have right there is pandas.
Where you will need mydosheader is one specific place.
And it's the reason we have that.
So if you scroll up, if you do so what you can actually you can do like spreadsheet formulas
in the sheet.
So like if you want to try adding a column, for example, you can just click add column
button in the toolbar at the top.
And then in there you could use a form do a formula that's like normal spreadsheet syntax.
So if you want to use like a concat function that you would do in Excel, you could do it
in there and sort of concat the two.
So just write concat and you'll see a suggestion pop up.
And then you can select the columns you want.
Ideally I'm trying to like let's say.
Oh, yeah, pretty soon.
I don't know why it wasn't working like that before.
Maybe if you try, if you close that menu on the side, I think it might make it a little
easier.
That A where it says AB, if you close that.
Yeah, but also, yeah, if you know the names of your columns, it will suggest it to you.
We use this thing called Q grid, which is like a no, it's our AG grid, which is like
a it's a JavaScript like table library essentially.
And they kind of help us like get a lot of these features out of the box, which is cool.
But yeah, so there's your.
If you wanted to put like.
And so, yeah, so it's actually kind of what you're doing right now is like formatting.
It's like one of the main uses I think people use this for and then you can rename it.
But then you to your question about like, can you just like copy this code into any
other environment?
You can't except for when you use something like.
Where do we have it?
Scroll down to the bottom.
Exactly.
But that function, it is pandas that we're using.
But we it's like, we had to use a complex combination of pandas to like, get that to
work.
So we sort of wrapped it in our own function.
But I thought I saw you opening spider.
Were you about to try and copy this into spider?
Yes, actually.
Yes, that'd be cool.
I've actually haven't done a ton of testing with that.
So it'd be interesting to see how that goes.
What I'm thinking is like the functions, you don't need the UI.
The UI won't pop up in spider or at least it won't work well.
But I'm thinking if you copy, if you also copy the Mito sheet package into spider, the
syntax might actually work still.
We haven't done a lot of testing on that.
I'll need to.
This your project.
Yes.
Are you looking to what are you looking to do in the local install?
I believe so, right?
Because I'll need Mito sheet to.
Yeah, you will.
So this is the thing.
It's the local install.
Yeah, it should.
It should work.
So you just it's so the thing, the way Mito works is a is a the back end and then there's
a front end, which is a JupyterLab extension.
But I think you probably if you actually don't think you'll have to even download the extension
part, just do the just install the package.
You should be fine.
So yeah, that'd be cool if you want to try that.
And I guess.
It is your principal audiences or data scientists.
I think it definitely is.
I think I'm I'm definitely doing a lot of thinking right now about like trying to specify
for like a more specific audience.
But yes, certainly right now, I think it's like data scientists who are like beginner
to intermediate at Python.
I'd say like the lowest level is like like a lot of people think like, oh, this would
be a good learning tool.
And I see that.
But I disagree because I think it would actually wouldn't it wouldn't let you like it wouldn't
make you become better at Python.
Like it would just make you be able to do some of the things we want to do in Python
more quickly.
So I think it's actually better for like someone who's like six months to like three years
experience with Python.
And they know what they want to do, and they know how it works, but they're just like they're
tired of constantly going to Stack Overflow or like Googling syntax.
It's just easier to do some things very quickly in here like a pivot table, for example, pivoting
in pandas is like a pretty common thing to do.
But I've talked to like even really advanced data scientists who just like never remember
the pandas pivot syntax, and they're constantly Googling that.
And so our tool is like, you know, quickly do your pivot table in our sheet.
And so one is like the ability to just generate the syntax more quickly.
And then two is just the visual aspect.
A lot of people just like, you know, looking at their data during their analysis more in
a more dynamic way than just like printing out a data frame.
Which I think is cool.
That's definitely one of the things I like about it.
Did the package install?
We're still installing here.
Yes the other thing is like it's installing Plotly, which is like for graphing, but like
you're not going to use graphing if you're not using the font.
So it's kind of annoying.
How about you Charles?
Do you have thoughts so far or?
No, this is really interesting.
This is, you know, you might be targeting people who use like, was it Power BI?
Yeah, I see that.
People who do data science stuff, but don't really know programming.
That would be, you know, I think that would be a good audience.
That's interesting.
Yeah.
What do you think about the cannabis space?
So I think one thing, one of my interests as a cannabis, obviously it's like, you know,
one from an investing standpoint and the two from like a standpoint of this tool maybe
is like a lot of places I've seen people enjoy the tool is like these like transitional industries
where it's like they're typically using things like Excel, more analog tools.
And because of like they have much larger data sizes, so they want to do more advanced
analyses now.
They're transitioning to Python.
So we I've seen that in like finance a little bit, like pharma, like bio research.
I'm seeing that.
Do you think that cannabis is a place where, you know, there is a push to get these like
non technical people more technical?
Well, I what I have heard is people do want a hold of their data and to be able to analyze
it.
So that's, you know, something that I'm not sure if it's their design or, you know, a
lot of people kind of are holding that hostage these days.
So it seems well, that's what I've heard from the industry.
So it seems like people are having a hard time getting their data out of whatever program
they have it in.
They just want to do statistics on their own.
I just kind of heard that just word of mouth yesterday.
So I think there is a demand for just people to explore their own data.
Right.
So I think people are becoming increasingly more sophisticated.
And so people want to get a hold of their data and look at it themselves.
Charles, in your work as a data scientist, do you think people are wanting to get a hold
of their data more and more these days?
Yeah, I think so.
I mean, I'm not involved in the cannabis industry.
So I can't speak to that, but there are a lot of people that do data analysis but don't
really know like Python or any sort of programming languages.
And again, they use tools like, you know, they use Excel, they use Power BI, they understand
how to use those things.
So this would be kind of good for people who aren't as technical but still do a lot of
data analysis.
So my other question is how large of a data set have you tried this on?
Yeah, that's a good question.
I'm confident that it will...
So the idea is that what you're doing is really in the sheet, you're editing a data frame.
So you're either passing a data frame as an argument and that will populate the sheet.
Or if you're passing in a CSV, the first thing it automatically does is turn your CSV into
a data frame.
So as long as you can fit the data frame...
You can fit...
As long as you can fit that set into a data frame, which I think that most of you have
done is like 15 million rows.
I'm not sure how many columns that was.
So it's more about obviously the area.
But then I think it's definitely usable and of course way faster than doing it in other
visual tools like Excel, for example.
Excel is just really slow.
But then there's also like...
It's actually a really whole space of the data science I'm not too familiar with.
It's like these people doing...
I think it's like PySpark where it's like they have too much data.
It's like they can't even fit in data frames.
It's this whole other dimension of data science that I really am not too experienced in.
So I think we're more focused on this...
I think the people who have these a bit more smaller data sets maybe are doing more exploratory
data analysis to start.
Smaller, meaning still 5 million rows potentially.
But we're definitely not there for huge productionized machine learning models for insurance companies
and stuff like that.
So we're just like crazy crazy, terabytes of data.
If you want, I just started to do this brief analysis on comparing Eastern and Western
Washington rainfall.
So that affects how cannabis yields.
So if you want, could I show you that brief analysis and see if that may be something
that could be recreated in your tool?
Would you want to try uploading that into the tool?
Whatever you want to do.
Yes, so here, why don't I run through it real quick and then we can try it with your program.
Sweet, that'd be awesome.
And that way we can try to replicate the results.
Just a brief overview.
Essentially we're just doing sort of a difference in difference.
So just kind of comparing one region to the next.
And so essentially we're just seeing, okay, it looks like there's quite a different climate
in these counties compared to these counties.
So over here it looks wet and incredibly wet.
And then over here it's much drier.
And so does that have any effect on cannabis yields?
Is everybody doing things inside and temperature and humidity controlled rooms?
And it may not have any effect whatsoever.
Or maybe people over here are having a hard time keeping their crops cool.
So we actually will have to do some statistics to see if there's any fundamental difference
in the cannabis yields.
So it's not actually that tricky of an analysis here.
It's pretty short really.
So essentially we're just going to use dummy variables.
So one, if you're in Eastern Washington, zero if you're in Western Washington.
Or I forget which.
Exactly.
So one, if you're in Eastern Washington, zero if you're in Western Washington.
And so the past few weeks we've been working with lab result data in Washington state as
well as earlier year in also what is the name of your program?
Hold on, MITO.
So in MITO we have already read in the licensees.
And so we're joining that data with the lab results.
Cool.
Okay so.
And you're reading this from an Excel file?
Oh that's the basis.
Did you say that one more time please?
You're reading this from an Excel file?
I think I saw a line in there that said read Excel.
Yes, I think I may have moved this file.
So one second and I am going to make a...
Let me pause this for one second and I'm going to...
Yeah no worries.
All right.
Hey.
Hold on.
I almost forgot Paul here.
Hey Paul.
Hey guys, how you doing?
Good.
Excellent.
So you actually joined in a good time because I'm sort of getting the data connected here.
So we've got Jake today who's building a data science tool and we're using it to explore
some of the cannabis data we're working with.
Oh very cool.
Nice to meet you Paul.
How you doing Jake?
Doing well.
Where are you located?
Southeast Michigan.
Where are you?
Cool.
I'm in...
Philadelphia.
I'm actually a bit Jersey Shore right now though.
Oh okay.
Okay.
So what we're doing Paul, so this is sort of merging both tools.
We're basically doing an analysis on Eastern versus Western Washington lab results.
So just a simple analysis just to demonstrate working with the data.
Is it kind of like an A-B test or something?
Essentially.
So this was just sort of an example of merging the data sets and just running a regression.
Right on.
But I'm just getting the data pulled up here real quick.
And while you're doing that, Charles thanks again for helping me out this week.
I appreciate it.
I haven't had a chance to get into the parquet files yet but I'm still trying to use PowerShell
to get some of the conversions over and it seems to be working.
It takes a long time though.
Yes.
It's converting it in pandas was...
It was slow going.
It was very slow going.
Pretty painful.
Yeah.
So that's interesting.
I was going to...
I thought about like...
I don't know a lot about R but I thought about trying to do some of this analysis in R and
thinking oh maybe that would be better but you're running into the same problems I'm
running into.
Yeah.
And after you're mentioning GOSH in Python, what is the name of the chunking application
you referred to?
Dask?
Dask.
Which I don't know anything about but within R there was a library that does essentially
actually is like three libraries.
I've tried two of them and those two didn't work but this other one seems to be kind of
promising where it just essentially takes the data and chunks it out into files and
kind of distributes them and you do some parallel processing on it.
So I'm going to give that a whirl and see what happens and yeah just test it out.
Cool.
Sorry about that.
I can focus now.
So that's good and I've read your emails or your messages that you're working to get the
UTF-16 converted to UTF-8.
So correct me if I'm wrong but I think you've got a solution.
Yeah it's essentially just using PowerShell and run a small script and it just chunks
through the whole file and then gives me a UTF-8 TSV file on the other side.
But then that's another script I can run that will convert the TSV to CSV and then I've
got the full CSV file and then what I've been doing is I'm in the process of doing this
right now is uploading them to Google Cloud with the free instance that I've got right
now and then just to get some of these larger files up there to see if this tests out the
process and then run some small queries against them.
That's what I'm trying to do.
Excellent work.
So just to show just a demonstration of this data to Jake and just some statistics that
I was beginning to tinker on.
So just always just start with the question.
So my question was does rainfall, so does like the amount of moisture affect the cannabinoid
results?
And so I kind of wanted to look at all of the different results maybe some fail more
for moisture content, maybe the water activity is higher on some than the others.
So the way Washington is set up, it works well for essentially just a quick and simple
dummy variable analysis.
So we're just going to get.
I think we just lost him.
Hey Keegan, I don't know if you can hear us, but you froze up.
At least it's a nice image to look at.
He's got his power fist going, so that's good.
Oh, there you are Keegan.
You came back.
My apologies.
So I actually should let you know Jake, so I am working at the Oklahoma City Library.
I'm in Oklahoma City for a CannaCon tomorrow and Friday.
So I'm going to be meeting people in the cannabis industry here, see what their challenges are,
how things are going, how the market is developing.
That's really cool.
So back to the work here, Eastern Western Washington.
What I'm doing here is just reading in the lab result data and the licensee data and
then merging it.
And so, for example.
So those are the variables for licensees and we can merge these with the licensees global
ID and then the lab result for MME ID.
So you can look at the data points, but that way we can merge them.
And then we have a lot of data points for this one lab result.
So now we have this is a lab result here.
It is with this piece of inventory.
So we could actually look up the inventory at this point and get even more data points.
For now, our analysis doesn't need those data points, but in the future, we'll want to just
get all the data points that we can together.
And then essentially what we've done here is just added on the licensees data.
So now we know, okay, this licensee is in Bellingham, Washington.
And they're in Whatcom County.
So now it's pretty easy just to create a dummy variable if they're in Eastern or Western
Washington.
Pandas has tricks for creating dummy variables, but I actually just wrote a quick script,
you know, just wrote a quick, you know, quick little loop here just to create that.
Can I jump in for a sec?
So I think what's interesting there is you wrote, so you wrote that little script there,
you know, it's an if statement essentially.
So what you would do in my tool is you would just use like, you would just make a new column
and then you could do an if statement in that column.
Just like if this, that, if x1, if y0, it's kind of interesting.
Yes.
So Paul, this is Jake's tool.
And so this is sort of combined as a user interface into a Python code generator.
Okay.
This would be...
That's called Mito sheet.
Yeah.
Here, I'll put the, I'll put some links in the chat.
So just for everybody's sake, I'll run through mine real quick, but then essentially I see
you would basically, you know, create your dummy variable here, sort of Excel style.
And then you'll have, you know, the code automatically generated.
Yeah, totally, totally.
Was it cool?
So we've got, we've got that dummy variable now.
And so, I mean, so this is actually interesting.
This is actually the first time I've calculated this statistic, but like, so this is an interesting
statistic that maybe not a lot of people even know.
So it looks like, you know, 62% of licensees are in Eastern Washington and, you know, you're
about 38% are in the other, you know, Eastern Washington.
And so what you could do is remember, I always say, well, I'm not sure if you've heard me
say this, but what I would say is you can get real interesting by just doing conditional,
conditional averages.
So if you just, just took the mean conditional on, okay, if it's a producer or processor,
you can now find out, okay, are there more producers in Western Washington or Eastern
Washington?
Because my, you know, hypothesis is that there are a lot of retailers in Eastern Washington
and a lot of cultivators in Western Washington.
And wonder, wonder what the implications would be on transportation for that.
Well, it depends on the way the laboratory, well, actually, well, there's a lot of factors.
So you're right.
So the people will end up, and that's the thing in Washington is essentially transporting
the cannabis from Western Washington to Eastern Washington, you know, first to test it at
a laboratory.
Here are some laboratories in Western Washington, and then to send it to all their retailers.
So just to understand the flow a little bit, because I'm new to this, so any grower has
to, there's some formality, they have to get it tested at a specific type of lab before
they can sell it.
They can't do any of that testing in-house.
Exactly.
So they need to get their products tested at a third party independent laboratory.
That's interesting.
I wonder why.
Yeah, that's interesting.
I think it's for, well, I'm just speculating.
I'm new like you, but I would think it's for overlapping interests, right?
Because you could probably cut a lot of corners.
If you're a transporter and a producer, I wonder if some of the checks and balances
would be kind of compromised by doing that.
Yeah, totally.
It was interesting if you could, like, if you could, there's a way you could, I don't
know, there's a way you could have it so you could do that testing in-house.
You could potentially save a lot of money.
I imagine it's the growers who are paying for that transportation.
Yeah, but they have a big problem with people trying to fake results.
Right, right.
Yeah, I mean, there's a lot of growers who are using a lot of pesticides and agents
to stop mold.
It wasn't a safe product to sell anymore.
So that's why you have independent labs to do that kind of stuff.
Also, we should actually be careful with the statistics that I was calculating because
I was only reading in 10,000 and there are about 2 million lab results.
So that is not even going to be close to accurate.
Yeah, you can do a random sample if you don't want to read it all in.
Quick question, Charles.
Do you know how to read in a CSV with just a random number of samples?
No.
Yeah, but so you may just have to hitch this, but this is sort of just a demo of it, an
analysis.
That's pretty cool.
I mean, we were talking about this before, just the fact that you're going through this
and answering some of those high level stats questions gives you that baseline of understanding
of what the market's like and how the market operates too.
So just having that information, you'll be way ahead of a lot of people.
Exactly.
So my red flag was, that's what you do when you're looking at data.
You've got to look at the data because there could be some red flags.
And so what my red flag was, was 100% of the cultivators are in Eastern Washington.
And so that's just because I just read in the first 10,000 observations.
One thing that comes to mind is you're showing me this, and we talked about this a little
bit last time, but like Washington and other states, they're in this kind of weird situation
where they're trying to grow an industry, right?
They're really trying to get it off the ground so they can get their tax money out of it.
But they're also trying to rein it in enough to where it's controlled and people aren't
getting sick from bad product or what have you, try to mitigate the risks.
So they're kind of this weird balancing act.
But at some point, I would imagine that they're going to have to do some auditing and compliance,
like pretty stringent auditing and compliance checks.
Or also checks for if there's any way to check for what's the word I'm looking for?
Shoot.
That will come to me in a second.
But to position yourself to where you can provide data that would help with auditing
and compliance would probably be a pretty good strategic move.
That's interesting because I think you're right, LEAF Data Systems is collecting all
this data just to look at it.
Rule number one about data, look at the data.
So you're right.
And it's one of those things where like I was saying, there's such a high demand for
this and such a low supply.
So LEAF Data Systems is probably trying to do their best to supply visualizations and
reports and statistics, but they may be underwater.
There may be such a high demand, they just can't fill it.
So you're right.
So that's where other people in the industry can kind of help out.
So I want to give a shout out to Jim McCray.
And so he is actually the person who originally provided this data set to us.
And that's essentially what he does.
He's sort of the canary in the coal mine.
So he, the canary in the data mine.
So he is digging around with Kim's data, essentially doing exactly what you're talking about, just
seeing if there's any oddities or being an extra eye on.
So who is Jim McCray?
Is he with the State of Washington?
I think he, I'm not 100% sure the organization he's associated with, but he's a researcher
in Washington.
And he's done a lot of research on the cannabis industry.
So you may be able to find him on LinkedIn.
That if you could honestly be the best way to get a hold of him, make him hear his LinkedIn.
I was just curious.
So yeah, so he's trying to, in a researchy kind of way, try and keep a pulse on the market
and keep his eye open for things like what compliance and that sort of thing.
Exactly.
So there's interesting things to look at.
So one is just the total amount of sales per lab result, because some products just sell
a lot and others don't.
And then you just want to keep an eye on failure rates just to make sure that there's not a
significant, where you just be aware of a significant trend one way or the other.
So for example, so just to give them a shout out.
So if you want a good example of someone who's sort of, someone keeping their finger on the
pulse, but they're doing it in California.
So they're just seeing, okay, how many things are being tested and they plot failure rates.
So you can see that failure rates are going down over time while the number of tests are
going up.
I think they may have, and this is, I think one of their best contributions is they actually
show the seasonality in, I believe this is seasonality in failures.
It may not necessarily be of cannabis.
Oh, actually, no, no, no, this is pesticide use.
So this is the analysis they've done that I think can be extended.
So in California, they record the amount of pesticides that are used over time and it
seems there's a cyclical use of pesticides.
I think it would be interesting to try to correlate pesticide use with failure rates
to see if somebody's using pesticides, does that affect the failure rate?
But enough of that tangent.
Sorry about that.
So extra eye on the data.
So we can actually use some statistics here to start answering some of these questions.
So back to our original question of the day, does rainfall have an effect on cannabinoid
results?
Hedge this, that this is not a random sample, this is just the first 10,000 lab results.
So you'd want to actually repeat this analysis with all of the lab results and then you'd
actually get a better estimate of the effect.
So hedge this, this is just a demonstration of the statistics.
It's not actually representative of what we think that your estimates are.
So Keegan, you pulled out like the first 10,000 results.
I was able to do the same sort of thing with the files that I have.
But to look across the entire data set, do you have the capacity to do that yourself
or are you kind of constrained on how much data you can work with?
I can probably pull it off.
I just haven't yet.
Were you thinking that you may be able to do this analysis or?
Well, right now, like I've been kind of complaining the last couple of meetings and leaning on
your guys's expertise, but I'm still trying just to get the entire data sets in a usable
kind of format and I'm getting there.
But as soon as I get it up to Google Cloud, I mean, running queries on this data is going
to be really easy.
Just running SQL queries on it.
Of course, how much is going to cost?
I don't know yet.
So I got to be careful of that.
But if I get those things up in place and it seems like they're intact, they're not
corrupt because I know that Charles, you're saying that like the inventories file you
found to be corrupted somehow.
So I'm just saying if I can get it up there and they're in place, I'll let you know.
And then maybe we can do something to kind of speed up some of your exploratory analysis
a little bit.
Definitely, definitely, definitely.
So how about this next week?
I can focus and then it seems that you're going to be focusing on this as well.
Let's get this data usable because that's the first step.
I think, you know, we've sort of mocked out how we can do some analysis.
But you know, we've got to do our YAC shaving.
And so we're basically stuck right here where we need to get all this data ready.
And Charles has come up with a couple of good solutions.
I just need to spend time and implement them.
So this next week, that's what I'll be focusing on is getting this data usable.
And then next week we can touch base again and I'll send you messages throughout the
week.
And that will be essentially the next step in our data analysis journey.
Just for this, just so Jake knows, I'm actually just finished up in a master's in data science
and I'm getting ready to do my capstone project.
So I kind of lucked out and hooked up with the guys here.
And I thought it would be kind of interesting to do a project around the cannabis industry
just because it's a novel data set.
And you can apply it like Keegan was saying the first time I was talking to him that because
it's a new data set, there's a really good low hanging fruit here with some basic analysis.
So that was kind of that's my focus.
And I have to actually try and get a data set together pretty quick because my project's
due by the beginning of August.
And there's a bunch of other stuff I've got to do as well.
So my goal is to try and get this data in a usable format as soon as possible.
And of course, whatever success I have, I'll share with you guys.
And then we can all kind of pound on it.
Cool.
I sound like the chap, if the tool, it's of course free.
If it's at all helpful to you in the process, definitely.
I put the docs in the chat if you want to try it out.
Awesome.
Yeah.
Thanks, Jake.
I appreciate that.
Just to take it home real quick and then we'll close out.
Basically we just ran a regression here, the THC on Eastern Washington.
So you know, you could interpret this coefficient as, you know, the THC difference in Eastern
Washington versus Western Washington.
Keep in mind, we have read in all the data, but you would interpret this if you were confident
that that was about a 5% difference.
But there were some red flags, R squared, and principally R squared is a pretty big
red flag.
So, Jake, this is essentially the final piece that would need to be done in your program
if possible.
So, you know, we can get all the way through here where we've created the dummy variable.
And so that could take us all the way to there and we can be content with that.
And then if you stats models, or if you're interested in doing future work, you know,
this may be something you'd be interested in looking at is, you know, just a way where
you could run simple regressions.
Yeah, totally.
That's definitely something I think I want to add for sure.
We're sort of, you know, we're building it out.
I think right now we have like the base functionality you need, which is like transformations and
pivot tables and merging and filtering.
And then I think the question from there is like, do we go more into like the visualization
space or the statistical space or the modeling space, you know, like machine learning, because
obviously there's all these amazing packages we could try and develop this front end for.
And so that's, yeah, that's just kind of the crossroads we're at.
But definitely something like that, like running a simple regression, like, you know, it's
not simple at all.
But you kind of hit the nail on the head because we all agree that, you know, with data science,
love it or hate it, you end up spending about 90% of your time or more, sometimes 99% of
your time just cleaning the data and getting it into the checklist.
And then, but that's the beauty of it because you spend all this work and then all of a
sudden your data set's actually manageable.
And then all you have to do is run a regression.
And if it's good, clean data and you have all the data points you need, often you can
just do an ordinary least squares regression and that'll be incredibly informative.
Yeah, totally.
Yeah, definitely.
So, yeah, no, Jake, Keegan has a bunch of data and I'd be interested, you know, it'd
be interesting for you to like try and load that into your program and see, because we've
had a lot of problems with this data.
And so if you're going to make it, you know, if it's going to be something sort of like
an easy to use kind of tool, these are the kind of problems people are going to run into.
Yeah, that'd be cool.
Keegan, if you're up for it, I'd love maybe, or wherever, I'd love to maybe just, if you
wanted to have like a 20 minute call sometime, maybe just try and go through some of this
together would be cool.
Absolutely.
And I also put the data in the chat because I think Charles was right.
Like if you want a data set to really, you know, what's the word, crash test your program,
that could be a good data set for you.
If I want to reach out to you, is this, this like contact cannabis email, is that good
for you?
You can actually reach me at keegan at canlyrics.com.
Cool, cool, cool, cool.
I'll definitely do that.
All right.
Great.
Well, everyone, thank you for joining today.
And we've got a big week ahead of data wrangling.
So I'm excited, Paul's got a grin, Charles got a grin, so Jake, I'll look forward to
talking with you and Charles as well.
All right.
Good talking guys.
Nice to meet you, Jake.
You as well.
How are we going, guys?
All right, take care.
Have a good week.
You too.
Bye.