And it used to start at 10 o'clock, but now they start at 930.
Well that's good.
There's also a letter engage seminar at 930 today and I was going to share that with the
group and I was going to conclude around 925 today.
That way people can make their way to their next event.
Okay, great.
Awesome.
So we'll just give it about five minutes for people to dig in here.
But I saw your GitHub repository where you've started to work on some Python, Jupyter work
books.
Would you be happy to talk a bit about the work you've done there?
Yeah, actually I've added a bunch of starter notebooks now.
Welcome Jake.
Hi Jake.
Hello.
So I did figure out how to the inventories and the plants.
I was able to read those in to pandas by just specifying the valid columns and not reading
in the invalid columns.
And so that works now.
That was a sticking point.
And so I put those up in the GitHub repository.
And then did you maybe see like what are like the common inventory types and the not so
common ones by chance?
I don't remember that right now.
You can download that workbook and look.
You can maybe dig into that next time.
But you're just in this opening bit.
I'm going to go ahead and share some links with you here just while we're waiting for
some people.
So welcome Nick.
Hi Nick.
Let's see here.
So David Busby over at OpenTHC put together this SQLite file for business to consumer
sales.
So if you were having trouble working with the large sales file, this is essentially
the sales file in a large SQL database if that's easier for you to work with.
And so I just received that in the past couple of days.
So I'm going to be taking a look at it myself.
And perhaps next week we can do some work with SQL taking some poke at the sales data.
But just keep in mind, was there a question or something?
I was wondering if we could use Dask to deal with those really large files.
Can you tell us about Dask please?
Dask sits.
Dask allows you to multitask, like multi-thread things.
And then also it reads pandas files, but it doesn't, it chunks them up.
And if there's not enough room in memory, it stores them on disk and so it saves them
like in pages.
But you don't, and then you can just use them.
It integrates with pandas, scikit-learn, most python.
And it just spelled Dask?
Yeah.
Well, it's definitely worth checking out.
So I'll put that on the agenda for next week.
So would that be useful for the raw files or?
I haven't used it that much.
I know it works well with pandas and scikit-learn.
And also if you have multiple machines, you can run Dask on each one of the machines and
it'll use the memory and processor on each one of the machines.
And that's, you may need a nice setup if you're working with this data.
So like David was telling me, he's basically, he's running this locally on his server.
So he's just got this giant SQL database just sitting there locally.
So it'd be kind of expensive to put this in the cloud.
But actually, I want to check out the SQL because there's a lot of things I feel like
you could do, a lot of queries and that with the SQL would be great.
Exactly.
So we can get our hands into that for next week.
And then I guess just go ahead and kick it off here.
Just for the people who are new here or who just got here in the past like five minutes,
I think we're going to go to about 9.25 today because Charles is heading to an event.
And then I'm going to be listening into this Flutter and Gauge at 9.30.
I'm not sure if any of you use Flutter, but it's sort of a fun tool.
So it's going to be interesting to see.
I would like to see that get pretty popular because it's still pretty new.
Anywho, just sharing that.
All right.
So I guess I can just go ahead and share my screen and just show you some of the things
I'm doing and then we can talk about it along the way if that sounds good to you.
Great.
So essentially we were starting to look at the different sample types last week.
And there's probably going to be different types of cannabinoids for the different sample
types.
So oils, of course, are going to have a higher distribution than flour.
So I just thought this would be an interesting thing to look at.
So over in Spider, or you could use your favorite text editor and your favorite programming
language.
But once again, I'm just working with the same LCB data dump just because the variables
are relatively the same.
So I figured if I can master it for one month, then it should be fairly easy to generalize
to the full data set.
So just go ahead and I'm not sure if this is actually going to be in the right location.
So never mind this.
Okay.
So essentially we're just going to read this data in here and we're going to start looking
at some of these sample types that we looked at last week.
So real quick here, I'm actually just going to go ahead and just commit everything over
to GitHub just so that everybody can follow along.
And then just so everybody can follow along with all the material.
You know, I'm trying to keep most things here in the cannabis data science repository.
So that'll need to get cleaned up a little bit, but so far just sort of archiving things
there.
And then, okay, so we've got our data read in.
And so the tool that I'll be using today is just Seaborn, which is sort of an extension
of Matplotlib.
So Seaborn requires Matplotlib, so you can see it more of an extension or an enhancement.
We'll go ahead and grab those packages.
And essentially, you know, let's find out what columns we even have here.
So it's sort of a handful, but essentially, we want to look at some of these cannabinoids
that may be of interest to people.
So we could maybe create a total cannabinoid calculation, but at the moment, let's just
pick a, you know, this first cannabinoid here, THCA.
And so, you know, in the lab, you could call this an analyte.
So you could say your analyte is THCA.
And so, you know, now let's just basically look at the distribution of THCA in the lab
data.
And let's see what happens.
Okay, so that's what happened.
I think we want...
So we'll just look at the kernel density of this analyte.
So this may take a minute since there's a lot of data.
And essentially, you know, you can add the hue, which is the actual variable type.
So the actual, you know, inventory type, remember, is the intermediate type.
So we'll eventually check out the hue with intermediate type.
So we may have to get just a subset of our data here.
I wonder if we can just do a sample of it.
Okay, so now we've just got a random sample of the data.
So now maybe it'll be a bit more approachable.
So we've at least got a chart now.
And so this is just an estimated distribution of THC.
So not the most informative chart ever.
It looks like there's a grouping lower down towards zero.
You know, perhaps that's edibles or things of that sort, or miscoding, or it's hard to
say at this point.
And then there's a grouping up here.
And you can see there's a slight bump up here, which must be the oils.
But you really need to separate this by the actual intermediate type for it to be informative.
So let's look at how we can adapt this plot here.
I'd really like to use this plot, but it looks like my Seaborn version may be outdated.
You could just add hue.
So it looks like my Seaborn version may be outdated.
So we may have to just come back to these at another point.
However, we can maybe now change gears into something else interesting that I was going
to show you today.
So we've been talking about mocking up the LEAF API.
And I thought, OK, the first step is we're going to need a database just to store some
things.
So I just thought, OK, we'll just go ahead and use the database that essentially I'm
already using for some of this open source material that I'm putting out through Candletics.
So some of the tools I'll show you today that essentially I'm using to mock up the API are
essentially the Candletics engine and then the Candletics API.
And I'll go ahead and commit the latest work here, which has these two mock endpoints.
So I've gone ahead and mocked the lab results endpoint and the MMEs endpoint.
And then we can go ahead and I'll gradually mock all of the LEAF API endpoints.
And then you can use them as you please for testing and whatnot.
But they'll be fairly limited with a limited amount of data.
So
and then I'll go ahead and share these links with you here in the chat.
The API and the engine.
And so essentially, I'll just go ahead and just kind of show you these.
Nothing set in stone here.
These are just sort of general tools that I was putting together just to help with cannabis
analytics.
So we're doing cannabis data science here.
So these are just sort of open ended packages that you can use to contribute to help your
workflow.
So the engine is just a Python package, which you can just install with your command prompt.
So you can just, you know, install can lytics to get the latest version.
And we're at the moment.
So you've now got kin lytics installed.
And then let me actually restart spider and I'll show you a bit more.
But essentially, at the moment, it just leverages Firebase for a database.
So if you want to learn about Firebase, I'll put that in the chat.
So Firebase is sort of a fork of MongoDB and I also like MongoDB.
So that's worth checking out.
And so these are what you'd call no SQL database.
So it doesn't have quite the same connective structure as the SQL database.
And just to kind of just give you a look at what the data looks like here.
So here is just some collections that I've made with some of the MME data.
So you have essentially a directory.
So you can think about it as a directory of collections and documents.
And then all documents are essentially a JSON file.
So you can think about these as key value pairs.
So you essentially just have a giant JSON database.
So it has its pros and its cons.
The cons right off the bat are going to be it's much harder.
Well, it depends on your perspective.
But it can be harder to do queries and aggregations than it is in an SQL database.
And then it has some advantages in that you're just going JSON to JSON.
And sometimes the loose structure can be beneficial.
And it can be fast if you know what you're looking for.
So once again, nothing set in stone.
But I essentially just tossed the MME data.
And I tossed about 1,000 lab results here just so that way you can just pull these to
start doing some practice.
So how did I get those there?
I used the Cantlytics engine.
I think I put that in the chat.
Yeah, so that's the Cantlytics engine.
And so this is just essentially a wrapper around Firebase.
So Firebase already has a Python SDK.
But this is an even more expedient way to interact with the data.
And so just put some useful tools here.
So you can initialize Firebase.
Use a bunch of wrappers to get documents, get collections.
You can import data.
So this handles importing data into Firebase from CSVs and Excel.
So a fairly way to get your data in.
And then so that's essentially one thing that Firebase was lacking was easy imports and
easy exports.
So there you go.
Now you've got an easy way to import data and an easy way to export data from Firebase.
And if you look in the tests, you can see how you can go about spinning up your Firebase
instance.
And then I actually use that code right there to populate these two collections.
So it's all open source.
So you can use it yourself and populate your own Firebase database essentially for free
if that's something you're looking to do.
You do have to pay for Firebase.
But at the moment, the cost is really low.
So as long as you keep your usage low, then it's essentially free.
So it's a nifty tool.
But I'm not set in stone to any one particular tool.
So use your favorite tool there.
And so then we've got our data there with the CanleLytx engine.
And so now we can retrieve it with the CanleLytx API.
And so once again, you can find the code here on GitHub.
But this is essentially going to be the interface to all the actions that you can do in CanleLytx.
So if you want to interact with any of this data, then you can use the CanleLytx API,
which includes batteries included.
It includes the CanleLytx engine.
So this tool is bundled up inside the API.
So once again, nothing is set in stone.
But it's just a classic example of dog fooding, where you eat your own dog food.
So you use your own tools.
So that's the idea here is just building a set of tooling to perform cannabis analysis.
That way, we can control everything from the ground up.
We can control database interactions so you can get pretty low level.
And then there's actually a lot more here than just database.
So for example, you can work with storage, so you can put files into storage.
You can download files, rename files, delete files.
There's also authentication, so you can create users.
You can give the users custom claims.
You can give them custom tokens.
And so the token would be for if you want to log in on your front end.
So this is essentially a backend system that can talk with your front end.
And then there's utility functions to get your users, update a user, then delete a user
if need be.
So it's just a little backend service wrapper around Firebase.
So if you need those services, then this may be an easy interface for you.
So let's go ahead and test some of these things out.
So you can essentially clone the repository.
And then you'll have this repo here.
Looks like everything went through.
And then just to kind of walk you through, so first things first, you'd clone the repository.
And then you can actually run the server locally.
So this is sort of to give you an example of how you could go about doing that.
So you'd go to the folder where you downloaded, where you cloned the API.
You can open the command console.
And then you can just do, so you can just run server.
It looks like one of my endpoints is not 100% correct here.
So we can just jump in there.
And so just to kind of show you a little bit about the API and tell you a little bit about
it.
So you've learned now that, OK, the database is Firebase Firestore.
And so the API is Python Django.
And so the typical structure here of a Django file, so in the root directory, you have your
requirements.
You have your read me.
I'm just using node just to kind of help run some helper functions.
There's no actual node dependencies.
And then just, and then Docker to just get it up to, it's actually running in Google
Cloud App Engine.
But you can run it anywhere that you desire.
And so just to give you a quick structure, so all the code is in the Canalytics API.
And so just to give you an idea of what a Django project looks like, you'll have a settings
file.
And this is typically fairly long.
You kind of want to try to keep it short.
And this is just where you define all your key parameters towards how everything's going
to be functioning.
So this is where you define any third party apps you're using.
And we're going to be using the rest framework, since we're going to be building an API here.
And then everything else is relatively just boilerplate.
So the rest of this is essentially just boilerplate to make everything else run.
And you'll need to set up essentially your example credentials.
So if you walk through the read me, I may have to flesh that out a bit about how to
get set up with credentials.
But essentially, you'll need to get these credentials set to really use this in development.
So you've got your settings.
And then where all the beauty happens, well, so you've got your URLs.
And so these are going to be your actual endpoints.
So for example, your endpoint here.
And so it didn't like that these began with a slash.
So your endpoint here is just going to be test, leaf, MMEs.
And so I'm pointing this to essentially a Python function.
So I just put these in this file over here.
And so then this can give you some ideas about what Django endpoints may look like.
So for a REST API, it's just a Python function that takes the request as an argument.
You check the method of the request.
So say you wanted to add functionality here for labs to actually create results.
Well, then you would add a post.
And then you would say, you know, right?
And then you may even just, you know, you could return a not implemented error.
So eventually, we'll want to go ahead and build out that functionality.
Since we're essentially, remember, let's not lose sight of the original goal.
Our original goal is we're essentially just creating a mock leaf API so we can, you know,
so we can just, you know, basically mock some of these calls ourselves.
So let's go ahead and see, you know, see this in action.
So you know, we can come over here.
You know, we'll just create a blank file here.
Great, and so just since we're working in Python, you know, we can use the request package
and, you know, the base.
So the production base, if you want to test out what's in production, it's going to be
so it's going to be API.canalytics.com.
But you know, for right now, we're just going to be using the development base.
So that way, you know, I can make, you know, rapid changes here just for, you know, just
to kind of show, you know, show what we can do in development.
And that way you can learn how to do development yourself if you're interested.
So we've got it set up.
Remember, we're going to be hitting these endpoints.
So let's try to get the MMEs.
So we just, you know, we're just going to make a get request, so requests.
Took a second, so we got a 200 status code, I don't know.
So I guess let's just do a data dump and see what we got.
No promises here.
This is a live test.
Oh, sure enough, look at that.
So that's a little much to work with.
So let's just look at the first observation.
Oh, so it's not a list.
So let's see what we're even working with here.
So we've got a dictionary.
Ah, so it looks like at the moment the CanLytics API, let's actually go and so this is what's
nice from building it from the ground up is you can actually now come over here and we
can actually look at the endpoint so we can see what the endpoints actually doing.
So that way it's not a black box, right?
So now we control the data import, right?
So we've now imported data here into Firebase.
We've got the records here, right?
So here we have somebody at 555 Brown Street, LCB Grow.
Okay.
So we want to basically be able to query that with our API.
So let's start tinkering.
So we're getting our MME data.
So if we make our request, it'll check to see if there's a limit.
Otherwise there won't be one.
So let's try, let's test this limit out.
So the way you could do that is you're just going to do Q or question mark, limit.
Let's just try doing a limit of two.
Yeah, that's right.
So now let's, it looks like we may have gotten an error.
So we got a 500 error code.
And what's nice is, you know, we're in development here, so we can actually see the debug error.
So you know, so this is how you would do development.
So you would see, okay, there was a get request to the LEAF MMEs.
Ah, and look, here's the error.
To has class string, but we expected an integer.
Okay.
So we can go and tinker on this MMEs endpoint.
So we now know this limit has to be an integer.
So just a hot fix here, we could do better logic later on, but you know, you could just
say, you know, you could just do something silly like that, you know.
So if there's a limit, you know, make sure the limit's an integer.
You know, and so go ahead and save that.
This should, I think this should just refresh.
I'm not sure if I've got the live server rerunning.
Interesting…
So.
60 seconds for you.
little bit.
And actually we may even just want to change this just to return the docs just
outright. I think that's just a bit a bit better. And let's actually just go
ahead and re-spin this server. When you're actually doing
development there is a live reload package which makes things a lot better.
Okay.
Okay let's try this again. Okay so now we've got a fast response. We've got our
200 status code and we've got two documents here. So now this
little fix here to the limit that seemed to do the trick. And if we
look at our data we've got 555 Brown Street, the LCB grow.
And so the next step is really to set this up to actually handle queries. And
so the nifty thing is the get collection routine is set up to handle
filters. So we can basically pass filters of the sort.
Key, you know, so say we wanted you know key, you know, name, operation. Right?
So on my to-do list is to actually set up these endpoints to be able to handle
queries. Because you know we could actually set this up you know to really
handle any sort of query. Because it would be cool to set it up so that way
you could query by phone number, address. You could even do something that matches
close to the address.
You could of course do global ID. You don't want to do if anybody's suspended.
You want to do type, city. So all of these are fields that we could set up to
query over here. Okay? So now let's go ahead and you know let's hit this lab
results endpoint to kind of see if it works as well. So we'll probably have to
do a similar thing where we fix the limit.
And this is also not set up for queries. But let's okay so we go ahead and got a
live reload there.
And so now let's check out. Now we're just going to hit the lab results endpoint
and see what happens. Okay it looks like we may have gotten an error. Okay so we
got a 404. So that means this endpoint was not found. So we probably want to
also set up our URLs you know to you know it's better handle 404s.
Because you know the idea of any endpoint I mean of any API is you want
to handle all endpoints elegantly. So you know if somebody hits your base
endpoint you know you want to so here's just a look at some of the other
endpoints of the CanLytics API. So you know so if somebody hits the base
endpoint so let's just say you just hit the base here you know you still get a
status code and you and then you know you actually get a message. So it actually
says okay you know welcome to the CanLytics API. You know you know we
probably want to touch this up instead of data you know we may actually want to
change that to message. But you know you're welcome to come in here and
tinker on this because this is just a completely open source just sort of fun
API that is just sort of set up just to you know tinker around with cannabis
data. And you know like I said I'm not set in stone to any technology so if
anybody has recommendations you know fly them out you know I'm not so you know
I'm not stuck in stone on anything. And so just to give you a look here so we
just hit the base endpoint we just got this message and so not to get too off
track here but the idea is you know to handle all endpoints elegantly. Okay so
the reason this one failed is we're actually we actually want to hit the
test endpoint the test leaf endpoint. So let's go ahead and hit that endpoint and
see what happens.
And I'll publish all of this afterwards so that actually I'll do that right now
that way you can actually hit the live API. Okay so now we've got a we've got
our response, we've got our status code, and hopefully we have two lab results.
Okay so we're still returning lab let's actually fix that real quick so that
actually is kind of annoying me so it may annoy other people. I think it's just
better if this just returns
I think it's just better if it just returns the documents as a list.
Got our live reload and see this is what's nice about you know cooking up
your own little API here because now you know with just a couple lines of code
you know we've now got this is a list and if you prefer it as something else
then you know get in there and tinker and change it. So now we you know we've
got our list here we've got two and you know we've got these lab results and so
this is these are the exact same you know lab results as
these are the same lab results you know from that LCB data dump. So don't be
here's the link to the data dump.
Okay but long story short so we've gone a long ways now we've basically gone
from this data dump here we've gotten the data into Firebase we did that with
the help of the Canalytics engine and so you're welcome to go and explore over
here you know and check out check out the Firebase module you know we've used
the import data function and then if you want just examples of that go check out
test because the test is you know this can this can get you started that's how
you initialize Firebase you know create some documents and then import your data
and then now that we've got our data in Firebase we need a way to interact with
it so we've got the Canalytics API and that's as simple to use as just you know
making some get requests to you know to these various endpoints and then you've
got this full suite of data and just to go ahead and polish it off I'll just go
ahead and round it out by just showing you so you've you've done the install
it
all right so you've got the Canalytics API so you've done the installation we've
done the development we've done the testing and so now it's time to publish
well you know that's as easy to do it's just you know NPM and so that's why I
have node is to expedite the publishing process so what's going on here so if
you go and look over here and publish so basically we're containerizing the API
so we create a container of the API then we push the container up to Google Cloud
and then we actually are then redirecting all hosting from Firebase so
over in Firebase we've actually got hosting and you can set up multiple
sites here and you know we've actually got the you know the Canalytics API up
here and so you know right now we're containerizing everything and you know
it's going to be a little bit of a build process here so we're publishing that
so I'll just sort of minimize this and then sort of bring it all home here so
that way we can get people out of here in time for their next seminar
all right so sorry to just kind of go on that long spiel there but you know what
your what's your input what's your input too complicated of a way to get data or
a manageable way I mean it's really cool it's probably a little more than what I
would need but it's really interesting well this just got posted over here so
here in this last couple of minutes
we'll see if I wanted to do like more advanced things this would be this would
be really useful yes so now it's published so now in this last minute
let's see if we can't move over to the production endpoint and get the same we
looks like
it looks like there is an error along the way that's interesting oh I need a
but it's interesting
oh I need a but we got 200 status codes and so if we check out our data here
we've got the two MMEs and we have our two lab results so thank you for bearing
with it but there's essentially you know an MVP you know a bit of a viable
product right so we've basically taken our data we've gotten it into you know
firebase storage anyways we've got it there using the engine and now we've
retrieved it from the you know the analytics API so you know you're welcome
to you know go tinker on that and
and see you know what happens
but
but anyways it looks like there's still some debugging to do so I'll go ahead
and wrap it up there
but thank you thank you for attending and I'll fill you in and let you know
what's going on next week and next week we'll be checking out that SQL database
see you then awesome thanks for attending Nick
Michael