Welcome to the Cannabis Data Science Meetup Group for October 13th.
Hopefully you're all doing fantastic.
As always, here to have a little fun, talk about cannabis data, and see if we can discover some insights.
So, I'm going to go ahead and kick off the presentation.
All right, we're going to do a little, you know, novel analysis today.
So, we're essentially picking up where we left off last week.
And we were given a market analysis of Massachusetts, and last week we estimated the crisis, and we were going to see if we couldn't have a rough estimate of profits.
So, to just show you some macroeconomic material here.
Here are some notes that I've written on macroeconomics that you can find on my website.
Nothing glamorous here.
Just a recap of some economics, and welcome to the meetup, Heather.
Welcome, Heather.
We're just going over a recap of some of the economics we've touched on.
Then we're going to dive into the Massachusetts data, and then we're just going to keep extending if we finish with Massachusetts.
I've already lined up a couple of data sets coming from California that we can start to wrangle.
So, we're just going to keep calculating these economic statistics, state by state, until we can eventually calculate the aggregate economic statistics.
In this case, we're doing a market analysis of Massachusetts, trying to discover the crisis, and then we can begin to perhaps estimate the profits of the companies.
As I was hitting this hit fall, I realized that the economic models, when we're estimating the competitive prices, assume that profits are zero.
So, this may not necessarily pan out, but we can give it a shot.
Long story short, we're just going to copy some of the code from the prior weeks.
And then we're going to get the data.
Welcome, Jessica. We're just doing a bit of just some crude analysis of some data in Massachusetts.
So, welcome to the group. Thank you.
Well, I guess just to give you a bit about us. So, I'm starting the company, CanLibx, and the main focus that we do is data analytics.
And so, we've started this group, and essentially, each week we wrangle some data and calculate some statistics.
So, you're welcome to share a bit about yourself to the group if you'd like, but that's entirely up to you.
Sure, no problem. I actually just came across your guys' meetup this morning. I recently started a data analytics boot camp with Springboard.
So, my career background is 20 years of marketing and sales, but within those positions, I've always done some types of KPI reporting and things like that.
So, I'm now getting certified and learning Python, Tableau, Power BI, things like that. But again, early stages of those programs, pretty efficient in Excel analytics.
So, I thought this would just be an interesting meetup just to sit in, kind of observe, maybe make a few connections and go from there.
Awesome.
Programmed through?
I'm sorry?
Where's your program through?
It's an online program called Springboard. It's powered by Microsoft.
Thank you.
Yep.
Microsoft, okay.
Well, excellent to have you. You're in the right place.
Fantastic.
Essentially, we're just beginning. So, essentially, I've just written some Python packages here that we'll be using.
And essentially, we're going to be using the Stokrata API to get data from Massachusetts.
So, for example, we'll be working with this data set here.
So, there's documentation about how you can get started and how to get an API token, which is optional, and then the fields that we'll be working with.
So, long story short, we'll be making requests to the API, to this URL.
And we'll essentially just start by getting this production data here.
And these are the variables we're working with. I simply defined them here just so that they're easy to see.
You know, we can see the last five observations.
If you want to look at the last observation, you can turn that into a dictionary.
And so, we see, okay, we've got data through October 12th, at which point there were 127,000 mature plants, so on and so forth.
And we can even see sales. So, this, I believe, is cumulative sales.
So, we can essentially calculate daily sales by simply taking the difference from day to day.
So, that's the first data point that we're adding.
So, now we can actually look at the last observation and see, okay, on October 12th, there were, you know, about 5.5 thousand in sales in Massachusetts.
There are a couple outliers here that we just need to clean up. So, sales that are much higher than believable, as well as negative sales.
So, those may have been adjustments that happened.
Next, we're just going to aggregate the data into, you know, monthly and quarterly series.
When you aggregate, there's two ways you can aggregate, right? You can either take the sum, so that would be what we're doing when we're looking at sales, right?
Because when you aggregate sales, you're looking at the total sales that occur during the week or month or quarter.
And when you aggregate something like total employees, well, it would make sense just to aggregate some total employees there.
You're more looking at an average, you know, what's the average number of employees at a given point during the month.
The same is true for for plants and packages, right? You're more looking for the average there.
So, we'll create these series.
Then another Socrata data set here is simply the licenses.
So, these are, you know, the business name, the license type.
And there's actually many fields for it, for the licenses. So, there's a lot of, you know, rich analysis that you can do here.
So, for example, let's go ahead and read that data in so that we can take a look at it.
So, essentially just reading in the licensees here.
Here are the last five.
And here it is the last observation.
As you can see, we have quite a number of data points.
We like to quantify things here. So,
we can say, oh, you know, what's the length of all of these?
There's 51 data points for each licensee.
So, that's awesome. You know, the more the merrier.
They've got some real cool things here. So, for example, they've got, they already have the address geocoded.
So, you could already do some interesting plotting on a map.
And we've been meaning to revisit mapping.
So, I'm not sure if we'll get around to it this week, but perhaps if not this week, then maybe in the near future, we can start plotting some of these on a map.
And then the main data points here that we'll be using are, okay, you know, the license type.
And simply just the count, you know, how many licensees are there?
So,
2750.
We're doing a market analysis here. So, ultimately, we're trying to price everything.
And so, we priced everything last week, just sort of recapping real quick.
And so, we're given the price of flour,
which is just the average price per ounce.
And then we just created a couple supplementary series here
just to put this into, okay, so what's, you know, the price per ounce?
I'm not sure.
I guess.
And this is something that we noted in prior weeks was, okay, where there's actually missing data here in April of 2020.
So, that's actually why there's the big dip. So, it's not that necessarily that there was.
And this is the unfortunate thing about missing the data there, right?
Because clearly there was, you know, a shock during that time.
So, their data definitely probably fluctuated.
But unfortunately,
the data reporting at that time is not great. So,
yes, we have a conundrum.
We'll just move forward.
Now, just stating that.
Okay.
I think we still need some of these data points.
So, essentially, we've supplemented these data points with just a handful of other data points that are collected by the Federal Reserve here.
So, for example, we're grabbing the population in Massachusetts.
And we're grabbing, you know, the GDP in Massachusetts.
And you see here, you know, quarter two of 2020, there is a significant dip here.
So, perhaps this dip in price is representative of what actually happened.
However, we'll show you, you know, if you look at sales,
if you look at sales, you'll see that, oh, we're missing sales really between the end of March of 2020 and the end of May of 2020.
So, that's real unfortunate because
it would have been interesting to know what happened with sales during that period.
So, that's actually an opportune time for, right?
So, we've been talking about creating value with data and by curating data.
And so, people are in possession of that data, right?
So, if you're a retailer, you may be in possession of your own sales data during that time.
And so, you know, wouldn't it be so cool if, you know, all of the retailers could essentially form like a data co-op
and, you know, basically say, report, OK, you know, this is what, you know, total sales were during this period in Massachusetts.
Because that data is valuable and missing.
And it exists somewhere.
Anyways, more of that to come.
So, that's, as you can see, a thing that we've been driving here at CanLinux is, you know, there's actually a lot of value gained by, you know, strategically,
you know, curating your data and, you know, getting it into a usable format and supplementing it with other data points.
And there's a lot you can do.
Long story short, I'm going to go ahead and read in this data in case we haven't already.
And, you know, that's simply the same data that we're looking at here, except just for the tail.
We're just looking at this segment.
So, any who you've read in this data.
OK, now this is where we were going to get into the economics.
And so, I just wanted to give a bit of a recap here and actually estimate a slightly simpler model this week and see if we can't get a better measure of the rate of return here.
So, so one way we can estimate the rate of return is, OK, so what if.
So last week we said, OK, there was two parameters, alpha and beta.
In this week, we're just going to say, OK, it's alpha and one minus alpha.
And so that restricts the model to having constant returns to scale.
So that's an assumption that's built into the model.
So that's not great.
Well, that's a subjective statement, but it is another assumption built into the model.
It's a simpler model.
So we're just going to be estimating one parameter now, we're just going to be estimating alpha.
So, and the way we do this is we basically just divide the whole equation by.
So here n is labor.
So if we just divide everything by n, the n is, you know, one to the one minus alpha is one.
So that n disappears and then you're just left with.
KT divided by alpha to the alpha.
And YT divided by NT.
And then if you take the log of both sides, so this is called log linearizing.
That way you have a linear equation that you can estimate.
Then you have the log of output per labor.
And that should equal a constant.
Plus alpha times the log of capital per labor.
So this is sort of our theoretical framework for estimating alpha.
Then, yeah, go ahead and just show you the rest of the economics.
And so I'm going to open this in another cab so I can keep this spot.
So the reason we want to estimate alpha.
Yes, because so, you know, given this model, right, where we've got.
Right, so production is now going to equal capital per labor to the alpha.
So this K is capital per labor to the alpha.
So alpha equals capital per labor to the alpha.
So.
Capital should get paid its marginal product in a competitive market.
So if you take.
So the marginal product of capital is the derivative of the production function with respect to capital.
So if you take the derivative with respect to capital.
So that's going to be the derivative of K to the alpha to the K.
So that's just going to be OK.
The way you do this is you bring the exponent down front.
Alpha times K to the alpha minus one.
And that will be the marginal product of capital, which will be the rate of return of capital in a competitive.
Market.
So.
So this is what we're boiling it down to.
So we're basically going to say, OK, the rate of return rate is going to be alpha times K over L.
Capital per labor already to the alpha minus one.
So that's the theoretical framework that we're going to use to estimate the rate of return.
Here in Massachusetts.
Because I wasn't satisfied with how we did it last week, so we're going to use a slightly simpler model and see what our estimate is.
That's the theoretical framework.
And so now we're going to.
Estimate it with the data.
So we can define our variables first.
So right out of the gate, we're just defining why as sales.
And we're going to be working with the monthly series here as.
As we'll see with.
Well.
We may not necessarily need to work with.
Let's actually work with the daily series here and.
And just kind of do something a little out of the box here.
If you were going to estimate the competitive.
Return Massachusetts. OK, so.
Similar analysis as.
Well.
Well, yes, let's go ahead and try it and we may just have to do some data cleaning along the way, but you know that's the way we do it here. We know we just have to sometimes.
And do this so.
Let's go ahead and define.
Why.
As just production sales.
Right, we're defining K as just the total number of.
You know, that are tracked.
And this case we were defining labor as hours worked, but now.
Wait, let's just use just the total number of employees. I think we're just.
Introduce less abstractions and we can use the daily series.
So we'll just say labor is just the number of employees.
This is the part that was giving me pause is we actually now need to exclude missing observations here, right, because.
You know, say we define our variables.
Labor is fine, except the.
Interesting.
That's right. So I'm real.
This spike here at the end of capital and labor are giving me quite the pause.
And I also essentially wanted to essentially drop this whole time period here where we're missing sales.
So how are we going to go about doing this?
Well,
Let's find the periods where we're missing sales, right? So this is going to be why where why.
Let's let's missing sales.
You don't necessarily want to just drop any days that are just.
You know, the reason it looks like, you know, there's not really sales through.
And so this is the period I was primarily worried about March 26, 2020 missing sales through May.
2020 and then, you know, it looks like there's a scattering.
Just for the sake of time here, I think we're just going to exclude all the days that are missing sales. However.
That may not necessarily be the best practice.
If it's closed, you can't eliminate that data. I mean, you'd be creating data where it should not exist, right?
Can you say that one more time? Like if the store is closed, some places around here, they're definitely open Christmas day, Christmas Eve or whatever.
But if something is truly closed, inserting the data would not be allowed, right?
Well, it's more just it appears for that day, we're just missing an observation. So maybe like the office was closed and then they just entered the data in on the 26th or what have you.
My main thing is just don't want to include sort of these oddballs in the analysis.
Because you're like when you're running a regression, if you know our output is zero and we have, you know, a certain number of plants and employees, well that may sort of bias the model in one way or the other when the sales weren't in fact zero.
It's sort of my worry.
So I just thought, okay, to avoid introducing bias for days that are coded as zero, but they're not actually zero, then I was just going to ignore those days.
So, you know, luckily, you know, we're given, you know, a little more than 1000 days here. So
actually, I was just going to exclude, you know, these 97 days from the analysis and saying, okay, you know, these days aren't your typical day. So let's, let's just do our analysis on, you know, your typical day, so to speak.
I hope that answers your question Heather, but please chime up if you're still, you still have concerns.
Nope, thank you.
But it's just sort of a, you know, a step that I'm taking to try to remove outliers that may not be representative of the normal.
So,
so long story short,
I'm just going to get these where we're not, you know, missing sales.
Oh,
Keegan, I'm sorry, I see my question now. Okay, so when excluding data like this, like, so for example, when I was fitting my crystal structures to the model or whatever,
like it didn't always fit right, but then there are points that I could, I could eliminate because the light beam did not actually hit there, even though the light beam was there.
So it's like when you eliminate data in this case, it's like data you're supposed to have, but that's not there or data that's truly not there.
It's a very theoretical thing, but I'm not sure that I'm maybe understanding it right.
So, in fact, there's a, there's a whole sort of, so in econometrics, they'll do studies about, okay, you know, what happens if there is measurement error or, you know, what happens if you do exclude a subset of data?
And what happens if the subset you exclude is non-random? And so long story short is, yes, it does introduce bias. And so, yes, if you, if you just exclude a segment of the data, like we're doing now, or if you were going to
have measurement error. So if we're going to, it's basically like you've got the bias one way or the other. So you've either got
measurement error, where these days are coded as zero, and they shouldn't be, or you've got selection bias potentially, where we're just selecting a subset of the data for analysis.
And by doing so, we may be introducing bias because you're right, there may be something special about the days that we're excluding.
So, thank you. Thank you. I think that's well said because it's like it tells you how you can interpret this data, because at this point, like as of yesterday, last week, I wasn't sure if the model was right because alpha and beta being completely
out of range, like when something's completely out of range, usually the means of model is wrong.
True, true. Or, well, we had a wide confidence and we had a small number of observations, so we couldn't really say with much confidence what the parameters were going to be.
So, another potential we could do is we could potentially, instead of just excluding the missing sales, we could potentially just look at this, you know, just from the last year.
But then again, you know, it's saying, okay, well, then you know we're excluding all the data from before last year. So, that's just a quick way to wash your hands of the problem of selection.
Sure to do that with everything in my life. Thank you so much. It sounds great.
So, long story short, we're kind of stuck between a rock and a hard place where, and this is where I like to, you know, stress to people, you know, really like when you when you see someone present their data definitely ask these hard questions and push them on these because, you know, when you're doing any sort of data analytics, you run into these sort of
conundrums and, you know, a lot of the times these solutions get done you know behind the scenes and they may not get talked about a lot, right. So like in our analysis, we've got this missing data gap.
And, you know, that may not necessarily be, you know, readily apparent in whatever statistics we end up calculating.
So, good point. And my only comment is just, just to power on so it's just sort of present your biases, just acknowledge them, be upfront about them.
Emphasize them and then, you know, not necessarily but I'm under the belief that some statistics are better than no statistics. So we'll at least try to measure the alpha, so we can get the rate of return of capital.
So we're going to hedge it that it may be just wildly, wildly inaccurate, but we're just sort of going to try to calculate it for a curiosity.
Real quick here.
I'm having a bit of trouble excluding these missing sales. So,
let's see if we can't solve this real quick.
Let's see if we can solve this.
Okay, okay, this may not be the end of the world because I think maybe the plots just off. So,
great. So this worked out okay. The plot is just that it's just confusing. Okay, so I'm sorry short. We're just going to try this real quick. YKL.
So now we can actually define, right, output per labor. So that's Y per L.
And then we're going to define output per capital.
Like, exactly.
So, we've got our fluctuation.
Capital per labor is increasing.
So that's, that's quite interesting.
Next, we're going to be essentially trying to estimate alpha here.
Right. And so remember, we have to log.
So, here, we have to take a log of both sides.
So,
then we can find the log of capital.
That's right. And we just have to add a constant, right, because we have our constant term, which is technology. We've got a
technology augmented production function. So, let's just add a constant here.
Oh, righty then. This should, I'm not sure why we're getting a bad syntax error, but
we've got double equal signs. That would make sense. All right, looks like the regression was fit.
But maybe it was not fit. So, it looks like, it looks like something went wrong here.
I bet you may have exactly, see here with, whenever you take the log,
if you take a log of a negative, you can't take a log of a negative, right. So, we've got some missing values for
y here. So, why is that?
Okay, so that's right.
So, yeah, that's because we're missing sales on October 15th because we have to do the difference.
All right, so there's several ways we can do this here. So, we can either just exclude the first observation, or we can start at a specific point in time. And so,
we could potentially do both. So, here I'm just going to exclude just the first observation because it looks like the first observation is missing for sales. So,
may not have quite done the trick. Interesting.
Okay.
So, this is a mess. So, what I'm going to say is, okay, we're just going to just, whatever was going on in this period, October 15th to what it looks like about,
you know, the 20th of November is not consistent. So, yet again, we're going to exclude values, which, as Heather pointed out, introduces bias, right, because there may have been valuable information in that time period.
But because it's just so messy, we're just going to have to restrict our data to after, to the 20th or after. So, how are we going to do that?
So, we basically want all values of y where the index greater than or equal to the specific date.
So, I'm going to promise this is going to work, but it looks like it may have.
So, then we basically just want to repeat this for capital and labor.
I have to restrict it to the first day anymore. Okay.
Maybe now.
But now again, you know, now I've gotten real, real uncertain about things. And in fact,
we just estimated a negative value for alpha. So, this something went wrong.
That may be for the best. So, let's see here. So, log y.
So, our analysis may have gone.
Unless we just have.
For some reason.
So, you know, this is where I kind of just want to say, you know, we may not want to just keep playing, you know, code pool here and, you know, we may just not have, you know, successfully estimated alpha in this manner.
And that may be okay.
Because, you know, last week, you know, we saw it. There's one more thing I'm going to try, but, you know, last week we saw that if we estimated this with two parameters, alpha and beta, we didn't have a constant returns to scale.
We had decreasing returns to scale significantly.
So, it may be that this assumption, you know, this assumed production function with constant returns to scale is not adequate or representing the market.
Before we give up, the last thing I wanted to try was, okay, let's just, let's just try doing this.
You know, like, let's just try like after, after May of 2020.
So, let's just start in June of 2020.
And just see if, if we start at this time, if the data is just a bit more.
You know, let's outliers, no promises. So, let's just see.
And let's also just say.
We'll just try with this before we.
And this is.
Yeah, like I said, I think this analysis is just going to need to get tossed out because basically, once again, incredibly low R squared. So we're not really explaining much of the variation.
Not really a significant coefficient here on alpha. It's close to one, which is not really value that you would expect for alpha.
So, I hate to have just run us down this rabbit hole here, but sometimes this is, this is constructive.
To see, you know, we attempted to estimate this model here, but I don't think we successfully did.
And so that's okay. And just to show you essentially what we estimated last week and then why this may have broken down this, this week.
So last week we were looking at the monthlies.
And that's the other thing we could potentially try is, oh, you know, what happens if you measured labor in terms of, you know, hours worked.
And so, you know, last week we looked at labor in terms of hours worked.
You know, on a monthly level.
And so here.
We're estimating this same function, except for letting one minus alpha be its own parameter, beta.
And so then we're just we're log linearizing and saying log y equals log a plus alpha log k plus beta log n.
What we find when we do that is
Yes, an alpha of around point two, a little less than point two, and then a beta of around point four or five.
So that's if you add those together, you know, that's less than point six five.
And so they would need to sum to one, you know, for there to be constant returns to scale here.
Right. Because that's what this model is. Right. So if we assume that this is alpha, then, you know, one minus alpha, those have to add to one.
And then in this case, they don't. So what that means is we've got decreasing returns to scale.
So that so that would violate the assumption of constant returns.
Can you say that one more time, please?
I don't know. I said thank you. That was just really helpful to see what you guys did the last time.
Yes.
The derivative would be different.
Well, maybe that was the lesson learned today that things don't always go as expected because I was essentially expecting this to to pan out and then compare the rate of return to what we estimated last week.
And so
Last week.
Just in the last 12 months.
Last week, we tried to estimate the rate of return, you know, essentially per plant. So and so this would actually be the cost per plant or, you know, what the investor would expect to get paid per plant.
And this was with our first model. And so I was hoping to to estimate it again with this second model and compare the two.
But we were we were unable.
So, so basically, you know, we've tried to, you know, a couple different models now, and it seems that the only way that we're able to is monthly.
With, you know, the number of employees, labor proxy as hours worked and capital proxy as the number of plants.
And, you know, I would, you know, put this series hedge on these numbers, but you know, this is how you know the best we could do, or the best I can do with the data given. So
So I think
I think I'm going to think about this the next week because I was wanting to kind of try to tidy things up today and I think I essentially just uncovered more more loose ends.
So,
So perhaps for next week we can finally start to really hammer this down or turn directions to California or another state.
But then that's another thing is I'm also going to ask a couple, you know, actual growers, you know, what is your, you know, cost per plant, you know, is your cost per planet, you know, around $100 or so like, does that sound reasonable?
Or, you know, is, you know, $500 per plant, is that out of the ballpark? So, so that's another thing is I can start to actually try to get some anecdotal evidence to see if that could support some of our empiric evidence.
So,
So completely dissatisfied with the analysis for today but that's that's probably for the best right because it shows, you know, what needs to be touched up and re-looked at next week.
But if any of you are interested, there's a lot more macroeconomics here to be had.
And then,
for next week, there's some interesting California data. So, you know, there's, you know, sales by county.
So we can start looking at, you know, an economic analysis of California.
So we don't have as many data points here, so it may be tough to make any, make any statistical determinations of any sort, but there's data to be had.
So, you know, until next week, I'm sure,
I may have left some of you unsettled with some of the analysis today. So if you've got any questions, you know, feel free to reach out and
we can continue to, you know, think about and puzzle out this data and
you know, if any of you have any inputs, definitely, you know, always open to, you know, new statistics that you may be interested in having talked about.
So, but, you know, it's a look behind the curtain, right, because it's,
you know, it's things don't always go as planned. And I think today was a good example of that. So, you know, you saw firsthand where we tried to go off road and calculate the daily rate of return and
it wasn't fruitful. And I do kind of want to apologize for that, but you can't always force an outcome, right? Right? If we tried, we tried dropping the missing observations.
And, you know, there's still more things that can be tried. So by all means, I would recommend to you repeat the analysis instead of using the total number of employees, use the total number of hours worked. So you can estimate that.
Yeah, we'll go ahead and jump off, but long story short, thank you all for attending and I hope you learned something and stay in touch and we'll pick up next week.
All right, all thanks again for coming. Thanks for attending the KAM State of Science Meetup group and until next week, keep your nose to the grindstone and keep having fun.