Welcome to Saturday Morning Statistics for November 13th.
It's going to be a really good day today. So we're going to pick up with some history of statistics because we want to do a nice introduction, a nice introductory lesson to statistics.
So that way we have a nice foundation for analysis. And of course, we'll be working with real cannabis data to apply some of these fundamental statistics.
And we can already begin making some awesome insights and inferences. So this is what's exciting. So you can do a lot with the fundamentals of statistics.
So there's a lot of people like to say low hanging fruit to be had. So you don't have to necessarily come at it with the most complex models that people may not understand.
You can come at cannabis data with simple statistics and inform people. So let's jump into it.
Want to preface everything with the fact that in this day and age, these characters in statistics are quite controversial.
So we're not here to really talk about the people or people's opinions of them. We're just here to talk about the ideas that they were they introduced because like it or not, they had quite an impact on the field of statistics.
And these are measures and metrics that we use every day.
You know, we heavily utilize some of these statistics. So it's interesting to know where they came about from.
I always find the history of mathematics, statistics, economics, all quite interesting because it's just interesting to see how the field developed because, right, you have these brilliant thinkers throughout time.
But at their time, things that we take as commonplace weren't yet discovered. So it's real interesting their approaches, right?
So they may do these real roundabout approaches to solve problems that we understand much more formally these days.
And we take as given, however, these people were just discovering them. So as you can see, you may recognize the name Pearson.
So Carl Pearson, a statistician, introduced the Pearson correlation coefficient. So this is a metric we will talk about today.
He also introduced the p-value.
So we'll talk about that today. Another common measure.
He was the first to formally do principal component analysis, which is utilized in the field today in the cannabis industry.
So you see chemists will use principal component analysis.
And dear to my heart, he created the first histogram.
And the histogram is probably my favorite type of chart.
So, well, you know, I'm a big fan of the distributions.
So I think they capture a lot of dimensionality with just one figure. So he was the first one to create a histogram.
And so he lived from 1857 to 1936.
So he was born more than 150 years ago.
So it's interesting to note that this is essentially when, you know, statistics was really starting to pick up steam.
Next, yet another controversial figure.
So if you want to read about the controversies about him, then hit the books.
We're once again primarily focused on some of the methods that were introduced that we'll be talking about.
Right. So we're always talking about variance and variability.
Well, Ronald Fisher, actually Sir Ronald Fisher, so he was knighted in 1952.
So he introduced the term variance.
And so variance, as we know it, is the sum of squares from the mean.
Well, actually, the variance is the square of the standard deviation.
He also introduced popularized analysis of variance.
And so this is where you can compare variances between populations.
And so this is where we can start with our hypothesis testing.
And so he also formalized what we now know as the hypothesis test.
So there were hypothesis tests before, but they heavily relied on alternative hypotheses.
And the mathematics was complex.
And so the single hypothesis, just having a null hypothesis that you can reject, was the idea was introduced by Fisher.
And it's a useful, simplified way to go about hypothesis testing.
And then the S distribution was not.
There was, it's interesting, while I was reading all of this, there is a current statistician, I believe, named Stigler.
We were talking about a Stigler in economics. This is a different Stigler, Stephen Stigler.
And he introduced a law.
I won't get the name right, but his law goes along the lines of any famous name in mathematics wasn't named after the person who discovered it.
So long story short, someone else really formalized the F distribution, but it is called an F distribution in honor of Ronald Fisher.
So long story short, these are two major proponents who really, you know, really formalized statistics into what became the modern day statistics.
So, you know, it wasn't really necessarily the field before these characters came along.
And, you know, these days, statistics underlies essentially every major field.
So, Carl Pearson and then his student Ronald Fisher from 1890 to 1962.
So also, you know, once again, we're moving on through history here.
So that's the history about where these ideas came from. Now, let's jump into it.
So I thought it was interesting that before these characters came along, no one had really formalized the idea of variance.
So what you would often do is just calculate means of different populations, and you would just see if the means were different.
Well, in some circumstances, and I'll just show you some today, the mean will be similar or the same across populations.
So here you have two different populations, red and blue.
They both have the same mean, 100.
However, the variance is different between the two groups.
So there's a lower standard deviation in the red, higher standard deviation in the blue.
So although the means are the same, this is where, you know, the inside of the histogram comes in, because you can kind of start to see that, you know, the tail ends of the distribution are going to be different.
And so this could have implications.
So, for example, for hypothesis testing, if you're trying to find if an observation lies outside of one of these distributions, well, say you have a value of 140.
Well, you can probably, and we'll get to it, you know, conclude that 140 is statistically outside of this. You know, we wouldn't conclude that that would be in the red population.
But, you know, 140 may be in the blue population.
And so, you know, you wouldn't be able to statistically conclude that 140 is not in the blue.
So long story short, the variance will have implications for hypothesis testing.
Now, we've introduced the idea of these two groups.
Well, actually, as we go along with this, let's go ahead and look at some of this with real cannabis data here.
Because, you know, that's what we're kind of all about. So, okay, so, right, so where can we find something where we may have the same mean, but different variances with cannabis data?
Well, let's start up a new console here.
Right. So, once again, working with Massachusetts data.
So, we're simply reading in this data here from Massachusetts.
And we're going to go ahead and define a couple periods here, right? So, we've got essentially when the industry went on pause and when it was resumed.
Because we've noted there's a gap there.
Well,
Okay, so long story short, the idea is, you know, to, right, we're trying to basically see, okay, are there two populations with different variances? And so the idea I was thinking was, you know, what if sales takes on every different, you know, underline, like, what if there's almost like the production function?
And so, you know, it's like the production function changes in some fundamental manner. And so it looks like, you know, you're going along.
And I even kind of want to compare going along, going along this period to this one, but we'll do pre to post pandemic first.
Well, not really pandemic, but more closure, because this was just, you know, the period where shops were closed.
But long story short, I figured we could start defining some of these variables here. So, how can we get these periods to have the same mean?
Well, what you can do is you can actually calculate the change. So here, you know, I've calculated the change in sales.
So you actually expect the change to be closer to zero, you know, if it was just a straight line, it would be zero, but maybe you would expect a slight positive mean.
So, for example, right, your mean change in sales is around 11 percent. So positive.
And so you would expect, you know, the change in sales to fluctuate around 0.11.
Well, you could, you know, calculate the change in plants.
OK, so I meant to restrict the time frame here from 2019 and onwards, because the 2018 data is partially missing.
So let's try this one more time.
OK, there we are. So here we have the change in plants and then the change in inventory.
So note today we're looking at total flowering plants instead of total tracked plants.
This is sort of an arbitrary change. I was just, I don't know, I started just have doubts about the total tracked plant count.
So I thought maybe the total flowering count may be a bit more accurate because it doesn't really matter how many plants you have in vegetative state, right?
Because some people may have tons and tons of vegetative plants and some may not really have that many.
So it's really more the ones that are like in their flower room.
And that kind of count. But the long story short, it's similar, but you can look at some various variables.
But we're just sort of picking some random variables today, right?
We're not really approaching this with much theory. We're just looking at rudimentary statistics here, right?
We're just looking at variance. So, OK, great. So these are like definitely two different populations, right?
Here you have change in sales and then here you've got change in plants.
So these are entirely different series. Change in sales.
Oh, yes, I wanted the mean of these.
Sales is around 11 percent. Change in plants is about 0.3 percent mean.
And then change in inventory is about 0.4 percent.
And so what does this look like?
Perhaps we could draw a histogram so you could see, OK, what does the frequency of these changes look like?
But here you see the distribution of changes in inventory.
And here you could look at the distribution of changes in plants.
So these distributions, you know, just on the face of it, look similar.
Let's look at the.
And here's the change in sales.
Right. And what's a little interesting about this is, you know, the change in sales.
The mean is higher, but it looks like, you know, it may have this sort of fat tail is what they call them.
OK, that's interesting.
Well, I figured we could do better than that. And a point and we're going to start getting into some new data here.
So there's a bunch of ways you can break this down by sales and we may come back and do this.
You know, we may return. OK, I'll just show it to you real quick. Long story short, I was saying, OK, look, you can maybe look at it by flower.
So this is the change in flower sales. So here, you know, here you just have weekly flower sales.
And, you know, you can calculate the, you know, the change in flower sales.
And then you could also, you know, calculate, you know, the change in processed goods.
So here, so basically, and this is where you kind of get into the art of data science. How do you want to break these goods up? Right. So if we're looking at the products.
Here, I've got a list of products. Some of them are similar and some of them are dissimilar.
So what I've done is first, I've just deleted some that I just didn't think were relevant.
So I'm not measuring waste. I'm not measuring seeds. Those are just not included in the analysis.
And then basically what I've done is I've separated these into two groups. Essentially, the flower type goods, which I determined are goods that I think a cultivator could produce and sell to a retailer.
So we'll have to, you know, check the rule book in Massachusetts. But from the naming, it would seem to me like a cultivator could sell buds, shake, raw pre-rolls, or trim to a retailer.
And then a processor to someone who set up a processing laboratory and is either processing the flower into concentrates, or maybe they have a food facility and they're just taking concentrated products and making edibles.
So this sort of may lump in dissimilar groups, right, because just because you're producing edibles doesn't necessarily mean you're producing concentrates and so on and so forth.
However, I just tried to have, you know, dichotomous, two dichotomous groups here. And so I just said, okay, these are the flower goods that cultivators will produce, your processed goods that a processor would produce.
And the idea is, is, you know, are flower goods growing more than concentrate goods? So like, as we saw, here are flower sales.
Then we can, you know, look at concentrate sales.
And, you know, we may note that, you know, overall flower sales.
I wonder if we can plot these two together.
Okay, so either something's going on, but flower, hold on, I plotted the wrong series here. Let's try this one more time.
Okay, there we are. So this is interesting here, right? So here you have flower and flower in blue and non-flower in orange. And they're actually similar quantities, right? It looks like flowers kind of broken away.
In other states, you see typically a 60-40 split with about 60% sales flower, 40% other. At least that's what we were observing in Washington state.
So we would be interesting to, right? So we could even, this is what we're all about, calculating new statistics here, right? So we could calculate the ratio here of, you know, flower to concentrate.
Or we could say, oh, what's, hold on, let's do this out of the total. So let's see what flower...
Hold on, I don't think I did that right. Because basically I want to see what is flower out of the total.
Okay, so this is real cool. So right now we just calculated the percent of sales that are from flower.
And so as you, as you know, I conjectured, similarly to Washington state, you see a similar breakout where flower stabilizes at around 60% of sales.
And we can even say, you know, what's the mean of this? Well, it's not going to work well because of this break.
But we could say the last year. Okay, so this is interesting. So in the last year, flower has been around 54% of sales.
So that's interesting. So Massachusetts may see a higher proportion of other goods sold than in other states.
And as we've noted, this may be because of income. So as income rises, you may see people spending more money on edibles and less money on flower.
So already we're starting to uncover ways where populations may be different. Of course, the means may be different, but the variances as well.
So, so those are some interesting ones. And then just to show you these remaining statistics.
This is just the percent change in flower pre-pandemic and post-pandemic or pre-closure, post-closure.
And so this is where you can see, oh, you know, are these, you know, are these histograms different?
So here I was, you know, trying to recreate this chart. But as you can see, you know, these distributions overlap pretty, pretty strongly.
So quite a bit of overlap there.
And then I was saying, okay, you know, you could also look at concentrates, you know, pre-pandemic or pre-closure, post-closure and similar thing, you know, is the like has the change in concentrate sales.
Like is that systemically different? And, you know, it doesn't, you know, on the face of it, you know, appear, it doesn't on the face of it appear to be different.
Right. You know, those distributions look quite similar.
Well, you know, that's just plotting.
But let's try to open this full screen again.
Well, we've now noted that we can actually quantify this.
So if we are given samples, samples of X and samples of Y, we can calculate the sample correlation coefficient
to see how correlated these samples are with each other.
And I think we can do that here in
the next part. So
I've already had this pulled up for some reason. I thought I had this one ready.
Let's just try some of these out. These ones I haven't tried before. But ideally, I think this would be a visualization of the
correlation here. Unfortunately, we may have to make this correlation matrix.
Okay, yeah, this is maybe something that we may need to
do. My apologies. I should have had this one pulled up.
I'm going to leave this in.
Okay, let's try this real quick. If it works, it works. If it doesn't, we'll move on. It would just be interesting to see, you know, what are some of the correlations between, say,
some of these change in sales.
So let's just see if we can't correlate pre-change in concentrate sales and pre-change in flower sales.
I think it's going to be because of this NAND.
Well, we can probably just get around that. Let me try this one more time and then we'll move on because there's more interesting statistics we can get to here.
Just skip this first observation. Okay, awesome. We were able to do it. So we just calculated the correlation coefficient here between
the pre-change in flower sales and the pre-closure concentrate sales.
As you can see, they're moving along and we can say that they are positively correlated.
And this measure ranges from positive one to negative one. So if it's one, then they're perfectly positively correlated. If it's negative one, then they're perfectly negative correlated.
Zero, they are not correlated. And so here you had 0.6. So you confirm that you would say these are moderately positively correlated.
The general rule of thumb is anything greater than 0.7 is strongly correlated. Anything less than around 0.3 is weakly correlated.
And then anything between 0.3 and 0.6 is moderate. Those are my rules of thumb. So a little bit of a panic, but we were able to calculate this statistic.
What's cool is, well, we can actually look at the correlation between these posts.
So let's see if...
Look at this. Post.
Does that make sense?
Let's visualize that. Look at that. Post.
Post closure, concentrate sales and flower sales are almost perfectly correlated.
Isn't that bizarre?
Actually, that is quite bizarre. I wasn't expecting that. That's quite a big difference there.
And so here we're starting to uncover some of these nuances.
And so, like, I mean, think about it. We've taken the most rudimentary statistic here, the sample correlation coefficient pioneered by Carl Pearson, you know, in the early 18th, I mean, in the late 1800s, early 1900s.
We've taken this old statistic, rudimentary, and already we're looking at this and we're saying, OK, like, look, there's only a 0.6 correlation between flower sales and concentrate.
Well, this is actually the change, the change in sales.
So, you know, for example,
you know, you could look at the correlation coefficient here.
Let's not get into it. It's going to be a whole can of worms.
But anyways, there's a lot of variables here that you can look at the look at their correlation. And just so you know, time series data like this tends to actually be positive, tends to be strongly correlated.
So, for example, like GDP and a lot of other of these economic variables tend to be strongly correlated with each other. So it's not that surprising that, you know, the change in flower sales and the change in concentrate sales would be highly correlated.
But I just think this is shockingly high, not like 90, like in like 97, like a coefficient of 0.97 is high.
You know, like I said, 0.6, like that's a bit more like that's a bit more typical.
You would expect. But like I said, what's typical? We don't even know what typical is here in the cannabis industry.
And that's why, you know, we need to start looking at these measures state by state, right? Because then you can start looking at the analysis of variance, right?
Maybe red Washington, blue is Massachusetts or what have you. So we can start looking at these two different populations here.
So here's, you know, one way to look at it, the Pearson correlation coefficient. And as you can look at the data, right? These series are kind of static.
You know, you know, it's a little bit of white noise here. And then here, yeah, I mean, they're kind of sitting on top of each other.
So that's an interesting, an interesting observation. But like I said, it's just correlation. You know, we can't really read much into it other than it's, you know, this is just statistical correlation.
So awesome work here. Let's just document that we did this.
Flower and pre and post. So let's just save this work over here just so we can come back to it if we need it.
Awesome.
Well,
let's go ahead and get into these next metrics here. So I just want to finish the presentation real quick. And then I'll show you some brand new work that can be done with these with these measures.
Okay.
Okay, back to full screen.
Okay, so Fisher introduced the analysis of variance.
Right. And so this is a statistical test where I should have
given a screenshot of the formula here.
So to cut the long story short, it's a test where we can tell if two or more populations are different.
Right. And so, you know, essentially, we'll be able to tell if, you know, these two populations are different or not.
So let's go ahead and bring up
this may be out of place, but I think it needed to be mentioned before the end of the day, which was essentially the hypothesis testing and the types of errors you can make.
Right. So we introduced the
test scenario, which is
dependent on your significance level. And so what's your significance level? Well, that's where we are basically want to state what our probability is for making various types of errors.
The general idea is, right, we've seen the distribution and the idea is it may go to negative infinity and positive infinity.
So, you know, we can never be, you know, 100% sure of really anything. And then I guess that's sort of, you know, almost then, you know, where statistics starts to get to merge into worldview.
And this is where, you know, frequentists and Bayesians, they start to butt heads and
the discussion gets a little philosophical and may even get over my head a bit. So I'm more of a practitioner. So, you know, pardon me if I don't convey the theory 100% accurately or formally.
But I'll just convey it the best I can, the way I understand it and the way I use it.
Essentially, there's various errors that you can make when you're doing a hypothesis test. So, for example,
the idea is you to set up a null hypothesis and then try to reject it. So the null hypothesis in this case would be these two groups are identical.
And then you would reject your null hypothesis if you can provide substantial evidence that these two groups are different.
So,
I really should have had the formula pulled up here. But long story short,
and I think that's where I may pick up. I think I'm going to pick up next week with a formal formula for analysis of variance, but we've laid the groundwork today.
But the idea is if you conclude that they're the same, so we don't reject that they're different.
Well, in reality, the two groups, well, they're either different or they're not. Right.
And so this is sort of where you get into the philosophical argument, but we'll just take it, you know, theta, our parameter here as given.
So in reality, we're saying there is a true parameter here and that the parameter does have a true value and it's either the groups are either the same or they're not.
Well, if we say they're not different and they are, in fact,
not different,
then that's what we call a true negative.
If we say they're not different and they are different, that's a false negative.
If we say they are different and they aren't, that's a false positive.
And then if we say they are different and they are, in fact, different, then that's a true positive.
So a lot of picking your significance value, your significance level,
you could argue that you would, coming from an economics point of view, you would essentially weigh the costs and benefits of making true negatives.
I mean, you would basically weigh the benefits of predicting true negatives and predicting true positives,
and you would compare that to the cost of predicting a false negative and the cost of predicting a false positive.
And in different areas of life, these costs are asymmetric.
So I think this is a really good starting point when you're determining your significance level here,
is you kind of want to start thinking about the outcome.
And this is where we started talking about the rules of forecasting, you know, acknowledge your forecasting error.
So here you want to acknowledge if there's a symmetric or an asymmetric cost to your error.
I'm trying to think of an example.
Okay, let's say we conclude that the change in sale and change in flower, they are the same, but they're actually different.
You know, people may stock the shelves wrong.
So the cost there is not readily apparent.
I'm more thinking more in like health care and diagnostics.
That's where it'll have a big effect. But long story short, this is where you set your alpha.
And so I would say, okay, you know, maybe you set your alpha either at 0.05 is what Fisher actually recommended.
So this is where you know, you're predicting, you know, the true negatives 95% of the time and you only have a false positive 5% of the time.
So you can actually, you know, do the math and, you know, if you can put a cost on your false positive, you can actually do the cost benefit analysis.
You know, if you can put if you can measure everything, if you can put a cost on the false positives, put a cost on the false negatives.
And so on and so forth.
But I won't bore you to death with this.
And then we'll resume with the NOVA next week.
But I would just say, so this is, you know, the very rudimentary framework of statistics that Pearson and Fisher introduced.
And then these all lay the groundwork for modern day statistics.
And these are models that we've already been using the fixed effects model, which is a type of analysis of variance, where we're basically assuming that there's been a treatment applied to one of the groups.
You can make this.
You can treat the treatment as random with the random effects model. So we may get to these two models later down the road.
I think next week we'll resume with analysis of variance, and I'll do this a bit more formally.
Just to end on some real world applications of this. So when would we be doing a hypothesis?
And when would we be analyzing the variance between populations here?
Or even testing if the means are equal?
Well, I promise that this data point would be interesting.
And so without further ado, let's start looking at it.
So the parameter I wanted to look at was the square footage of the licensees, so we can start seeing what the size of these operations are.
So for example, one of the things, metrics people are always bandying about is what is the square foot required per plant?
So we could actually measure that on average, right? We could look at the total number of cultivators, add up all their square footage, and then look at the total number of plants.
And then so we can just divide plants by square footage of cultivators. So that's an interesting metric.
We may look at that on Wednesday in the cannabis data science group, since that's more cannabis data science.
But for Saturday morning statistics, well, we can look at the statistics.
So if we just look at the square feet per licensee, we see, okay, you know, there's around nine, there's about 900 licensees.
This average square foot is 30,000. But check this out, 64,000 square foot standard deviation. That is a big distribution.
So, you know, so here's, you know, essentially our distribution of square feet in their establishment.
And as you can see, you know, you have a lot. It's almost a log logarithmic distribution here where you have clumping around zero.
And then you've got this, you know, long tail going way out.
So there's a couple ways you can look at that. So that says a whole, but I figured, well, not all of these businesses are probably the same.
Right. So, for example, you know, we could look at the mean by all of the different types.
So here we have the, you know, the average square feet.
Let's just round this just so it's something that we can look at without having a headache.
Okay. Actually, we can just round this to the nearest foot, right? We don't need.
Anyways, so it looks like we can kind of skip the craft cooperatives. Maybe there's, they aren't issued yet.
So as you can see, transporters, they probably don't need much more than an office space.
Laboratories, right? So this is, you know, I think this is going to be interesting, right?
What, well, I may be biased here, right? Operating analytics. We help out a lot of laboratories.
And so it would be interesting that I can know, okay, you know, what's the distribution here of, you know, the square feet needed for a laboratory.
So if someone asks you off the top of your head, like, you know, how many square feet you need to run a laboratory?
Well, you can tell them, you know, well, let's find all the licensees where their license type is an independent laboratory.
Right. And here are all the different, you know, observations here.
So you have some that are real small, right, around 800 square feet.
That doesn't seem possible, but maybe it is. Maybe they're growing here.
And so, you know, so you have you have them all over the board, right? So you have some that are like 1600 and you have some that are 16,000.
You know, 16,000 is on the high end there, you know, and so once again, our handy dandy histogram.
That's the best way to visualize the data with the histogram. But long story short, it appears, you know, there's different square feet.
You know, we could you could even do a, you know, a conditional variance, right, right. So what we can calculate the variance, variance in square feet.
You got more.
I guess we hold on. I could have just done it this way.
Should do this. All right. Cool. So, you know, so now we can see the mean and standard deviation for the different types.
Right. And so it's going to be interesting to compare these. So, you know, the mean for the laboratories is 5000 and their standard deviation is 4000.
So, you know, we could do this.
So,
So, you know, the long story short is it looks like there are differences in square feet, depending on the license type. And so this gives us a nice opportunity to do it in analysis of variance and conclude and see if we conclude if their sizes are statistically different.
For example, you know, can we conclude if the cultivators at 55,000 square feet on average, you know, can we conclude if that's statistically different than manufacturers who have 40,000 square feet on average.
We may not be able to because look how high the standard deviations are. So the standard deviation is quite high for both manufacturers and for cultivators.
Right. But look, you know, for retail, you know, the average is around 8000 and then you only have a standard deviation of about 20,000. So if you just use Fisher's ballpark of okay two standard deviations.
Well, two standard deviations away from the mean here is around, you know, 48,000 square feet.
So if somebody told you, you know, they had a, you know, a 50,000 square foot facility or 100,000 square foot facility.
Well, you could probably, you know, you do a hypothesis test, and you could probably conclude that you are more than 90, you know, you can say, okay, I'm 95% sure that that person is not a cannabis retailer.
So you could start to, you know, predict license type based off of the square foot of their facility.
Why would that be useful? Well, maybe you are in
the formal name is skipping me.
Which the branch of the economy where they buy and sell homes, real estate, sorry.
It's getting on there. So for example, if you're in real estate and somebody approaches you and they want a 50,000 square foot facility. Well, and you know, they're a cannabis licensee.
Well, you may be able to conclude that they're a cultivator and not a retailer, just by knowing how many square feet they're asking for.
That's sort of maybe not the best example ever. But, you know, you know, it could be useful, you know, maybe they're asking for 35,000 square foot facility.
Well, you could find the probability that they're a manufacturer. You may be able to conclude if they're a manufacturer versus a cultivator.
So,
so long story short, there's definitely differences of means here. There's definitely differences of variances here and we're going, we're, you know, starting with the basics to quantify these.
And next week, I'm going to continue with an interesting application here with licensees and square feet per facility. So there's a particular
ANOVA test we can do that just like Pearson and Fisher, we can be controversial. So I'm going to plan it out well for next week and
prepare well for next week. That way we're not having to do things like look up these correlation coefficients on the spot. We were lucky and we're able to calculate those. Don't want to have to get lucky with this. So I'm going to prepare the ANOVA analysis for next week.
And we are going to do a quite interesting analysis on licensees square feet.
We'll incorporate license type.
But we're principally be looking at square feet and another variable
that, you know, so we'll also be using, you know, another variable
that's provided with the licensees. And so we'll pick up next week with hypothesis testing.
We'll be looking at groups that may have different variances or different means and see if we can't
perform hypothesis tests to say how confident we are or not that the groups may be different. So, you know, next week, we'll, you know, start looking at
square feet conditionally and we'll perform some ANOVA tests and I will include these statistics, the formal formulas that I should have included this week. I'll go ahead and type those up and include those for next week.
So we're well underway to getting up to speed with modern day statistics while building up a strong foundation with the fundamental statistics here.
So. Going to go ahead and pause it for now until next week. So if there are any questions I am, I'd be happy to field any.
Yeah, Keegan, can you hear me? Yes. Ah, here we are. Can you answer my questions in the chat?
Okay. Skewness and kurtosis. So.
Essentially, these, if we're talking about the methods of moments,
these are the statistics that you gather from a particular group, a particular sample population. So the first would be the mean. So if you're given it right. So we saw given this sample, we can calculate the mean.
The next moment is your variance. And that's where we saw, you know, how much variance that there is. So either how tight the distribution is or how wide it is. So that's more width.
Skewness is the next moment. So it's if you should read research method of moments, but it's essentially.
You know, I don't know the formula, the formal formula off the top of my head, but it's it's similar to variance where, you know, these are statistics that you can calculate from a group of data. Just like given a sample, you can calculate its mean.
You can calculate its variance. You can also calculate its skewness. And that is basically the thickness of its tails. So remember, today we were talking about.
The change of sales, having a thick tail, like having one side had a thick tail and the other side had a thin tail. That would be skewness.
And kurtosis, don't quote me on this, double check, but this is my understanding. Kurtosis is almost like the way I feel about this, the way I picture it is almost like the curve getting like a little like bent to the side.
So, you know, you don't have quite the normal distribution curve, your curve gets a little tilted towards one side.
And these are things, these are like we saw with the visualization. These are aspects that wouldn't really be captured by a mean and skewness wouldn't even necessarily be captured by a variance because you can have samples that have the same mean.
They have the same variance, but one has thick tails and the other has thin tails. So it's possible to just vary on skewness and kurtosis alone.
So these are just sort of higher order ways to classify data versus a mean.
So it's all sort of talking about the how the distribution looks. That's how I would describe it, just more characteristics of your distribution.
That's an incredibly informal explanation.
Spearman correlation, okay, and spurious correlation. Okay, spearman, I'm going to have to research.
I don't know that off the top of my head. Spurious correlation, is it real? So that's essentially where we were talking about with the time series, where we may have two different time series moving along.
If you did a regression on the two, they would...
The regression would look as if one variable was highly explanatory, but in reality they just move in similar ways.
Trying to think of good examples here.
I think it's just really just any time series.
I'm having a tough time because the way I think, I always think about all these factors as being dependent on each other. So I'm having a hard time thinking of any two factors that are independent.
So I'm having a tough time thinking of an example, but the idea is if you have two time series,
that's why it's often useful to take the difference and look at the change in the growth rates like we were doing today.
Because if we had done a regression of sales on plants, they're both just increasing over time, and we may have a spurious correlation where they may not really be correlated with each other.
Long story short, you can sum that up by saying correlation does not mean causation.
Just because you have correlation doesn't mean you've proved causation.
And the final question, if means are equal.
They may have different standard deviations and variances.
And so as you hinted at in your question, you can't really conclude that these two groups are the same.
So it depends on for what purpose. If you're only concerned about the mean, then you could make your decision and just say, OK.
So this is the classic in the stock return, the stock market.
The efficient market hypothesis is that nobody can make a return on average in the long run greater than 0%.
So you may see somebody make a 20% return, but then they may also make a minus 40% return and a 5% return.
And so then you may see someone else, and they're just doing 1% return, minus 1% return, 2% return, minus 3% return.
So the idea is the means are the same. They both are getting 0% return.
But you may actually care about the variance. So in the stock market, variance is risk.
So you people have a preference for low variance.
So if you had two traders and they both had a mean of 0, but one of them had a high variance and the other one had a low variance,
you would prefer the trader with the low variance.
And so this is actually taking into consideration in the stock market.
So say you have two assets. They both return a, let's just say, a 2% yield or say they both return a 5% yield.
And that's common for assets to return a similar yield on average.
Well, then you actually have to look at the variance. What's the variance?
What's the range of returns that you could get?
And so maybe they both return 5% on average, but one has a greater variance than the other.
Well, the risk adverse trader would, economists would argue everybody's risk adverse, would favor the bundle with the lower variance.
So that is an actual example where you've got two groups.
They've got the same mean, but you clearly have a preference for the group that has the lower variance.
So it's circumstance dependent.
And then survival functions.
I don't know as well. This is more like how long things will survive or go unchanged.
Maybe. Well, like I said, all of these things are kind of built upon each other. So I'm sure, you know, in its own way, survival functions are built upon analysis of variance.
But I'll have to do some homework to connect the dots on that one.
So, so hopefully you're welcome, Cheyenne. Hopefully that answers your questions.
As I said, I can do a lot more homework myself on statistics.
And that's what's, that's what's awesome about Saturday morning statistics is hopefully you can learn a thing or two.
And then I, I learned a thing or two myself from preparing because, you know, sometimes I need to knock the dust off of some of these statistical concepts that I take as granted that we need to formally define and prove we can do well.
Because that right, we have to start with the good fundamentals. And that's, that's what we stress here at the Cannabis Data Science Group is, you know, you got to walk before you can run.
And so we, we're here proving that we can do simple statistics well with these, with cannabis data.
So that way we can build up trust and confidence in ourselves. And we can make some, some good insights because as we showed today with simple statistics, you can have deep insights.
And that's where we're going to keep delivering.
Awesome Cheyenne.
Thank you for saying a little extra long longer today.
Next week we'll pick up with analysis of variance and keep building upon it. Because like I said, we can eventually build upon it to the point where we get to fixed effects models, random effects models, we'll look into survival functions and see if we can't tie those in.
And we're going to keep, keep making some awesome insights. As I teased, we've got some real interesting work next week. So definitely tune in.
I think we're going to have a good time and make some good discoveries.