Right.
Welcome to the meetup for the 31st.
Got a jam packed day for today.
So I was going to introduce to you forecasting.
A book that I'll want to share with you is economic and business forecasting by john Sylvia and.
Professor equal as Eric as Eric.
Last week we talked about Colorado data and so there's been a lot of talk about what may happen in 2021.
So we're going to build a forecasting model.
First.
We need to take into consideration that, you know, we need to, you know, maybe have a set in stone methodology for forecasting.
So we're going to follow these 10 steps.
Number one, we need to know what we're forecasting.
Number two, we need to understand the purpose of forecasting.
We need to acknowledge the cost of forecasting.
We need to rationalize the horizon.
We need to know what variables we're using.
We need to know what.
Forecasting model we're using.
We then need to know how to present the results.
We need to know how to how to decipher the results.
We need to iterate over time.
Using recursive methods, so we need to not just leave this. We need to keep working with it.
And then we need to understand that the model may evolve over time.
So we may understand we may figure out new and better ways to forecast.
Just.
Well, well, actually, I come back today to the statistics here in a bit.
So.
Sorry for the rocky introduction.
But.
We'll just go ahead and start here. So knowing what we're going to forecast.
So we've been working with the data in Colorado.
So.
Last week, we showed how given sales.
So given sales.
Implants.
We were able to.
Calculate a production function for Colorado, and we were able to estimate.
A competitive wage and the real rate of capital in Colorado.
So those were historic. So if you wanted to set wages or invest in 2021, it would be useful to know what you could expect.
The minimum wage or the competitive wage and the rate of return of capital to be so.
That will bring us to step two, understanding the purpose of forecasting.
To get better expectations for the size of the Colorado cannabis industry in 2021.
Let me pause.
Real quick and get my coffee and then we'll move on to step three.
All right.
So.
After all.
We're doing.
So.
All right.
So bearing with me.
We're now going to understand the forecasting error.
So after all, we're using this to predict the number of plants in the number of sales. So if we overstate.
We may lead to full hardy decisions.
But if we understate.
We may leave money on the table.
We're just going to rationalize the forecasting horizon.
So we're just going to estimate 2021.
We think 2021 should be informative.
Any time frame less than that.
It may not be worthwhile.
And then any longer time frame.
We start to lose credibility in our forecasts.
It may be worthwhile to forecast 2022.
Anything beyond 2022.
You start to have a good bit of skeptic.
You start to approach with a good.
Lens of skeptic habit.
Next, we want to understand our choice of variables.
We're simply going to be.
Using sales.
And plants.
And under the principle of never throwing away data.
We're going to be using the time frame of January 2014.
June of 2020.
To introduce the forecasting model.
We've been.
Working with regressions.
So this is actually an.
A theoretical model.
That utilizes your standard regression model.
So we're essentially saying.
That sales.
Or plants.
In time period T.
Depend on sales in T minus one.
And they may depend on sales in T minus two.
All the way up.
To a certain period.
So.
This is.
Not theoretical, but it's a useful way of modeling data.
And it actually can be quite useful for forecasting.
And then the moving average process is essentially saying that the error.
Also depends on errors from prior periods.
And we'll approach that with our statistical model.
As always, we'll want to show the data.
This best way to do that.
If you've.
Know of Edward Tuft is a figure.
And we'll want to look at the forecasting results.
We'll want to look for seasonality.
We may want to look at the totals.
So anything of interest.
Of course.
You want to use recursive methods.
So.
After we make our forecast.
In several months time.
We'll want to come back and.
Double check the forecast.
See how far they were off.
And perhaps make new forecasts for the coming year.
So.
Perhaps in six months time.
We can double check the forecast.
See how far we were off.
And try to do better next time.
And.
That's.
Takes in that leads us into principle 10.
Understanding that the forecasting models will evolve over time.
For example.
We may see that there's seasonality.
And think of ways to control for seasonality.
Or introduce.
Seasonal effects.
So.
There's a quick dirty presentation.
And so now let's go ahead and.
Go ahead and.
This data that we've.
Collected from the Colorado.
Reports.
These are quarterly reports.
So currently we only have data through.
June of 2020.
But we have data on.
People working.
We have the number of plants.
We have total revenue.
We have some.
Interesting metrics here.
So.
First things first.
We're going to go ahead and use our.
Good friends pandas.
And we're going to read in the data.
Always look at the data.
Look sound.
And so now I will introduce you to a.
Forecasting model that I wrote several years ago.
So.
We are essentially.
Using.
Arima.
From.
Stats model.
And Arima will calculate.
This model for us.
So we're essentially estimating an ARP.
Plus.
Right so we're estimating.
Essentially an ARMA.
Plus.
B.
Q.
So we're essentially combining these two processes.
Sorry for that dirty.
Writing.
How are we going to pick the best model?
And so I wanted to introduce to you the concept of.
Root mean squared error.
So here.
We take our actual.
Predictions.
And what's the predicted.
And you would sum that over all of the periods where you made predictions and you have actual events.
Actuals.
I mean how do we cut.
How do we calculate a root mean squared prediction.
If we don't have predictions.
So essentially we're going to utilize a holdout period.
So here we're going to have a holdout period of.
A certain amount of months.
So looking at this data.
It seems relatively arbitrary.
But I think.
A holdout period of six months.
Maybe.
Maybe a interesting span of time.
So you could essentially use all of the data.
Up to.
December of 2019.
And then forecast.
The first six months of 2020.
And then whichever model.
Forecasts.
The first six months the best.
You'll then use that model to then go ahead and predict.
All the way.
Through.
You know.
Through December of 2021.
So we're going to make forecasts.
For the next 18 months.
So just to kind of give you a brief of this code.
Just to kind of show you what's going on under the hood.
So.
Nothing fancy.
Essentially we're just taking a time series.
We're separating it into a training period.
And a testing period.
And then that's just going to be by the holdout period.
Then we're essentially going to scan.
So.
If you go back to the presentation.
You'll see that.
He and Q are arbitrary.
So you can have an arbitrary number of.
Regressors.
Auto.
Auto regressive regressors.
And you can have been however many.
Moving average regressors.
So.
We can't.
Calculate an infinite number of models.
So we're going to limit our scope of models.
So we're essentially going to limit the lag order.
To perhaps six.
So we're essentially you can do more.
This morning I tried it with 12 and it just.
Wouldn't estimate.
So.
The more data you have.
You can typically use longer processes.
But you can also use some rationality.
And.
You know.
You may not want observations from that long ago.
But you can actually.
You know using to predict what's happening in period T.
You can use a mix of common sense plus what actually forecast the best.
We're going to limit our.
Scan.
Of Arma models to six.
So now we're basically just going to loop.
And we're just going to say okay.
So now we're going to show for.
Q.
We essentially just create an Arima model.
And this is actually does a rolling forecast where.
You you you enter you.
You you'll use the your your forecast to go ahead and predict a new model in each iteration.
Once you've estimated.
A Rema model.
For each P Q combination.
So.
Keeping in mind that if.
So if even if you're only doing it to six.
Lags.
If it's still is an unstable model.
Then we're essentially ignoring it.
So we're just ignoring unstable models.
And keeping the forecast.
And the root mean squared errors of.
Successfully fit models.
Using.
The.
Root mean squared error of.
The.
Handful of a Rema P Q's we estimated right so we're basically estimating.
Right so we're essentially estimating great like a Rema.
Like zero zero one.
You know are my one zero.
Or my one one.
So on and so forth.
You're then going to see.
Which of these models.
Predicts.
These variables the best.
So does the arm.
Zero one is on zero two does the arm of one zero.
Which one of these predicts total revenue the best.
Or our holdout period.
And then we'll select that model to go ahead and predict the next 18 months.
So.
We've identified the minimum.
Mean squared error.
And then we're going to use that model.
To predict the.
The forecast period.
So is this like a grid search.
It is.
It.
It's essential it is a grid search.
It's and.
You could make it more complex.
You know today.
For you know this is.
Just scratching the surface of what you can do.
So this is.
What you would call an auto.
Or an auto well this isn't even a.
This is an arm.
So this is an auto.
So you're essentially scanning.
You're like you're doing a grid search.
Of a handful of models.
And then picking the best one.
Just to kind of so that's the logic.
That abstracted away and actually use it.
So we're going to be using a lag order of six.
We're going to be using the six holdout periods.
And then we're going to forecast 18 periods.
The head.
I wonder if we could print out.
Oh just let it run.
Right.
So.
Here we go.
To hear we're fitting a bunch of different models.
You'll see that.
This maximum likelihood.
So the map the model.
The models being.
It by maximum likelihood.
So.
Seeing some error statements being printed out.
When the model fails to converge.
And that and that will happen if you've essentially.
Overfit the model.
So if you try to pack in too many regressors.
So for what they call high order.
Armors.
So like an arm of six six.
With very little data.
Would likely fail to convert.
And so see here we're getting into these higher order.
Arima's.
Like a rema four six.
So.
So it looks like these higher order.
Arima's are mostly sailing.
So.
Well.
And here's our final.
Arima six six.
Arima's are mostly sailing.
Looks like it's having a little difficulties.
All right. And so.
After all that rigor moral.
We got our best model.
Arima four six.
So.
We're basically saying that we think.
That there are four.
Auto regressive terms.
And six.
Moving average terms.
Or regressors.
And then we're going to go to the data.
And rule number one.
Go with the data.
And then we're going to go to the data.
Are forecast for the next 18 months.
And.
Well, that's maybe not the best way to do it.
I would create a quick.
Yeah, there is a, you know, a really slick pandas way to do this, but I can't tell you off the top of my head.
I'm not that familiar either.
But that's essentially.
Brings us to.
Knowing how to present the results.
And now we need to get this into a decent looking figure.
Okay.
So is this.
So this is like an ensemble.
You have a series of regressors.
Right.
And so you're.
You're taking.
You're coming up with a final result.
And then you're going to put in a weighted average of them.
So it's kind of like a.
Yeah.
We're waiting them.
So this is like an ensemble of regressors.
So here.
Just to show you a bit more detail of what's going on here.
Now that we've we've we've decided, okay, this Arima four six is the best model.
Let me just show you what an Arima is.
I think.
I think that would be a light.
Lightening.
This is a four.
Let's see if we can't fit this model real quick.
So then you can print model summary.
Okay.
So this is.
This is the regression that we ran.
So.
We have 78 observations.
We have a constant.
We then have.
Four.
Of these moving average.
I mean of these autoregressive terms.
And these are those their coefficients.
So.
You see a positive coefficient on beta one.
That would indicate higher sales in the previous period would lead to higher sales in period T.
However, you can't really.
Remember.
The pox Jenkins methodologies a theoretical.
So you really can't put too much interpretation on these coefficients.
But essentially what you have here are your your four.
Autoregressive terms.
And you have your six.
Moving average terms.
I can maybe send you some material on how to calculate the moving averages.
It's.
I'm still a bit more involved, but it's.
So I'll want to.
I'll want to get some material on moving averages and add those to the repository.
Do you still have.
Does that answer your question or or you still have the question.
Yeah, no, I see what's going on.
I mean, it's sort of like.
It's actually more like a neural network where the output is.
It's a weighted sum.
Of these outputs, right?
These autoregressors.
You could think of it.
You could think of it that way.
If you want, if you essentially if you want to.
Think of beta as as your weight.
But keep in mind.
It's almost more of like a magnitude.
Versus a weight, right?
Because some of these may have negative coefficients.
Right, because if it was just weighted, then they would all have a positive effect.
But, you know, some of these may have a negative effect.
Some may have a positive effect.
You know, some may have a small, some may have a large effect.
Okay, yeah, I get the gist of what's going on.
Luke, that's awesome.
And so.
This is, you know, it's it's a little bit of an ad hoc way to forecast.
But.
You know, as you saw.
I'm just fitting this whole thing in.
Well, here, let's let's just do the same thing, but maybe with with plants to do a smaller.
Here. Yeah, in the remaining time, why don't we just do it with plants and then add the add the time series so we can actually interpret our results.
Because we still have a good 15 minutes.
Okay, now.
Let's reduce the lag order to three, just to make things more manageable.
And then while we're doing that, I'm going to do a quick Google search for how we can add a.
Two things when we need to add time index to a panda series.
And then to we need to create a time series between the two points.
Let's see if there's not.
Okay, so that's what we want.
Bear with me, but this is this is how you get.
This is how the sausage gets made sometimes.
So we'll do total revenue for the time being.
And I'll refactor this.
The leader point.
Yeah, you have to make like a list of the dates that you want as the index.
And then when you create the series, use that list, tell it to use that list as the index, but I can't tell you like the exact syntax off the top of my head.
I would have something I would have to look up to.
It's one of those things where I think I've coded it up before.
So instead of digging for an old snippet, I think I'm going to see if we can't just code this up real quick.
So apologize for making you bear through this.
I know that's fine. I do this all the time.
I can't because we're basically we're just trying to go until 2022.
Essentially, let's see if.
That looks like it actually looks like the index we want.
That's it.
Not so painful after all. See, this is why.
And this is just a beauty to work with.
So now we've just in a matter of five minutes, we added the time index on there.
So now we can actually.
Move on to.
Now we actually have presented the data in a barely acceptable manner.
So now we can actually decipher the forecasting results.
So we're essentially forecasting. It looks like monthly sales.
I want to say that.
I guess we could maybe even plot this.
Wonder if we could plot this in millions.
OK, awesome. So now I've essentially plotted this in millions of Canada sales. So you'll see that we forecasted sales will dip.
Down in October and January.
So that's about 165 million a month.
Increase during the summer and peak at around 195 million dollars a month in Colorado.
So that is.
You know, quite.
A lot of revenue. And so just to remember, so if we hold out the first six months, because that's 2020.
You know, you're looking at about two billion dollars worth of sales forecasted in Colorado in 2021.
So.
So that's.
You know, so that's how we essentially.
You know, we'll want to now use recursive methods. So now hold me to that. So now, you know, so now mark my words on March 31st.
2021.
We predicted that in 2021.
We're looking into the future.
We're predicting that there's going to be two billion.
2.1 billion dollars worth of sales in Colorado in 2021.
So now.
But it's a we play the waiting game.
So now we'll wait.
And as this data comes out.
We can double check our forecasts.
And we can wait till the year's over.
You know, plus time for them to get the data out.
And we can see if.
You know, this 2.1 billion dollars was that far off.
And so how do we do that?
We'll use our root mean squared error.
So it's nicely defined. We'll simply take the actual.
And 2021.
Minus our predicted.
So we'll actually have a measure of how well this forecasting model.
Predicted.
And that brings us to the final step.
You know.
Understand that this can evolve over time.
So remember, we.
We estimated the.
You know, the Arima.
Four six.
Maybe next time we won't even use Arima.
Maybe next time we'll just use a theoretical regression model.
Or maybe you could compare it to your manager's rule of thumb forecast.
You know, there's a bunch of different mechanisms of forecasting.
So you could use an entirely different model.
Or you could just use a slightly different Arima model.
Or you could use the exact same Arima model, but with more data.
Less data. A longer training period.
A shorter training period.
Less memory. So you may not want to include data going all the way back to 2014.
If you think there's been a structural change.
So.
There are a lot of, you know, there are a lot of toggles.
That you can adjust on your forecast.
So.
That's why it's an iterative process.
Because this is our initial forecast.
But there's always room for improvement.
So that brings us nearer to the end.
And so I'm going to go ahead and I guess turn it to questions.
Do you have any questions about this quick and dirty forecasting that we've done, Charles?
No, no. This was really good.
I'd be interested to see, you know, like some other kind of models.
How they would predict versus this.
But yeah, this was interesting.
You know, that's kind of my whole thing right now is I understand the programming end of it.
And I can wrangle the data all I want. But like what you do with it.
Is what I'm having. It's kind of what I'm struggling with or trying to learn.
You know, and so these are good. These are interesting techniques.
And these are the things that I don't know.
Well, I'm awesome too.
It's awesome to hear that, Charles.
And, you know, I just sort of want to share those things with you.
Because a lot of these are, you know, they're just.
Tools see a lot of mileage.
So, you know, like the arm of forecasting, like people in a lot of different industries use that.
And, you know, these are just sort of tools of economists.
So that's maybe why I know them. But yes, it's awesome to share them with you.
So if you. So let me get that formally.
So you essentially are looking for new forecasting models or.
New data science techniques or.
I mean, I know most of them. I know most of the models and how to use them and stuff.
But like, I don't like, you know, like these.
You know, the way that you apply them to economics, like I don't really have a good grasp of like how to apply them to real world things.
I see.
Well, that's exactly what we're doing here in.
The state of science group is we're essentially we're taking the economic theory.
We're using some statistics and we're applying that to real world cannabis data because we're empiricists and we want to take.
What we're given. So remember.
Remember, this all comes from.
Well, you remember, so this all comes from the Colorado.
All these Colorado data dumps.
So we're.
You know, we're just taking this raw data like this is as raw as it gets. It's not even in Excel. It's just raw from PDFs.
Just completely raw.
We're getting it into an acceptable.
Format is an Excel spreadsheet that like is the very least we can do.
And then we're reading it in and then we're using, you know, modern statistics, economic theory, and we're.
We're applying it to try to make useful insights. You know, we're trying to have some some useful takeaways here.
And so what's what's exciting about this.
Is remember we were calculating the competitive wage over time in the historic rate of return over time.
Well, now you can even forecast that forward and then apply these data points to your economic theory.
So that way you can.
Make predictions about will wages rise or fall in 2021, which is.
Which is incredibly helpful insight to people working in the cannabis industry, because knowing what the actual wage maybe isn't that useful.
But knowing if the wage may rise or fall, you know, that could that could be vitally important to someone trying to staff their business.
Yeah, no, this is really this is all really cool. This is, you know, this is the kind of stuff they don't teach you in these in classes.
And just like dealing with these large pandas data frames. I mean, you never encountered that in a class. You never learn like these tricks.
Exactly. Then pandas is a big, big tools toolbox. So it's tough to get all the pieces right.
And they always give you these sort of toy problems, but these are actually like real problems using like real, you know, real, real theory, you know, coming out with like sort of real, you know, real estimates.
So this is good. This is kind of the stuff I, you know, this is this has been really helpful.
Well, that's awesome to hear, Charles.
And then here in this last minute or two, I'm just going to add a time index to the data, just so we can't see if we can't just plot everything.
Is it total revenue?
So now this is sort of sort of our takeaway from today, right? So we have the historic sales going way up. And, you know, you not you naively may think, oh, this is just going to go on forever.
But our model is smart enough that it's it's taken into account a little bit of this business cycle seasonality.
So you have, you know, so you are actually, you know, as you can see, it's smooth. So it's not, you know, a perfect forecast, but it can at least kind of give you, you know, an idea of the path that sales may take in the coming year.
So I think I'm going to go ahead and wrap it up for today.
But it was a little bit of a, you know, a quick and dirty exercise, so it could have could have been a bit cleaner. And so for next week, may clean it up a bit and extend upon it.
So that way we can do forecasts and actually apply economic theory to the forecast. So just want to thank you for coming. And I'll be including it here. So see you all next week.
Okay, see you. Thanks.
All right. See you Charles. Have an awesome day.