Okay, or not?
Sounds great.
I like it now.
Awesome.
Welcome to Game of Nodes, a weekly podcast on the Cosmos from independent Validator teams.
Okay, so now the phrase is punking the guests.
That's apparently what we're doing now.
Hello and welcome to Game of Nodes, a weekly podcast on the Cosmos from independent Validator teams.
And we have the lovely, lovely voiced Greg O'Seary to with us today,
despite his complaints to the contrary.
Thank you for humoring us, Greg.
We're going to be talking about, well, funnily enough, we're going to be talking about Akash today.
Is it, well, actually, first question, Greg, to kick us off, warm us up.
Is it Akash or Akash?
Akash.
Okay.
And what is there a deeper meaning behind the name?
Of course, Akash means the sky in Sanskrit or the ether, the first element in Sanskrit.
That is interesting.
Since clouds are formed in the sky, since Akash is a cloud, a cloud, or marketplace, or meeting point for clouds,
the name was derived from there, essentially.
As soon as you pronounced it, ash, I was like, he's going to say Sanskrit.
And many of my ancestors are going to roll over in their graves.
Thank you for that.
I now feel very embarrassed.
Okay.
So this week, obviously we've got Greg, we're going to talk quite a bit about Akash.
And but to kick us off at the top of the show, we're going to do some regular follow ups,
basically rolling off of stuff we had last week.
And the three topics I've got on here are Docsgate, Brontide, and Prop 2021.
I want to go with Brontide first, because that is the most personally exciting.
So Yussepe, you have some thoughts on this excellent band?
Well, I think I've been kind of following, based on some of your recommendations and kind of where Spotify's taking me and everything else.
Brontide has been showing up more and more on my pretty excellent prog slash metal playlist here.
And that's the number of great bands coming out of the UK that have a lifespan of less than three to five years,
but produce amazing albums.
And then those members go do something else and produce other amazing albums.
It's very 2000-y type of thing where you don't have a, you know, somebody's just playing for 40 years just to try to knock it out.
It seems like there's a lot less, a lot more jumping around from bed to bed, if you will.
Maybe that's just creatively to try to find the right thing, or maybe that is, you know, they just understand it's not going to happen or something else.
But it sure produces a great amount of music and very diverse music too.
Fun fact, I think the basis from that band went on to, I think is now the basis in Roto Tamsi, if you kind of more of a metalcore type of vibe.
But, you know, there is a lot of moving around on all these sorts of different projects.
When you're programming hot new features for a cash, Greg, are you, what kind of music do you listen to?
Boy, my favorite programming music is probably
Eagles of Death Metal.
I mean, they're not Death Metal band.
It's called Eagles of Death Metal.
Yeah.
Or BMRC or Black Rebel Motorcycle Club.
Something that's just like jing-jing-jing on the background.
It's quite raunchy, kind of, sort of almost down back to basics, kind of bluesy rock and roll, right?
Yeah, blues rock and roll is usually my go-to, but something that gets away in the background and doesn't take my focus away to the words, right?
Like, yep.
But I'm listening to music at least 10 hours a day.
I'm plugged in all the time when I'm recording.
Yeah, especially when I'm recording.
Like, yeah, what about yours?
I think we've got over our tastes quite a bit on the show previously, but this is obviously why we're talking about music as a regular thing,
but I think Söpp and I are kind of more of a progressive rock kind of vibe a lot of the time.
And I think the actualsy I think is into mainly achingly cool stuff, right?
Yeah, something like a more melodic type music.
Yeah, yeah. And then I feel like Noel's a stealth metalhead. He keeps really quiet.
Yeah, like metal and trance.
I like a bit of trance while I'm like bashing on the keyboard.
Stuff that I can't understand any words to, because I'm just not distracting.
I actually went and saw Dvorak's New World Symphony over the weekend, which is like, it's got some nutcracker in there.
It's got all sorts of nonsense with the organ symphony. So that was great timing.
That's cool.
It might be a bit like classical music.
I mean, in the classical music, I've got like a 101 symphony album that I just like put on in the car and when I'm working sometimes.
I reckon it's great. She'll see.
It is. Not the age myself, but I'm a big rock man enough and most are fan too.
I don't know. I think those are not that far from these other type of genres we're talking about.
I know it seems like there's a big divide between some of that stuff and trance and those types of things.
Not that much of a divide actually if you get into the right.
It's a percussion, isn't it? Like that. I think that's the big, like having kind of like a really specific backbeat is like very uncommon in classical music.
And if you cannot live without that.
For example, like, kind of quite common.
That thing of what people say like ambient music is like not that far away from like string, string sort of classical, maybe that is very sort of based on a large string section, moving at a slowish pace, you know, obviously not like Sturm and Drang or something where it's very dramatic and
with brass and whatnot.
But, you know, the snare drum only comes along as an instrument in the orchestra in the romantic period, right?
So please, musicologists, tell me.
I mean, I can only go as far as back to like violin of like the 1600s, everything after that. I'm like, I don't know.
Yeah, dude, I don't know the history of it. I just enjoy to see it a bit, all right.
Yeah, that's exactly right.
I'm just saying that like the best classical music is the succession intro where it is a dissonant string section with a tritone in it and the hip hop beat that that is like, what class if classical music had a hip hop beat, it would be est here in my
If classical wasn't classical, the best part of any show is probably the tune up for me. When you get to hear everything started coming and harmonize. That's the best.
No, no, harmonization, the best.
Okay, so I have a successfully completely derailed what should have been like a one minute how are we doing.
I sorry about that. So do we want to talk very, very briefly about what's going on with prop 20 prop 21. As we've just we've just done a prop 20 upgrade on on Juno.
And there is a slight little problem with it isn't there.
Not with the prop. Well, the upgrade went swimming well, right? It sounds like that in the the contract says we're testing everything else associated with prop and the software upgrade was was actually a cosmo visor running I actually forgot about it until it was over.
I just got notified just with the pause but then I assume that prop 21 you said there there is something that was found or at least something that needs to be updated.
Yeah, so there's a there's a basically a small problem in the IBC handler which we've actually been running until now without noticing.
But essentially, if you want to update an IBC channel via gov at the moment, you can't do you know we've never had to do it before which why we didn't notice that there was a bug like a line of code that is changing.
So I think that's in prop 21 the upgrade and also crucially the the funds need moving from a from a.
It's basically like I think it's more of a it's either a fat finger or miscommunication essentially that the funds were moved to a null address.
And then we're supposed to be moved to smart contract but instead they would just move to another dress, which was not what we did in testing.
So there was a little bit of a disconnect between what we ran on the test net and then what actually ran on main net in terms of like each one of the steps.
So now the second half needs to happen where it just gets moved over where after being on stake to moved it gets moved over to smart contract but I didn't really have any involvement in that part.
So it was a surprise to me as well because I didn't write or work on any of the code for the actual move side of things.
So I obviously saw it tested because you know we're helping out with the test nets and whatnot and helped out with smart contract but because it obviously worked in the test nets.
It was like okay cool.
Well, you guys who are working on that bit you've seemed to have under control whatever.
And then today like I was also away from keyboard when the upgrade happened and it worked whatever got back and it's like, oh, hey guys, what was what and I'm like, what's happening.
So yeah, it's gonna be an upgrade in a couple of days.
Not really a big deal.
I think that's kind of the end of that.
Actually, you know, as another, since we're doing follow ups and tie ups from last week that have come back rounds.
Well, something we were talking about last week on last week's episode Greg was that the the upgrade model of cosmos.
I think has gotten a lot of attention recently because of the variety of chains of upgrades and halts, etc. that have been happening.
And we were we've sort of had like quite a long conversation on whether or not it was some fundamental property of the way cosmos works or whether it's just that these chains are now.
Scaling and sophistication and user base and it's sort of correlation rather than causation just that there are a lot of chains moving very fast now.
Naturally, sometimes they're going to be issues they get stuck the teams do some work on stick them and it's not really anything to worry about.
So what's the question.
So the question is, do you think it's like correlation or causation or do you think it's like, do you think there's some fundamental thing about cosmos that means these problems sort of bubbling up now or do you think it's just noise because we're moving quickly.
I think it's lack of coordinate.
I mean, it's hard to say right like we had an issue with upgrade module recently we upgraded the chain and that was a disaster.
I feel like I mean, and it was also like, well, we didn't include the upgrade model in a testnet.
So we weren't able to test the upgrade model module, assuming that upgrades are going to just be smooth.
And assuming that's just going to work out of the box because that's the least sort of like critical piece of the the whole main net upgrade.
But that didn't end up to be true. And it was just a holy hell of like pushing out a patch waiting for people to, you know, come online the validators are biggest validators dropped and then people were out of sync so that we sink and it's a whole nasty sort of like, you know, the mediation process right so I think, I think it's just
I think the people are moving quite fast and there are different use cases I think are not being tested properly.
The test is the idea of testness have sort of like gone out of fashion.
Some of them do it, but we even the testness there no there's no guarantee that if the code passes through the test is going to actually like work.
Right.
So, with the kind of with the kind of speed people are moving at so I think they need to be a cosmos collective for testing these like, like modules.
I don't know if there is one or if ignite team is fully like plugged in to what's going on in the ecosystem and also like it's wrong for us to actually expect ignite team to to be the, you know, the
responsible parties right so yeah things are moving way faster than the core team of cosmos members can keep up with.
I think that's what's causing all these issues right like.
Great that was a SDK issue or a wasmish I forget, but the SDK issue module supposed to introduce cosmos wiser and supposedly the ideal scenario is when you when you vote on on a proposal agree on what block you're going to grade it'll automatically do that great.
Supposing that was a promise, but the reality is like well, you know, there was some no pointer exception some memory some some nasty thing.
Right.
And I don't know why we didn't.
Why we assume just going to work and that just gives you the importance of actually having to test the actual binary you're going to know the other commit hash you're going to like upgrade on and not assuming that this coming touch is going to or this this version is going to work right like low hand and most of
I think Juno also had the problem or even sorry and most have the same problem right like.
Juno and have more so hard holds recently Juno was a cyber attack F moss was with Sir, she'll see no you guys.
That was a upgrade snafu.
Basically during the upgrade there was a we didn't get consensus straight away and then there was a lot of node resetting and losing state files.
So was it was it validator error then.
It was just it was a communal like error.
I guess you would say like there was a lot of people telling people to do things that they shouldn't be doing and.
The impetus on that was I think there was there was a wasm there was a zero day was some sort of zero day wasm type Dylan Pratt keep you going but there was some there was some issue that was kind of forcing an emergency upgrade and then that emergency upgrade.
Turned into a mix of bad communication and then there were spreadsheets and trying to get the make sure we had the consensus on it.
And then there was a version switch at the last minute which is always a bad thing right like somebody set it up.
That was was it last minute or during trying to get consensus.
I think it was during what I'm trying to get it through.
That's right. Yeah, it was so yeah.
And so they did a bin switch and then you know something and then consent it just the banners were just confused as shit and then trying different things to make it work and people to do stuff they shouldn't do.
And then a bunch of nodes got double signed big ones as well.
So they just had to stop the network.
What happened to the double signers to the lose money.
No, because they they wound back to the block before the doubles. So they when they restarted the network, they did like a state migration and then wound back a block.
That's right. Yeah.
Oh, they are the rollback. Yeah.
Most useless rollback ever.
But yeah, it was like one block.
One block. That's right. Yeah.
Just that one block that they were trying to, you know, get going.
So it was the I think it was pretty much just it picked up where the original upgrade was supposed to pick up.
That's right.
Right. So yeah, that so the upgrade module leads to the consensus failures.
I've seen that quite a happen quite a bit and worst scenario is if you have a big validator that was, you know, that has a state that's out of sync, and they just keep like printing blocks.
And everyone else is like out of consensus right so you're going to have these like crazy scenarios that you have to re coordinate and then the big validator has to like re sync the state to
a to a to a to a trust group that's much smaller.
But also like we're learning quite a bit like we're learning the importance of like better distribution right like is one of the one of the reasons why my gosh upgrade took about 1618 hours, and we're up all the time and the big reason was we had one of our biggest validators didn't show up on time or at least they showed up on time
they actually did what they're supposed to do, but they weren't present for us to be able to debug and like coordinate and overclock lapsed the company, the bills, cash network, the core contributors we don't have enough voting power to influence a vote in the in the, you know, in the thing so
it's decentralized launches that extremely hard and especially highly risky when you have validators with with a lot of voting power.
Yeah, right.
And we've spent a lot of time on Juno, especially given the cadence of the many upgrades that have happened in the last two months, what was the cyber attack what the three security patches that we've had for cosmogism.
Plus the plus, you know, the fact that we've had routine stuff going on as well that we've been trying to get through.
What we've had to do is basically have there is a coordination channel where we say we have a whole, there's now a releases procedure in the GitHub repo say even if people don't really want to engage with, I suppose the Juno team, they know where they can get those updates and then we've also had to be like well for a security
perspective we can't reveal everything publicly all the time anymore. Right. So some of the validators who sort of were relying on us just tweeting all the information and then just like knocking it out.
We've had to say well, look, the problem is we can't do that because there's security risks so we kind of need to know if not exactly who you are you don't have to dox yourself we do need to have a place to contact you otherwise it's not going to work.
Or at least it's not, I mean, well, it will work but the first notification you're going to get of an upgrade is when everybody else stops upgrades and you have hash.
That was one of my main concerns with upgrade or cosmos prop 69 the cosmogism upgrade, because I was like well, for Juno we've had a couple of like shadow upgrades right and that's going well because many of the validators are pretty plugged in.
However, with cosmos those upgrades tend to take quite a bit longer because it seems like those validators aren't quite as engaged so if there is a security breach or a topic that needs to come out now, or we can be able to react as quickly.
That was one of my main concerns.
But we're ultimately voting yes anyway. Yeah, yeah, so you told me the addition of cosmogism to the hub. Yeah, yeah, yeah, yeah.
I think, I think in Juno we're a little bit more lucky in that we don't have as many corporate validators in Juno.
So it's like a little bit more accessible to the actual people who are running the nodes.
I think the other things as well I mean with Manita we did two tests we did so we did two tests that's pre launch and then pre monitor we had the mainline test net which got killed by we are cyber attacking it me and Ben X for and then we had to restart it didn't we because we killed it and so any validator that was a part of those tests
nets had like the experience of like chainhawk restart with Cosm wasm and all the like, oh now you need to shuffle the wasm folder and all that kind of stuff and then obviously then the final upgrade out.
We did like what was it for test nets in three weeks.
I think that you mean there's like a core of validators there who really know their stuff and like can help other validators when stuff goes wrong. I mean, and to be on site sometimes there's things that when I'm tired I screw up and I'll literally ask and one of those validators that was a part of that
a starty set will literally be like DM me and like right dude here's what you need to do don't worry about it and you're like ah this is why these guys are great this is all these guys are so good.
We have a pretty good participation in uni like in terms of number of validators from the mainnet set there's quite a few in our test net which is good.
Yeah running.
The whole upgrade is a bonding process for sure. It's a good way to put it.
It was amazing like most people forget how important validators are for supporting these networks and and really.
I mean a lot of cash validators is just amazing to see how supportive you know especially the small guys are right like the big guys don't really care as much as a small guys do so makes you really reevaluate your your delegations and we're going to do a whole revamp of our own foundation delegates
to people that actually added support and were critical when we needed them but boy it's just been such a such a bonding experience and also a learning experience and and a reminder to thank you validators should have a validator day.
Well that's a validator day.
Every day is validator day for us.
Well that was part of the reason to actually talk about game of notes was was that the independent validators I think sometimes get overlooked or you have you know everybody and I think there's a there's a pretty blinds pretty large blind spot in in terms of.
Delegators knowing how to pick validators what's important and then they're all ranked right so when you see one to 10 out of 125 people I'm safe right that's what that's what you see and there's there's really that idea around decentralization and some of these errors errors that we were talking about issues we're talking about.
Involvement and buy in and those types of things like like somebody in that top 10 I know like maybe a an exchange wallet or something similar that doesn't have they're not involved the way you know the phrase or cake notes is or lavender five is right.
But again that's difficult to you really have to take some time to really build the marketing and the branding around that for people to understand what's the difference there so I think that that whole idea on proof of stake.
That is a real misconception I think in terms of delegators in terms of what's important.
Do you think you surfer the the the difficulty there is I mean I say from like my my capacity and our capacity as a small validator company is like.
Maybe as well because we do that development and we have other projects but like we can basically support to change to like a meaningful like we're more engaged than the average right and more than that we have to be just another validator in the pack even though we're an independent validator and we rely on.
Like to the extent this is why we've only we've only run three chains and the third chain we run as a meme chain because it's a way for us to test stuff out in production to some extent right.
But it's quite I mean so you know Greg we've looked at.
Joining the cash set because it's a really interesting project right yes but I kind of feel like it's the amount of work I do other than just validating personally and that we try and do as a company.
For the chains that we work on like I already feel like I don't do nearly as much for Stargat is say then we did before they launched like when they were doing the test and stuff we tried to be.
Because those folks were in the US we tried to be helping out in the European time zone and doing some kind of stuff and you're chipping in what we could and what not.
And I feel like because the amount of work in June is quite big.
And we were quite involved over there that's meant that we can't be as involved over here and so then I look at other chains like like yours really interesting project.
Seems very involved validator set you know we're for some depth development stuff we're doing we're looking to move stuff long term it's like you know dowels running their own infrastructure on a cash and all that's really really interesting to us but we're also just like well if we can't make.
A meaningful contribution outside of just validating and we try and buy our way into the set and maybe knock somebody out who we don't know maybe is really helping out area right what are we adding should we even be in that validator set probably.
Not right.
I mean.
Folks that are adding value on a cash will will get delegations.
I'm very active in the discourse in the Twitter's and especially in the upgrades I was there for the whole 16 to 18 hours right.
So I know what validators added value.
And.
And about it is that don't add value right so if I see a validator getting knocked out of the act to set a validator that I feel like should not be knocked out.
I'm going to give my personal or and I'm going to propose our foundation to give delegation right.
So it's a very much approved statement very much not very much like can like can contextual conditional is no.
You know rulebook right like but if there is someone who just you know has a lot of money I mean we have some validators who you know just bought their way in they don't do anything.
Yeah.
And they're not going to get the color they might personal love and foundation delegation right so.
I mean there's no right way to do it and of course we want our validators to be hyper involved.
But you know buying tokens and holding on to those tokens is also contributing to security network right so it's not they're not like useless they are contributing to security network.
So.
But.
Delegations are one way to keep the validators in the active set without having to buy the tokens right and I think that should be a more natural.
We want to see that because the value of contribution is much higher than than than value of capital.
So.
And we're also increasing our validators set right we do are I think the proposals every three or four months to increase the validators set.
I mean you got to keep in mind because the validators at the slower and the larger the chain gets right.
But they're always straight out but you know we feel like if the if the minimum required to be in the active set is not reasonable.
There's somebody in our community and most of the proposals don't come from us.
Somebody in our community will put a proposal and we'll have you support it in whatever way we can.
And and then you have like.
Ethereum and like these other chains here thousands of thousands of validators.
I wonder how.
Upgrades if there is anything of such in Ethereum is going to happen in the proof of stake network right.
Very curious about networks that have.
Enormous amounts of nodes right.
I guess Ethereum you don't really have to worry too much because I think it's the leader finance folks that control like 50% of the nodes is some crazy number like that.
I just need to go to one and the whole notion of like Ethereum is more decentralized because has no more nodes is just.
Ridiculous right because you know one company controls a lot of these nodes.
Yeah but I guess but in a really true scenario where you have thousands of thousands of nodes and truly distributed.
I wonder how.
Upgrades happen especially when you have to do security patches the really rapid security patches right.
Yeah, so I wonder if there's like a model like I'm trying to remember the name of the protocol.
I think you have hollow chain.
Yeah, I am thinking of hollow chain.
I think where the the sort of the date of the validation integrity of the blocks if you like a kind of push to app level on they.
So I don't exactly what their upgrade model is but I presume is that most clients are obviously not keeping full state it's more of a bit torrent model where.
It's just more about current version of protocol for two things that want to actually sort of be clients in the network so.
There's probably a bunch of quite interesting different ways of cutting this problem.
But I wonder if like you know the more fundamental thing that's maybe relevant for Cosmos is you know should we be tying.
Upgrades to the concept of governance right because that there are situations in which a software upgrade might be conscientious and we've we've.
We've certainly seen that in June.
Oh right that's that's definitely happened over the last 10 weeks.
But for like the two that we've had prop 20 prop prop 20 is funny because it's controversial prop 21 is literally a like oh minor bug let's knock out a release in a kind of.
If you worked in industry and done continuous deployment or continuous delivery you would literally hit the button the second time ago oops.
And then it would be out five minutes later right.
It wouldn't be a thing and so there is like there is an interesting question I suppose given that we're seeing that governance in Cosmos is coming up against challenges whether or not.
I suppose whether or not there there will be changes in that direction like are there types of.
So something I think D me on the core team for Juno was talking about and I think Jack was talking about maybe on the podcast a while ago was like whether we should be having to speed.
Governance for software upgrades like you say emergency is like one day.
Regular is like five day and quorum is maybe different for those two.
But then you know people try and sneak through things on the one they shouldn't on the so it becomes back it comes back to this like.
How trustless are we and do we have to trust our validators and I think the answer is and I've been shitposting about this on Twitter for some time now is that.
In my opinion proof of stake is actually just a distributed system it's not.
For some definition of decentralized decentralized is a distributed system with multiple operators who don't necessarily trust each other.
But in practice they actually have to so in practice you've got a hundred operators rather than one operator and that's only like one order of magnitude different right.
So it's a good question.
The.
I don't I think the term trustless is fundamentally stupid.
You trust something at the end you know beat at the cell level beat at whatever level there is something that you're trusting if you're running a Bitcoin node you're still trusting the CPU is going to act.
So it's going to be good right you're trusting something.
I think decentralization is about removing attack vectors right.
When in considerable chain whole thing or double spending or you know verification right things of that nature but it's not removing trust entirely you can't like.
Like you got a trust.
Look at the access point let's talk about Kepler right.
You got a trust that Kepler you're installing on your computer is valid and verified by someone else.
You're trusting Google at that point.
But you can't use it a decentralized network.
Without some level of trust.
The only question is how much of trust do we require right so something like.
Chain upgrades right like I mean you're trusting validators to do the necessary diligence intelligence and you know review the code like I don't know if you guys do review the code but most of them don't.
Right review the code.
And you know.
Literally like act on behalf of the people right.
And you have to coordinate you have to learn who other validators I think there's no truly.
Even as a Bitcoin like this is upgrade for Bitcoin there you're still trusting the core team.
That they are acting on behalf of the people not everybody's being you know my grandma has Bitcoin she's not reading code.
She running the client.
Right like she's just trusting somebody to run the client for her.
But that trust is inherently not dangerous because you know her trust in the client doesn't mean is removing the trust from the network.
Yeah.
Like so trust minimize is a better word than trust less.
The minimal trust you you you need the lower friction that you get to interact between two parties in that particular network.
The lower friction you have the higher scalability you have for a system.
Right.
So Bitcoin or decentralized systems is removed or lower the friction of interaction between two parties for whatever reason.
We had kind of commerce in a car she now do deployments in Juneau actually run smart contracts.
Right.
You don't need a high level high degree of trust from the other party in order to function.
Right.
And that's what makes decentralized systems fascinating.
Yeah.
Whatever term on it right.
Ultimately everything is a distributed system.
Ultimately.
Right.
The only difference is how much how much trust you need to participate or run the network.
Yeah.
And when it comes to like yeah.
I feel that upgrades should be.
Frictionless the actual process of upgrading from one version to another migrating the data.
I think that process should be fixed frictionless.
But what goes in the upgrade should not be for us.
That should be on debate.
Absolutely needed.
We all after you come to a consensus what goes to the upgrade.
I think from there on it should be completely frictionless.
All that means is we should be able to ship upgrades when we need and have enough time for the community to come to consensus.
Right.
Yeah.
And the code that matters then there you know that you can you can look at it a different way which is maybe back to the like I said not an expert on Holochain but gather there are some similarities between their model and the bit torrent model.
You know maybe we should be looking at a thing where you dirty secret most validators don't run an archive node.
Most validators doing aggressively especially on large chains especially on smart contract chains.
If you're part of a chain upgrade you know what the difference between prune or full load is.
Yeah.
Right.
So you know there's a whole bunch of things here where you go OK well actually a lot of the complexities of managing that upgraded to do with the state and the validity of the state changes effectively and whether or not it's consensus breaking and what the next block off the upgrade is going to do.
But keeping almost no state on the client would sort of make that whole process a lot easier but then you kind of get back into the well OK so where's the system integrity come from.
And that's like OK so if we've got lots of peer to peer nodes that just going to interact with each other where's the archive of state.
Then we've designed a new system that's not quite what we're currently working with right. So it's I don't know to me this is all very as a kind of like systems designer.
I think this is all quite interesting.
Now which gives us the archive archive now or something that gives us a whole new states and very fine what not.
I mean I mean I did to be perfectly honest I mean I think you're you're semi joking about that. But I don't think it's necessarily a bad idea. I mean like there is an argument for doing something that is stateless with more of like a cell towers model where your archive nodes are effectively a cell towers going like.
Yep. No you've just asked me about data integrity and I can tell you the hash that you've just given me is the one that the majority of nodes or a sub graph of nodes have given me and that seems reasonable.
You know it's all about fast finality at the end of the day. It boils down to how fast you need your finality right.
Someone has ideas actually good friend of mine. Dan Lynch is it's an up and coming Cosmos Tower.
He is actually doing a lot of work for Osmosis now.
But he had an idea for what was called but the idea was like we have a big challenge is bootstrapping and discovering RPC nodes and discovering state bootstrapping in general.
It's a huge problem right.
What's having a chain or a client as a matter of fact.
And how do you know. Well first question you got to ask is how do you know what you're getting Israel or not.
How do you trust that RPC server.
But the only way to do that is like run by yourself from the code compound area area area.
And I assume that GitHub is the official source code and it's not you know it doesn't have any exploits or not trying to steal your money right.
So you're trusting GitHub again here and you're trusting that the canonical source code and you're trusting GitHub actually verified the authors are right.
Anyway the the I think we need to we need to do much better.
We need to do much better without GitHub involved necessarily right.
And we need to do much better with bootstrapping and in a DAO idea seem to be very very attractive especially at DAO that is where the members are effectively validators right because they are most connected with with with managing multi managing multi or validating multiple
chains. If they all come together and have some sort of like you know decentralized way to to sign off on these packages or bless the packages or bless the state and host the state and get some fees for it or something.
It's not a bad idea actually.
It could be a chain by itself or it could be a DAO on June.
I think somebody's somebody's trying to build a get chain in cosmos right. I think Paul could choose sent round a thing.
Paul could choose an excellent valid by the way I'm not sure if they validate a cache but one of the pop up in there.
Yeah, one of the absolute best. They were they were talking to us the other day about I forget the name of it but there's a chain launching in cosmos sort of with some get integration hooks and I think they want to do some of that stuff in the future whether or
not it will pan out exactly as intended.
I don't know.
So, some specific questions on a cash.
I keep saying a cash is because I'm pronouncing it the English word I can't.
I know showing them whatever anyway, culturally why unfortunately.
So, for those that may be less familiar with a cash.
What what drove you to start in what was like your original vision.
So, bit of a backstory my been a programmer for a little 25 years, and I founded a company called Angel hack.
Angel hack is the largest hackathon accelerator in the world hackathon based accelerator, but it was really we introduced hackathons to the world hackathons before Angel hack or a way underground concept.
And I helped launch several developer tooling companies I mean my primary focus.
Over the last I would say 10 years I've been building and shipping developer tooling.
And one of the biggest company help launch was fire base.
Fire base became absolute absolute darling in developer tooling and got acquired by Google.
And so my passion was always developing tools that are hyperscalable without the friction, because a lot of times you have distributed systems with hyperscalability, but that extremely hard to use.
And vice versa right systems that are super simple are not very scalable.
Right so the idea was how do you really create infrastructure that's easy to use at the same time.
You know, easy to scale right.
It's still a problem to date.
And we.
And I stumbled this 2013, two years into Angel hack.
I really wanted to solve the deployment problem, right like so deploying something from your workstation to the cloud.
Was extremely, you know, difficult, especially if you're trying to do if you're under duress at a hack.
Right.
It's easy to like do that on a hero cool.
Now you have an effort finders or so what not.
But those systems generally can scale right.
Heracle literally wouldn't scale like just ask you to go somewhere else.
And if you try to use the Amazon, and it's just a nightmare, right.
And you have to do like, you know, share for a puppet or one of these like nasty things, which are very unpleasant to use tools.
And so I discovered this thing called containers.
And I fell in love with them.
And I was I was like, and containers back then this is right before Docker.
Came along with containers really simple.
And, and I was like working on how do you make these containers more fault tolerant.
And then I discovered Kubernetes in 2014.
This is way before Kubernetes, it's right, you know, very, very early and spoken for a conference and whatnot.
I started contributing to the project quite a bit and found the company overclock labs to take Kubernetes to market.
And I love Kubernetes because it was on its Docker containers in general would solve the parody problem.
You build once you test it and you can take that container and deploy it on a server.
You don't have to worry about dependencies, right, which is most of the time the, you know, the most of the time where the issues happen and back then mutable infrastructure was a thing.
I want to immutable infrastructure.
Right.
So, and containers solve a lot of these problems.
Kubernetes all the orchestration and for tolerance problems.
And, and that's, that's where I saw the light to build a truly scalable, easy to use infrastructure and Kubernetes manual is not easy.
I guess it's still not easy to use, but it was horrible back then.
I mean, I know documentation is very, very early days.
And we built overclock labs today, Kubernetes market, but also and do so in a, in a, in a product that was high performance, edge capable scheduling.
Right.
So you could literally use overclock labs product to schedule to turn any data center, mostly bare metal data centers into, into cloud enabled products.
Right.
So, and federated them.
So you can literally create this beautiful layer and that you can deploy to let's say 10 different regions in like three minutes or four minutes using all running communities and communities didn't have multi cloud coordination capability.
So we wrote the, the, the control plane for the, for the multi cloud to make Kubernetes multi cloud in 2015.
And it worked amazingly well.
And so that got me to talk to the community's conference and the, and I met a lot of incredible people.
A lot of the code I wrote for that became helm and, you know, helm came around way later and they used a lot of the code I wrote to do this deployments right to do this, you know, package management style deployments.
And when we were deploying this, we discovered most compute sitting in this data centers we're deploying to was unused.
The number was somewhere like 90% of the compute was not used.
Right.
So, and well, then then it becomes very obvious to create a marketplace to unlock this compute and make it available.
This is my new high performance compute status centers with their metal storage.
Right.
And, and, and, and bare metal is a lot more efficient than the hypervisor in terms of like resource allocation.
And it's a lot more faster because you don't have the double kernel problem you're going directly to the box right all the good stuff and from a security standpoint all that all the good stuff.
And, and so when we decided to really build a marketplace we wanted to make sure it's open, because we're open source people.
If you've been working on open so we didn't want to create a close source sort of like company and big problem with open source is well, it's not sustainable right as a business.
And how do you really create a network or a, you know, or a product that can set self sustain itself.
And, and we were also like being a edge and evil product we had to create a, you know, sort of like central as system, right because I'm a big fan of optimistic schedulers.
And a big thing about optimistic schedules versus pessimistic schedulers is pessimistic schedulers need a control plane that can that, you know, controls the nodes with optimistic is self, you know, propagating right.
It's slower but actually more, more, more.
I'll put my hand up and say, I didn't, I didn't fully grasp what's going on why why why does the optimistic schedule and need a centralized router.
So a optimistic scheduler would try to optimistically schedule and then fail and then, you know, fail to succeed and doesn't require a lock.
A lock has to be stored. Why, because when you need a lock that means that lock has to be maintained by a system.
And that system could go down like in pessimistic schedulers.
So we wanted a lock less system.
Right.
That's where the optimistic scheduler design really shines.
And we kind of built a optimistic scheduler at overclock that was amazing for scheduling the edge nodes.
Right. So and these nodes were like self coordinate without a central control plane in a peer to peer matter.
And they would use me with it CD those most popular data structure, you know, that store.
And we kind of wrote like a bit torn style replication mechanism.
So we kind of had like a quasi blockchain and a mercury integrity check.
Right.
So yeah, I had like a quasi blockchain without even realizing we both blockchains.
This is in 20, 15, 2016 time frame.
And then we were like, well, I mean, we had a blockchain without our tokens to, you know, to any permission blockchain.
Right.
And then we realized, well, we might actually buy the bullet and go full blockchain.
Instead of.
I think leads us on to the next kind of question area that we kind of had for you, which was like, this sounds like the original vision was was very, very strictly kind of going in this direction.
You're already going with Docker and Kubernetes and whatnot.
And it's like, okay, well, but how can we make this sustainable?
So it's a combination of the economic model and the fact that you're already kind of already building a blockchain.
So then the next question we were going to kind of ask about, you know, your involvement in the cosmos, which is what made you build on cosmos in the first place?
And I what what technical benefits did did that bring and what challenges.
So, so the decision was like, well, we want to do this, do we do a one or do we do an Ethereum Ethereum was the only chain.
We actually prototyped in the Ethereum on Ethereum, Crypto Kitties launch and crash out chain.
And we're like, well, we're not, this is not going to work.
Right.
Ethereum like Crypto Kitties literally took down the chain for a few remember that for a while.
And we were not even able to do anything and we decided to go L one. And the question was to build our own chain from scratch.
Or if we can find some help to do that quicker and then somebody introduced me to Jay Kwan.
And this is 2018.
Early this 2018.
And they were like, Hey, there's this library called Tenderman people are building which does the consensus for you.
And everything else you build on top of it.
And we liked it because number one was Golang.
It's easy for us to go pick it up.
And we were looking at the development on Tenderman.
It was how to high cadence on GitHub is very important to us.
And number three, and the fact that it's Golang gave us confidence that we can actually go build on something that's incomplete and actually, you know, contribute and build together if you have to.
And the consensus.
For those at the back, Tenderman is Golang.
The Cosmos SDK is Golang Docker and a lot of the tooling around Docker containerization Kubernetes.
These large scale, maybe inverted commas enterprise software suites for doing orchestration on a massive scale at like data center level.
A lot of these tools are written in Golang.
So obviously that Gregory talked about his background in that area.
So that's why you when you see the goal, the kind of go light bulb go on there.
That's why I've been doing go for since 2014, officially full time go.
If you look at my GitHub, you'll see a lot of history.
I've contributed.
My libraries are used by Kubernetes.
I was early contributed Terraform Docker just that era of 2013 2014 infrastructure software.
That's that's my background, right.
And go is very comfortable for us.
We try to pick a language or a framework usually in the language that we're comfortable with so we can actually make contributions.
And Tenderman wasn't stable for a long time.
And it didn't have a module.
So we ended up writing all the modules ourselves.
And then Cosmos SDK came in 2019, I think.
Cosmos was developing parallelly to our to our, you know, project.
And then it we drew apart, right, because we had our own one we were developing.
We didn't really, you know, Cosmos was not ready.
So we ended up just and the lot of the modules we wrote, of course, in a small team, we're not going to be able to have the same level of contributions.
The core ignite team did.
And in 2019, we decided to bite the bullet and switch to Cosmos SDK and throw away a lot of our code.
Adam, a co-founder did this passive PR somewhere.
Somewhere in the GitHub's, but we deleted like 80,000 lines of code.
Because that was the kind of like it was amazing because you know, it's like deleting code.
And like the leading code means less attack surface.
You're really, you're really exchanging code more than deleting, right?
Something that's more reviewed, more PR review, that's more ideally more stable, ideally more maintained, right?
Like so quote unquote, well maintained, right?
But there's always we write the least amount of code as possible.
That's my policy, right?
Although we end up, we end up writing a lot of code.
We have the saying that hey, months of coding can save hours of planning.
Yeah, yeah.
But anyway, you want to write less code and SDK significantly reduced our amount of code that we need to write.
And now it's a thriving ecosystems amazing to see the modules that are developing and we wrote a few more modules we contributed to and whatnot.
So this is a long winded history as to how we started and why we went down the rabbit hole.
And believe me, when we were doing Cosmos, nobody, there are not too many teams that are doing Cosmos, very few actually.
Yeah, yeah, it was one of the first projects, right?
I would assume in terms of larger projects that came off the hub.
Yeah, we're the first IBC chain, right?
Like after the hub, the first IBC chain.
So when Osmos is launched, they launched with Akash as the first token besides Osmo and Ion.
Like pool number three and pool number four is Akash, right?
So and Sunny is also a good friend.
But but yeah, being early in the ecosystem, you could get a lot of help from the Cosmos or G's.
Sunny was very, very helpful and we wrote, you know, I've read almost every paper of Cosmos and, you know, like, you know, a lot of the economics were influenced by Sunny's work and Jack was very helpful.
Jack worked for Akash for like six months, I believe, helped us really, you know, get in there.
And it's fun to be early because you get access to all these amazing people, you know, working like, you know, on the team for even like, what's the name of the most founder for their work for like a few weeks at Akash as well.
So we had a lot of like Cosmos or G's working at Akash at some point.
So they have like the biggest knowledge base, right?
For the thing because they wrote it.
So yeah, you probably have the best knowledge base.
So that's the that's the history of Akash.
Where do you see the project going, you know, in like five years?
Like what's your vision now?
Before we get into that, the fray, I sort of have an interim question between those two.
And that is, you recently upgraded.
And I didn't go into the test net and I'm not a validator on the Akash main net.
I wasn't inside that, but I like resigned from that somewhat lately because I just don't have not been a main net validator there.
I don't really have the additional time to put to that.
But my question is, so it was a major upgrade this one, I believe.
And I'm just curious as to what feature upgrades there was, like what additional functionality did this upgrade bring?
Or was it just more of a background upgrade?
No, this added storage.
Well, so for context, cloud being able to compete or having complement or having feature match with Amazon is a long game.
You're not when Akash launch, you know, we didn't plan on having all the features Amazon has on day one, right?
It's a long roadmap for us to get there to even match all the features.
So initially, when we launched the Akash platform, what a year ago, we wanted to perfect the container orchestration, fmr container orchestration.
Because once you add state to it, you have a second variable that is extremely hard to debug to stabilize and whatnot.
So we want to keep the features that small enough so that our teams can, you know, have focus and we have less things to worry about and less things to validate.
So we put up our plan saying that we're going to do fmr containers without state, then we're going to do state, then we're going to add compute capability, this GPU is in bandwidth and whatnot.
Then that will give us a foundation to actually build a real cloud on top of it, the true serverless cloud, the way we're going, right?
What a true serverless cloud will look like essentially is, you know, writing serverless code, be it no code, and being able to literally orchestrate across different services that are, you know, offered to you in an open market and in a competitive manner.
And, you know, an ecosystem where there are a lot of players and benefiting from each other, right?
The idea is to remove a need for a centralized gatekeeper to all these technologies.
Today, if you want access to any good technology, I mean, you got to be on the cloud, right?
So where we are going is first solve the fundamentals and then we get into the actual services, right? I want to be able to get Postgres, manage Postgres on a cloud, right?
And I would want that Postgres to come from the creator of Post, creators of Postgres, not from Amazon.
Because Amazon is literally, is widely labeling it and adding a premium right?
The creators don't make any money, right? The operator, the people that contribute to the network don't make any money.
So our goal of Vision of Akash is to change that.
So this Mainnet 3 is essentially the first step in us upgrading capabilities and adding elastic storage, essentially, right?
Right.
So now we have unbounded storage. You can attach how much ever that provider is capable of providing you in terms of storage.
There is no limit. And this is network attached storage. Now we have NVME storage.
You have all kinds of high performance storage that you can get access to.
And we are seeing a lot of activity from Chia, Project Chia is one of these really good projects that is storage heavy, essentially.
The consensus algorithm is storage based. So for them, they can get specialized storage on the current cloud today.
They're able to come to Akash and get that high performance storage.
They want something like a million IOSs on them, right?
So Akash is opening up and in the future, if you look at what's happening with MyTea, which is an app that started by Sahel, who's a known founder, a lot of new technology.
And for MyTea, they need their own data centers because they can get the performance on the current cloud.
So the way Akash is going, Akash want to be in a position to serve these new founders with specialized requirements.
So they should be able to put an RFP and be like, hey, anyone that can provide this kind of capacity, talk to me, I'll get you, come to Akash and I'll be able to consume you in a peer-to-peer manner.
So really the goal of Akash is to open up the market to anybody and anyone that's capable of serving or providing compute to anyone that's capable of consuming compute.
And serving compute is not a complicated thing. Running a data center is not as complicated as writing software.
I mean, it is being standardized.
That's a pretty hot take.
I mean, it's got a lot of work.
I mean, writing software is a lot more heavier than maintaining, you know, changing your disks, literally.
Like the 8.4 million data centers in the world.
Right.
And again, the modern software is designed to be fault tolerant.
It's designed to be interpret tolerant.
It should be if it's not.
Yeah, really interesting to hear you talk about, you know, sort of serverless technologies and make a kind of on-demand compute element.
Because obviously, so pre coming to work in crypto, a lot of what I did as an independent consultant and we did as a company was trying to get enterprise-sized companies to ditch running metal.
And if they weren't in the cloud, moving to cloud, and if they were already in the cloud, move to serverless.
And if, you know, sometimes jumping the whole way from bare metal to serverless, right?
Because fundamentally, for most kind of consumer grade applications where it's, you know, e-commerce or some kind of thing that can be served headlessly, some kind of content website, some other bits and bobs, maybe a next JS site,
maybe, you know, some React or Gatsby or, you know, whichever kind of, whichever front-end it is.
It's not particularly heavyweight and the API requirements on it aren't particularly heavyweight.
And you don't need lots of big heavy servers sitting around, you know, to serve a few web requests with a TypeScript API, do you know what I mean?
Yeah.
And so, you know, serverless is a great paradigm for that because even on AWS, you can, you know, have 80 serverless functions in a test environment, a prod environment and have like quite significant traffic and pay like $200 a month.
You know, I've been in that scenario where we had a large team sort of running this project and people were like, oh, what is this going to cost us?
What is this going to cost us?
And I was like, I guarantee you our monthly cost for the serverless function invocation is going to be less than one day of one developer's time.
And I was right by a good margin as well.
But you look at that and you go, well, hang on.
Like you said earlier, there's a lot of unused compute sitting around.
What if you can take that model of on demand, pay paper, micro second or whatever it is that AWS and Google do, you know, for their on demand compute for serverless functions and then also be spending at the rate of unserved effectively compute.
It could get an order of magnitude cheaper, right?
And for somebody that's kind of like a serverless advocate, where I'm like, I only really want to pay for what I'm using. And I think that that's what people should do.
And it's an obvious place to have a market.
That's, I think, a really exciting idea.
Like if you can get it to that level of granularity, right?
That's where we're going.
That's where that's the direction we're heading towards.
And I think like once there's so much opportunity and so many problems right now to solve.
I mean, just before we can get to this true no code serverless, just write a function and run it, plug in in a low latency environment, you know, you know, run all that stuff.
We need to solve bandwidth.
We need to solve storage.
We need to solve like, like high percent high performance compute HPC computing like with GPs and whatnot.
Unless we can solve these problems in a high distributed and decentralized way.
Then we're going to see the true benefits of serverless come to fruition, right?
Like today in pricing, cloud has this course screen pricing model, which results in partial usage risk.
So like you pay essentially one hour, one hour is a minimum.
And get a standard tier model run the serverless pricing on the program with traditional instance pricing on demand pricing.
You get this tier you pick from.
And you and you have this time slots you pay with.
And that's the course screen pricing model.
And we all sort of suboptimal for running anything like a function.
Right.
And the pricing is not and most of the time and almost all all the time, the requirements for a function vary.
It's not going to be like 16 gigs of memory.
It's going to be probably like, I don't know, 12.1 gigs of memory.
Yeah.
It's so fine green.
And it's very important for the future of the cloud to have a fine green model and the cost rewards you find a model.
Provide it because very challenging.
How do you price this on the on the backing and a lot of the work we're doing right now is like, how do you optimize, you know, your profit and loss.
But so we have taken fundamentally solid primitives with pricing, solid primitives with distributed design that will bake into that will come of relevance.
Once we have a serverless working because serverless will translate directly one to one down to instant pricing with our fine green model.
And then you're going to see a lot cheaper serverless function.
So this is not affordable today on the cloud.
Right.
Yeah, it's okay.
If you're running something with a lower scale, but try to scale that up and it's scary.
Some of these bills are especially in Amazon.
Right.
The Fright knows about that.
It's scary.
Right.
More do you scale.
I mean, I'm talking about like, I mean, I think, I think my, my, my valid.
My, my validator bills are validator bills as a company are because we are running on AWS.
It's the environment that we're most familiar with are substantially more expensive than the large scale serverless deployment with, you know, Aurora RDS in three AZs fully backed up.
You know, we are talking deployments where everything's terraformed, everything's automated, do maybe a minute, two minutes tops of DBIops per year, that kind of level of automation.
And, you know, that's that entire setup, I think, even on a per month per year basis was probably RDS was the only thing that was particularly expensive in that model.
And serverless execution, like I say, we were paying under $400 a month, but we very heavily optimized and very, very heavily knew what the billing pattern was.
So free.
So like, it's an open secret of a CNAWS that you get more, I can't remember which slider you get, whether it's for the CPU or for the RAM, but if you maxed it out, it makes it gives you, yeah, it gives you more virtual cores.
So it makes it go quicker depending on what language you're using.
Yeah, so sometimes it's better to bash the slider all the way and it actually executes quicker and cost you less.
So we kind of obviously optimized, we tried a few different settings to work out which one was the best one.
So, I mean, you're right there, there's obviously an angle for optimization, right?
But yeah, it's exciting to think of like how much more accessible that could get and especially, you know, again, how much better the tool that you get around it.
So we're at about an hour and 10, which is when we usually start thinking about putting the fray to bed, given our time zone here.
So we've had a couple of questions for you, Greg, which I like to pick up on before we start wrapping up.
I have a lot more questions.
We, well, yes, I was going to do the viewer. Do you want to do the viewer ones now?
It's only 10.
Well, from, so, well, do we put those in the spreadsheet?
But in any case, so, Todd is, and this is a brief answer to this one, I know, Todd has asked if Akash has a container registry.
There is one in the works called Open Registry, it's funded by the Akash Foundation, the other CAB actually community awards board.
It's ready to go or very close to completion.
And the idea is to have the Open Registry, and it's Akash Optimist running Akash.
So to use that as official container registry for Akash, and right now we had it still depend on the Docker Hub or whatever container registry as external to Akash.
But yes, there is one in the works and it's almost complete.
Okay, that's pretty awesome because I think at the moment, like most people just use GitHub container registry and Docker Hub, right?
So one thing that I know that people have been asking for for a long time is HTTPS capabilities.
I think most people at the moment just use Cloudflare to proxy to the servers.
So is that something that is sort of on the back burner because of the work around with Cloudflare or is that something that's actively been worked on?
It's on the roadmap. It's actually being worked on right now.
Is it something that's a, is there much higher priorities than that or is that like one of the higher priorities?
It's a think schedule for either next quarter, if I have to.
Okay.
It's a priority.
But definitely on the lower set of priority because people use Cloudflare and Cloudflare gives you a lot more functionality.
functionality beyond just HTTPS, right? It gives you DDoS protection, which is a pretty helpful feature.
But again, but it introduces centralization, right? So we are aware of what's going on.
But yeah, it's in the works. Hopefully they'll be ready in another three months, but we do have GPUs as a top tier priority, right?
And when you talk about like running validators, Minakash costs 90% cheaper now than Amazon roughly for the high end boxes, right?
80 to 90% cheaper.
That's crazy.
And it's also something called Omnibus, Cosmos Omnibus that gives you a lot of fault tolerance scripts and whatnot.
Mind your culture is very, very self-hosted model and doesn't have all the bills and vessels that you get with Amazon, especially fault tolerance and like auto scaling or whatnot.
So you have to end up writing a lot of these things yourself.
So like if you want auto scale, you monitor on chain, if you're monitoring performance and because offline, maybe you have a smart contract in Juneau that will do it.
So there's still a lot of work that needs to be done to host, but the advantage you get is like a very fine-grained pricing model that's 80% cheaper.
And soon enough, we're going to talk about Juneau.
We're going to, in the next upgrade, which is much sooner, within the next month at least, we're going to enable interchange accounts.
So with that, you'll be able to use your Juneau account to deploy directly on Akash.
You don't need an Akash wallet.
And that Juneau account will own the deployment.
It's great when you have a DAO, right, because DAO can own the web app or whatever you're using.
Say if you're on a matrix server with element front end to have your own Slack that's completely sovereign to you and fully decentralized that you own completely.
And you don't want to deal with a credit card or put it on Amazon.
You can use your Juneau account, fund the deployment using your Juneau account directly on Akash.
We're actually waiting for that functionality.
So there's a project that I'm working on with some other very smart people called Howl, which because of my pronunciation, H-O-W-L, Howl, not Howl9000 is what a lot of people think.
But it's sort of governed by a main DAO. And that DAO is the goal.
We're kind of hoping that that feature will be ready for us to kick the tires of it to get the HowlDAO.
HowlDAO with the Howl to actually host itself on Akash, if you like, and pay for its own hosting out of its development fund.
Because it's only the main front end is only sort of React.
There's not a hugely expensive service to run. So if we can work out a way of putting on Akash, it's going to cost even less.
And it can sort of pay for itself and support itself.
And that is kind of really interesting to us.
So there's an idea of making the actual separating out the running of the service from the development of it.
Maybe all the way back to what we were talking about with the Validator Set and Core Teams and how much trust you have.
As soon as you give the community the ability or the DAO the ability to run its own code, it can kind of turn around and say,
well, we don't like the direction developers have taken.
Might take it a different direction actually, lads.
And maybe we're rugging ourselves by looking at doing that.
But I think it's like an interesting development of how you can do things, right?
Yeah, it's great, right? I mean, like, it's, yeah, I mean, I think that should be the way to go.
I mean, you should, your member should control your code.
And what goes on to the, in that should be voted upon using governance.
That's, yeah, I think that's really exciting.
And I hope that the DAO DAO tooling advances to a point where you can make deployments through voting.
That'll be pretty awesome.
Yeah, I think that's being discussed right now.
There's a PR somewhere with, with,
Yeah, yeah.
I think interchange accounts is part of the IBC 3.1, is that?
Don't remember correctly, but.
Yeah, so it has to go, I think from Juno's point of view, the interchange accounts is dependent on an upstream.
It going into, fix going into Wasm.
Sorry, not Wasm.
So X, X Wasm, the Cosmos module.
But I think that's going to happen pretty soon.
Like I gather Ethan and Simon were talking about it.
So it is soon to be M.
So Greg, you did mention Cosmos Omnibus, which is mostly maintained, I think by Tom.
Tom, you on?
Yeah.
So thank you everyone, make sure you delegate to him on, certainly on a cash and the other chains on his, lately been contributing a lot and starting up.
Validators, but he's been working on Cosmos Omnibus for a very long time.
And I don't know how he keeps so much of that up to date.
He's all over most chains and there's a couple of other contributors in there as well.
I'm also becoming a Dow.
I think Omnibus has a sort of like the best scenario because they maintain the snapshots, they maintain the code, they maintain the scalability and fault tolerance scripts.
They maintain all that.
I think that should be like, that could be a good source of truth.
So I think one of the problems at the LB, awesome.
Are you saying they're already maintaining snapshots?
They are.
Okay.
That's awesome.
I think that's awesome.
I know that the storage was always a problem there.
So now that there's persistent storage now, right, then that might be bowed a lot better now for management of those nodes.
I'm not sure if there's an integration there yet.
But certainly.
There's also a discussion to run.
So I think we're using either storage or for your archives or storage or or SIA.
There's even a discussion to run a local gateway servers in local Akash providers.
We're running the gateway servers for these storage network.
So your downloads fees are going to be really, really low.
I mean, high, high, and show it.
So historically, the keys were kept on S3.
But these days, I think leaning towards you can still use that same.
Well, the same contain like the same code base, I think with Cosmos omnibus with very little variation to that to actually use like a like a soft sign from an external server like TM KMS, which sort of eliminates your
key storage issues for that, which I think is worth a look.
But one one other question I had outside of that was it's quite difficult to manage your deployment in terms of updating it.
So one thing that we do at the moment is on Juno, we've deployed the Juno network dot IO website on Akash.
And every time so it's quite fluid that website in updates for the ecosystem page where various projects add their information and then we update the container and then update a cache.
So there's no way that I know of at the moment on Akash and I'm a cache guru to have a system where you can automatically update from a GitHub commit to update your instance.
So basically now at the moment the way I've got it set up is if you make a commit or when you tag the same as on Cosmos omnibus, it will create a new container.
And then you have to manually go and like redeploy the instance.
Has anyone worked on like some sort of update from commit situation for React apps?
We do that internally for Akash website.
We actually have a get action that will handle the whole workflow for you.
We don't even touch the deployments, but we're going to open source that way.
There's some amount of cleanup that's happening and will open source will get enable deployments onto Akash.
The two sets of tools that you will be looking at.
One is a build time.
Another one is deploy time.
So the build time is building the container and deploy time is taking the container and deploying it.
So for build time, if you have a Docker file, great.
Use a Docker file for your build.
If not, we can use something called build packs.io.
These are literally Herico build packs that are community managed and that are amazing.
It gives you Herico like capability, like get pushed like animals or the dependencies and all that good stuff right for you.
So we have a fresh project that you haven't built a Docker container yet and building container is a nightmare.
Build packs is a great way to do it.
I have lots of examples on how to use build packs and do the deploy time using a GitHub action to take your container.
Or you can use a GitHub action to invoke the build phase as well.
But effectively, creating a sdl file around the container with that particular version, updating it completely on the background using GitHub actions.
We have that in the open source site very soon to deal with a lot of these issues because I agree 100% with you.
That's exciting to me.
I'm not sure if any of these other guys are actually working with the cache deployments, but that really excites me.
That's one of my biggest issues with using a cache is if you have a lot of deployments keeping them up to date with,
especially when you work with a decentralized team like we do on Juno,
if they go and you either have to get set up an alert that someone's tagged something new or go in and look at the commits and see what they've done
and then tag something new when you think it's far enough along and then go and do the update.
I use the cache analytics which makes it a lot easier than using ZLi.
Yeah, cache analytics is awesome.
We're also working collaboration features.
A lot of new features that are coming in, authentication, using a cache login essentially and effective login mechanism.
That'll open up collaboration.
You can effectively build apps on top of a cache using a cache as an identity provider.
You can add collaboration and you can have an external state on top of a cache, something similar to Terraform,
where you can have metadata for applications.
Metadata is important because it's very hard to identify applications right now.
You just get a big long string.
Challenging decentralized systems is just an experience.
It's just going to be so decentralized native that you have to rethink on how these applications work.
I'll be making a video announcement soon about our new head of product.
He was leading Terraform at HashiCore.
We're bringing a lot of former Cloud people into the mix, people that actually built Terraform.
One of the beautiful things about Terraform I liked was it has great automation, but I hated state synchronization.
I believe they have like cloud state synchronization now, but back in the day we used to manually ship the state if you wanted to collaborate.
I think the cloud product has done quite a bit and we're going to take a lot of lessons from the Terraform cloud to add collaboration
in a fully decentralized way, and great for teams that are collaborating across the pond and whatnot.
There's a lot of deployment side of things from a user experience side of things that we're focusing heavily on.
While we build the capabilities on the side, our latest hires are all focused on making the product simple.
I hate to use the command line. There are so many steps right now.
There are literally 14 steps to deploy something on a cache.
If you do it manually, it's a lot of work to like, and then maintain it as well, and you have to keep a lot of information to be able to maintain your deployments.
The metadata is hard. I literally have like a make file that I had on my command center, and the make file also has this gigantic state file.
My make files look like it's embarrassing to open source my make files because they're just crying right now.
It's just a flat file system to do metadata and whatnot, but we learned a lot.
It was important for us to not work on workflows and really focus on functionality and stabilizing functionality because if we dictate a workflow, then we limit ourselves in terms of creativity.
We're not going to let people come up with, right? Like, Akashlytics wouldn't exist if we made deployments super simple.
And in terms of Akashlytics, it has done a phenomenal job, much better job than what we could, with almost no resources.
And that's the kind of commitment we want people to have towards building Akash, and now they're actually doing really well.
And we learned a lot from them, right? That's how to deploy. And of course, we're hiring people to make things simpler, but simplicity, removing friction is a large, large part of our work right now.
Even with a new command line that we will be coming up with, a deploy command line that's outside of the actual command line that makes, that has opinions, right?
Because just to provide things, you've got to opinion. Yeah. Well, that's all pretty exciting.
And so it sounds like you're still, you know, there's a long roadmap and we're moving through it.
I think it's going to, as things get easier and more inclusive in features, I think the like take up of users will come along with it.
Once, I think, yeah, for a lot of people, I think once it becomes just easy to use, it'll become prolific.
If people can just like, you know, here's the thing, click, here's the GitHub repo, click the button, it's done, it's up.
And, you know, we want five instance instances with a load balance area, just go, doot, doot, doot, all over.
And like, and a good command line experience, I think good command line experience is so integral to high quality developers.
Web UI is great, but I think web UI, for me, especially, it's a secondary in extension, right?
I use web UI to get a bigger picture, but I prefer command line always.
Web UI requires like a certain amount of centralization as well in the workflow.
Oh, I bought some GUI, like a cashlet is a GUI, right?
Yeah, but it's, you download it to your thing, it's not dependent on like some web server somewhere.
It's like a local, but is this not a web deployment of that, is it?
It's a web-based, it's an electronic app, basically.
Okay, right.
But the point is a GUI, right? Forget that.
A GUI is definitely a complementary, right?
You'll be able to do, like, I think you should be able to do things very quickly using command line
and you got to use a GUI if you need to see a bigger or compare things, right?
Like logs, for example.
I'm a big grep person, right?
I want my greps, I want my filtering capabilities, like right on the command line, right?
But if I do want to share the logs, I should be able to do that using a GUI, right?
Or if I want to collaborate with other people, I want to be able to use GUI.
Maybe I just want to deploy once and just click a button I want to use a GUI, but, you know, things like that, right?
But if I want to do anything advanced, I'd rather prefer a command line.
A command line, a good command line because it's part of the workflow, right?
Like you're developing, I use Vim, my workflow.
I use Vim to write code and I should be able to exit that Vim or stay in that context and deploy immediately from that Vim, right?
Go to a website and open that because that's a whole different experience and a whole different context you're switching.
Well, it sounds like your hire from Terraform is in the same similar mindset, right?
Terraform absolutely follows that same type of structure where?
Command line first.
Command line first and then GUI, there is some GUI to be able to support that and there's some third party tools that help support that,
but that command line straightforward, reusable, right?
I love Hatchiko tool in general.
The way Mishashi Moto had a lot of influence in my way of designing command lines, right?
Like, a Hatch command line is a crime because we literally took the Cosmology StK modules because it gives you all these like Cobra modules that you just plug in.
Right.
And you get all the functionality.
So we didn't want to like mess with that.
And so we couldn't do anything beyond that we want to see in a command line.
So, but a command line we're building right now from scratch that doesn't inherit the SDK functions, but essentially is designed to do deployments only and not anything else.
The other answer functionality that you get with your chain management or not, right?
So deployment first, a very Hatchiko inspired, you know, look and feel or experience where we're hiding people from Hatchiko and help us to do this stuff.
Yep. But also, if you know any good command line developers, they're very, very hard to find by the way, like they're like almost UX people for the command line, you know, if you know anybody out there that's going to command line, please come talk to me.
I love command.
Look at my GitHub.
Personally, any repository that I have over a thousand stars is all command line works are written.
And also Mitch Mitch Hatchimori uses my command line libraries, which is, which I take with, you know, which is my biggest flex.
So because I consider him to be the master of command lines, right?
So I love command lines.
And I will support your command line, you know, aspirations, like we wanted to kick ass command line come talk to me.
Right. So, yeah.
All right. Well, thanks for indulging me in my questions.
I guess now we should let the fray wrap up is probably getting very, very late over there now.
It is now very late here.
So yeah, we always do we always do around Robin on the final question, Greg, just to just to wrap up, which is what are you most excited about this week in the cosmos coming up?
Well, I've been so ingrained in like a cash test net.
This week, actually, we have quite a few new providers coming out in a car or providing persistent storage.
And we are having torching.
Validated around our car should be the first real big validator.
And it only has 21 validators. And they've been very kind enough to to delegate foundation delegations to to running on a cash because, as you know, a lot of the torching nodes around on AWS and GCP, which they're not happy about.
And they're getting a lot of flack in the community.
So I'll be working on my first torching first validator.
I'll be running in production myself.
I mean, a lot of people run validators on a cash and all the proceeds that we get from validating torching, we go back to the community.
So we'll probably put that on a community pool and the community decide if they want the tokens to be distributed to themselves or burned or whatever.
So it'll be fully community run, you know, running on a cash, a very publicly run validating and see how that happens.
And I'll be joining the validator set, of course.
Nice.
Awesome.
Right.
Setting.
Yeah, your prop 21, of course, that's, that's, is the most exciting.
Is it more upgrades?
It's getting up at 3am.
It's most something. I'm not sure.
I feel like I'm so disconnected these days from community at large and what's exciting was not, especially the main net and the upgrades and all these crazy things happening in a cash.
I think being, being like, you know, such an integral part of a big, big project sort of limits your scope to what you can pay attention to.
Sure. Yeah.
Yeah.
Okay.
Sorry, the fray.
What do you think of me?
I'm most excited about getting my taxes done.
It's five, five days or something beyond the end of the tax year.
And everybody involved is going, Hey, man, where are the spreadsheets and saying, Hey, there's been 10 weeks of drama.
And I am, I am also the book company bookkeeper.
And we're three months behind nobody buys that they say, well, get it done.
So I'm looking forward to that being done so I can get back to writing some code.
That would be pretty cool.
Mine then two weeks ago, April 15th, you played into that line here, right?
Yeah, April 30th here.
So, so yeah, too far they don't have to be filed for ages.
It's just that your accountants, as soon as they, as soon as they crack it out, then their job's done, then they can invoice you and you're done.
I know, right?
It's invoice me.
Right.
Especially in our case, because the crypto is so complicated, they, they're basically like, you provide us the spreadsheets, then we will work it out.
And I'm like, I'm pretty sure the harder bit is the bit that we're doing, not the bit that you're doing, but you know,
Nevertheless, I'm looking forward to getting out of the way.
There's quite a few exciting test nets coming up actually, which the defund one, which is like a ETF,
ETF and the Cosmos space, I think is really interesting.
A dropping to a cash holders, I hear.
And there you go.
Yeah, nice.
So I'm quite interested in getting involved from that test net, maybe if I have time and there's,
Oh, there's something, there's one other that I think a couple of the other folks on this call.
There's one other interesting project that I saw this week and was like, well, that's kind of, oh, also Quicksilver is,
I think just test that start, the Quicksilver.
That's quite exciting.
I'm back to your point about Tom from EcoStake being a great person.
The Cosmos Interchain UK meetup, he's talking at the one in June, as is Joe from Quicksilver.
So nice very good as well as cool guy, cool projects, which one am I talking about could be both.
Talk about meetups.
I go and hack with JunoFox quite a bit here in San Francisco.
Yeah, because you know me out.
I hacked in there, but every Saturday they come together at a cafe and then they hack together.
It felt very pure.
What was that?
Are you talking about like Jake and Noah and Zeke?
Jake, I only saw Jake and Dan Lynch is a good friend and Dan and Jake and there are a bunch of other people.
I saw like 10 to 15 people all building cool things.
I used to do that quite a lot in 2010, 2010, 2011 timeframe during my angel hack days.
And like the mission coffee shop hacking on a Saturday is a very pure endeavor.
I think we're going to, we're actually opening our office here, a big one, hopefully in the area, in the mission area.
It's pretty much the same thing. The idea is to invite Cosmos contributors or Cosmos.
Anything if you're working in a Cosmos ecosystem, doesn't matter what project, you should be able to come and hack with us.
So I'm going to make it a thing and I'm going to make it very accessible to ecosystem projects.
Need some hot offices for us bloody validated plebs when we're getting there.
If you're a San Francisco, you're a validator. You can use our office.
Is there a patent that you can use?
Yeah.
So I think, yeah, like offices are sort of becoming back in vogue now, right?
It's fun to hack with people.
Really fun to hack with people.
So something interesting that the Daudau guys do is that they just sometimes jump into like the discord stream.
And, you know, it's not organized or anything.
It's just a couple of them all want to work on something and they just jump in there and stream it publicly.
You can go in there and watch what they're doing and chat to them.
It's really fun experience for me anyway.
I'm not like a massive coder and I learn a lot every time I go in there and those guys are always willing to help me understand what they're doing, which is kind of cool.
Probably counterproductive for them.
I don't think I can code while streaming.
I mean, it's really cool to collaborate anyway.
Sorry.
It's whether you can talk at the same time, right?
It's kind of like the whole pair programming thing.
I find I've literally, I will not take a job if they say pair programming, you got to do it.
It's sometimes fun.
I'll give you that.
And pair programming, especially if your peers are at the same level as you, it's not fun when your peer is not at the same level, either higher level or lower level.
But if you're the same level in terms of experience and seniority, it's a lot of fun because you're like actually exchanging ideas.
And, you know, and one person is researching and other person is coding, you know, or you switch it up.
It's actually a lot of fun.
I got a lot of work done that way.
It used to be a thing back in the day, but, you know, it used to be a religion back in the day.
I hated that, but I like it, bearing with like my peers.
But when I do a mob programming, which is just another like another escalation of this stupidity in my opinion, but that's, that's, I've given enough hot takes today.
In this instance, they were working in like UIs.
So I guess they were just, you know, talking about how it looks and then just changing stuff and talking about how it looks like iterating.
It's like voluntary pairing or mobbing, isn't it?
Like, I mean, I think the problem, I think the problem is that you, if you work in the industry and you do this stuff long enough, anything that people spontaneously do, because they have a couple of people who they know and trust and they like and they want to do work in that way.
They'll work that way and be incredibly productive because you can't replicate that experience of working with people you like and trust and respect and work well with in the way that you want to work with them.
And then people see that and they go, ah, the thing that must be good about that is that there were four people on the stream, not, oh, they're motivated to build this thing because it's a lot more interesting in building another fucking web shop or whatever it is you do for a big company, right?
And so they go, right, thou shalt work in fours.
And if you're a developer in particularly like, so my background is functional programming.
So I am already a spiky character on some teams because my design patterns, the way I do stuff, the way I think about problems is very different and opinionated.
And also because I've quite often been like a leader or whatever as well.
So I can be a little bit like, have you thought about this and maybe we're going to do it this way.
Use common lisp.
I mean, well, I mean, as you can see from my name on the thing, I am actually a lisp and I've spent the most of my career.
But you know, that said, like you end up in this situation where you're like, OK, well, you don't want to.
The reason the reason that the solution that you come up with here might be better is actually because of the clarity of your thinking.
It's not because of like working in a for or whatever.
And again, like it can be quite difficult to communicate other than with a working solution why an approach is better.
Like a really good example of this would be things like in reduction, reduce elimination of intermediate collections if you're using map iteration and reduce.
So if you're used to functional programming, you have the idea that you and Russ Russ has this so relevant smart contracts is that you have lazy evaluation, right?
So, you know, just like this idea ended up in Apache Spark, right?
It's the foundation of modern day processing, which is that you lay out everything you want to happen and it doesn't happen until you fold.
Right.
And you do like a right folder or a left folder or whatever, but like a reduction operation or an action or a collect or, you know, however, your language does this, you know.
And there's a whole bunch of different ways you can approach any given problem and it's quite hard sometimes to communicate though.
I don't know what I agree with.
I mean, totally.
Like when you're doing stuff like that, you get so ingrained into that.
The map reduces like to especially with functional, you multiple levels deep.
You're like five levels deep.
It's hard to talk and hard to explain.
Right.
So, exactly.
It's like whether you can stream and talk and also communicate with somebody else when you're filling your idea space with these things.
And I think some also, I mean, I think in the philosophy of programming languages, some programming languages are geared towards lone wolf programming because they give you much more expressivity, much more terseness at the expense of being less readable.
I mean, nobody's going to argue Haskell is more readable than go, right?
But you can probably write a 40,000 line go program in like 40 lines of Haskell in some cases.
Like if you like pure script is an example where if you see like a React application written in pure script, it can be tiny by comparison to the React equivalent and Clojure script has that property as well.
You can have a multiple less lines and you would in JavaScript.
Right.
But it comes at the cost of these languages.
They have extra inherent complexity because you can you have all these additional language structures like juxt and you know, whatever that are just quite alien.
You're writing calculus.
Right.
Sorry.
I was going to say something.
The house coffee is a place that yeah, I was going to say I think Alan is one of the gents who are frequent horse from the dad out guys.
Sorry, that was a bit of an interruption.
It's a really cool group.
I feel like I'm just going to highly recommend the hack.
So sorry, the phrase.
Did you have any more to add or I should not be adding anymore.
I mean, any more to the functional programming fire or the what's called fire the programming fire.
So I'm going to shut up and because I mean, say you're like literally an hour.
I was always late in show.
It turns extremely technical from the fray.
Right.
Like I started getting to some real.
It's like a symptom of him being tired.
I think it is like all all the filters go off.
I just start talking code.
Like if we let you go, you just start speaking assembly.
Yeah, my best work like traditionally like at the point is you know, you have like 90 minute sleep cycles.
Yeah.
I have my most productive period.
I think when I'm just on the crest down towards literally falling asleep in my chair.
So, so I think it's probably that like this time in the morning.
In fact, this is exact.
You know what it's one of my sleep cycles hits almost exactly at midnight.
So I'm actually probably about 10 minutes after the end of a fatigue point.
So I'm probably coming back up.
There you go.
I think my most productive time is fast becoming 3am.
I've started just waking up at like 2 30 and just rising like a friggin like a vampire out of my casket.
Just like directly straight up.
Did I miss an upgrade?
Okay, so you said what are you excited about next week in cosmos this week and moving all my stuff to a cache.
That's number one.
Anything that's not a cash is going to a cash.
I really appreciate this.
Hold the phone.
That was a bit of a great what's going to a cash.
Anything that I have like in terms of web hosting or yes, anything outside of well right now valid values.
I need really need to look more at that because we run a pretty large colo structure where we use some cloud services and things like that.
But I haven't really got to that point about running nodes where we run a tremendous number of non-validator nodes, right?
And to be able to reduce the cost of that and be able to be able to have there's they are femoral right now.
Those are things that they have uptime but are not exactly necessary in terms of in sorts of state right now, which I know is coming.
But right.
So having been a user of cosmos omnibus for regular nodes, it is very good.
Even without like, you know, if you're coming in from a say a poker to snapshot or one of their main team snapshots.
I'm not sure how soon they are how, you know, small they are.
But man, it's Europe, you can point stuff to it and Europe.
You just want to, you know, tune your resources so they don't get nuked when someone starts hitting them on the RPC port.
But I think you'll have a very good time if you want to try and move some of your resource into a cache and I'd be happy to like give you a bloody pointer on omnibus, which is super easy to use anyway.
Fun fact made the PR on that for the Juno edition when Juno first started and Adam the Liborism, which wasn't working on omnibus at the time.
That's my contribution.
Delegate the king notes.
Isn't and you surfer, if you're moving your stuff to a cache, and if you want to delete if you want to validate an acacia, but not only that actor said, we'll do the delegation because one of the requirements is like,
if you're running on a cache, if you're a validator, you got a higher chance of delegation.
So talk to me.
All right, I like that.
I think away from all that, like the, you know, we're all in the cosmos and we have a tremendous amount of positive energy towards projects that are going on.
And I think the projects to me personally as a, as somebody who runs a small business within the cosmos, an independent validator, ones that make the most that that called me the most are extremely,
are extremely effective in utilitarian projects that use blockchain as a method of incentive.
And that's kind of how I got involved in blockchain and not so much just around the defi structure of things, but how do I build?
It's impossible to build a future vision around a really grandiose structure without either an absolute shit ton of cash or a way to be able to incentivize tens or thousands or hundreds of thousands or millions of people to be able to buy in that vision.
And I think blockchain and I think the cosmos, especially with low gas fees and a low barrier of entry and everything else is the right tool set to build a tremendous amount of incentivizing around specific ideas.
And this is a great idea to build incentivizing around.
And it's just a worthy cause to be able to do that.
And so I didn't understand, I didn't really know the history and going all the way back to 2013.
I think it's fascinating around how this vision, it wasn't, it wasn't a cosmos based vision, right?
It was a vision around how to be able to solve a very complex and fairly scalable project, a problem, and then really kind of fate brought you together with a tool set that allows you to get a partial of the way there, right?
Yeah.
So anything to do to support that, man, that's awesome.
Like that, that's something that's a worthwhile thing.
So that's, that's one thing this week.
Otherwise, oh, let's see.
But that was a lot.
I think outside of that, we have, um, no, I think that's, I think that's it for me actually this way.
That's a great goal for this week and something I'm excited about, about making some changes too.
Nice.
She won't see.
I'm excited about the secret upgrade next week.
And especially with the soft, that's going to be joining us next week.
More or less during the upgrade, I'm really curious of how it's going to go down.
Is he going to folding out because there's not great to manage who knows, but I'm excited for one way or another.
It'll be, that'll be, I think mine and blockpains first secret network upgrade.
So I love the names they come up with for those upgrades, by the way.
Yeah, no, they put, they put some good effort into the marketing side of it for sure.
So I think that leaves this with me.
I'm excited about like upgrading some security around my nodes.
So we're moving entirely to TM KMS as an interim measure until we're more comfortable with Horcrux and deploying Horcrux.
I'm a little bit reluctant on Horcrux at the moment from reading an issue that someone else had where they double signed.
So making sure that we're outside of that narrative when we are ready to deploy Horcrux.
But I think that TM KMS is a good interim step to make sure that our nodes are completely key free in that area.
All our signing material is completely firewalled off and secure.
So TM KMS is remote signer, right?
Sorry.
Is TM KMS a remote signer?
Remote signer, yeah. So that's what I was saying.
I think TM KMS is actually a good fit for Cosmos Omnibus.
Because at the moment, you pretty much have to keep your signing material on S3 storage the way it's set up, which is a little bit.
It's probably, wow, I mean, it adds a vector that people could get to.
It's just another thing to manage in terms of security, I suppose, if you're trying to run an actual validator.
But like I was saying, the way Cosmos Omnibus is set up, you would have to rejigger some of the scripting and possibly, well, actually, that's not entirely true.
Because you can actually set environment variables for your config files.
So it probably would then, in that case, be very easy.
It'd be nice if you could create some automation to link the deployment address that you're assigned after you deploy to automatically update your signer.
That'd be fun.
We've been looking at TM KMS. I think we should use TM KMS. It's amazing.
Well, I mean, really, it is already an option for anyone who wants to use Omnibus because of the environment variable configuration.
So it's just another line.
Well, it's two lines, one to turn off your local key.
And then you just wouldn't add TM KMS.
Sorry, you just wouldn't add S3 and then point towards your, well, allow your TM KMS IP.
So which kind of, I think, were, no, yeah, something that worked just fine.
But just linking that on the TM KMS side of things, I think, in an automated way would be pretty handy.
Yeah, we should use that as some examples there, for sure.
Yeah, for sure.
Anyway, that wraps it for me.
Did you want to walk us out, Afra?
You haven't cut me off yet. It's amazing.
Yeah, no, I've spent a few weeks kind of basically trolling you too aggressively and I kind of feel bad for it.