Why can't anyone put a fucking Genesis out on time?
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent validator teams.
Yeah, well, nailed it. Nodes.
A weekly podcast on the cosmos from independent validator teams. And what are we going to talk about this week now?
Probably you, Serpa. I'm my brain's fried from not being awake, so I'm only in a question and answer today.
The background to this is actually going to be relevant to the topic we are going to be talking about,
which is that Evmos have just restarted. I'm not a validator and needlecasts don't validate Evmos,
but it's just restarted, right? And it was very early your time now, right?
Yeah, so it's two days in a row of like 2am. We had a dry run previous day on my Wednesday,
and then the main net this morning, which is my Thursday, both at 2am. So then I had a full day
after those, and I'm a little bit tired. I mean, actually, the dry run might have been later in
the day, like three or four. I can't remember. The days are all becoming one.
As they should be for a validator, why even have calendar thrilled away?
Yeah, I mean, it's not like we need a break or something after Prox 16, 17, 18, and 19.
Who needs rest and recreation? Yeah, I mean, to be fair as well, in that time,
they've been what, two Stargates upgrades as well I was thinking about, in terms of just like
chains that have Cosmos and therefore have the same security cadence, it adds up a lot of chain
upgrades, which again, put a pin in that because we will be coming back later in the episode,
right, Yusef? That's right. Yeah, we had a couple things to follow up on just from previous weeks.
We want to change the format a little bit and make sure that we're responding to questions that come
up both in Twitter and also on YouTube and make sure that we're bringing those forward on future
episodes. We did want to talk a little bit today about some chain halts in the Cosmos because we're
kind of seeing a lot of, I don't have a lot of halts, but maybe a lot of noise around pauses
and things like that. So I think we'd talk about that. And we have a couple of few good
questions for Ask Game of Nodes. We need to come up with an acronym for that, maybe Askon.
Askon. Askon, something. Yeah, so that should be it. Yeah, cool. So we're abandoning the stupidest
shit we've seen in Cosmos this week because we felt it was a bit too negative and we need some
more positivity. So instead, we're going to talk a little bit about stuff that's happening
at the moment. And to kick it off on a positive note, we obviously all talk about what music
we're listening to. And we actually talked to, I think this was maybe off the back of the conversation
with Shane and Cicela in the previous episode where we were asking what they listened to when
they're working and stuff like that. And I think Whisper, sorry, Ghost was shouting out NoTaker,
right? And that's then gone on to be rotation, I think, for the entire crew.
It's on rotation for me all weekend. Yeah. I mean, apart from the entire playlist of NoTaker.
Yeah. Apart from like, you know, I think there's a strong progressive rock de-gening going on
between me and you, Serpa, but there is. And there was some, yeah. Well, I've actually been
hitting those bands as well. I've been developing a bit of a playlist. Yeah. But you didn't notice
the didgeridoo in the Australian progressive rock, did you? So there was another song that started off
with didgeridoo, but I didn't notice this one that was at the end of the song. But to be fair,
I was like listening to this while I was doing work. So I wasn't fully concentrating on it.
Or it could have been coming from outside, right? Maybe just outside the window.
Yeah. Yeah. People just walk around there with their, with didgeridoos.
Pretty well. Well, I'm going to cul-de-sac. So it'd be unusual. Maybe I have on the main road.
Just circling. You know, the thing that really shits me about Australia was just the fact that
there are, you can just like walk into a field and that's just like a hundred wallabies.
Okay. Like, why do we keep, it's like sheep or something, you know, like, or like, you know,
the north of England, right? You go to the hills, there's just a field full of sheep.
And you go to Australia and just like, there's a field full of wallabies and you're like,
Holy shit. There's like, you just walk in there and just like with a bag and just a net or something
and just collect, collect the fuckers up. If you, if you're so inclined, it feels like, I don't know
how quick they are. Have you ever caught a wallaby now?
Oh, I think he's out. He's frozen. Oh, he's frozen.
There's a question. It's too heavy duty. I think it's, I think it's interesting that
no matter who's on this podcast, somehow it comes back to Australian culture.
There's always that topic. We should just put it in the topics because it's always like,
a didgeridoo comes up or a wallaby comes up or something comes up. But that's Australian
wildlife, not Australian. I was trying to be nice, but yeah, whatever. It's a mix of different
Australian. Yeah. So yeah, no taker has been very heavy this week. And then I guess we want to
also have a little bit of talk about the June upgrade that's coming right because Prop 20 is
now happening after all those props you knew and loved, 17, 18, 19.
So maybe to give an update for everyone who's not following this as much as we are,
anybody want to take a shot and see what's happening now?
You just don't want to do that.
I mean, so I read a tweet by the whale that I was curious about. Isn't it normal for a
binary to be assigned to an upgrade rather than kind of like the binary being
created after it was assigned? I didn't know you could do that. I thought that they were like
bound together one to one. No, just a tag. Just a tag. Yeah. It's just a tag.
So the binary exists, but it is dependent on like, yeah, it has to, it will run the thing that
it's cool. But as long as it's the same name. That's it. Well, isn't that only a Cosmoviser
type of thing that it's calling out? There's nothing actually in the binary with that name.
There's nothing in the binary. Yeah, you can just halt and switch over because that's what we did
for security patches, right? Yeah. So there's no connection between the binary and the actual,
no, it's just like you can put any binary in there. It just won't agree with everyone else.
So it's got to be coordinated. But the only thing that the prop with the software upgrade,
the only thing that's related to actually upgrading is the directory that you put the
binary in with Cosmoviser. If they're using that, that's the only thing. So you can have,
you can say that it emits the upgrade info.json, which is the output of the governance.
Yeah. And I want to say you, you could technically like have your upgrade and in your upgrade
proposal say you go into version three and then separately agree with all of the validators to
go to version 28. And I guess to clarify a little bit, I think it's more snippet to me that
the smart contract isn't actually merged yet, right? Isn't that true?
You know, that's bullshit. Oh, is it? It's available.
It's like it was not live for a grand total of about 10 seconds because I think I didn't have
admin privileges on the repo in the Juno organization to make it live. And so just had to wait for
somebody who had admin privileges to wake up because it was like, guys, this link's dead.
We just like make the link undead, please. So yeah.
I'm private.
Yeah, there was, it is true that the contract wasn't
publicly available before the prop went live. But obviously, like we wrote up a blog post about it
and the design was the same as on the original Unity prop thread and blah, blah, blah, blah.
So in that sense, nothing's really changed about that other than the code was written to conform to
the Unity prop and then it was audited. And then some more unit tests have been added as well just
after the tag that's mentioned. So in the text that's on chain, it specifically references a
Git commit on the repo. So it's like read specifically, this is the tag and the binary and it has a
a Wasm blob and a checksum. So the governance proposal is pretty specific about, unless there's
a huge security hole in this that's discovered between prop and hypothetically it being launched,
this is the one that is expected to run. Got it.
But yeah, it wasn't public before that just because it wasn't basically. I think it was
more of a too many cooks type situation because there are
a lot of feelings on this one and a lot of people and a lot of noise and the difficulty of finding
people who are just going to implement a thing because it's all open. So again, the thing with
the Unity prop is it's just an open source smart contract. It can be used to arbitrate between
any parties and it just kind of needs to be done and then checked over and stuff. And so it was
just one of those things that was done. So obviously we did a lot of work on it and we were against
the burn, right? But it is pretty clear, I think, to us at the very least and obviously Jaby who
did a lot of the work on the upgrade side and also on the testing is also against the burn.
But it's kind of back to, I think we were talking about this with economic incentives and whatnot
like a few weeks ago on the show when we were talking about the original drama, which is that
if you hold enough Juno tokens and you think they might be at risk of being confiscated,
your optimal strategy is probably to stall, right? Because if the token goes to zero,
you lose nothing. If you lose more, you lose nothing. You lose the same amount, you know?
So in terms of breaking the impasse, that's kind of, I think both, I can't obviously speak for
Jaby who is currently, I believe in the Jaby wagon, but will be joining us in a few minutes time.
I think that was kind of both of our interest in helping out with this piece of work is that
there are only so many devs in the Juno organization. There's only so many that
can help to ship this particular problem. And it's kind of clear that something needs to happen.
So even if we don't agree with all of the potential options on the table, it's kind of hard to see
how we can move forward, you know? Right, absolutely.
So yeah, that's anyway, that's the essence of it. And yeah, so it was one of those where all the
devs on the project, I think in the GitHub organization could see it during the process
it was being built, but we forgot to make it public when the first thing went on chain. So
ultimate tease, no, we're not teasing anything. We're talking about the Unity Prop Smart Contract
at the moment. So yeah, so that's where we're up to with that. What else needs to be covered,
sure to see? Sorry, I interrupted you a little bit there.
I mean, I think that's really the big thing is that Unity Prop is a software upgrade that
basically takes the funds from the whale and puts it in the smart contract. That's the summary of it,
essentially. I suppose I have a few questions run upgrade the next prop up to 20,
like whether there will be more security patches in it or is it specifically for
initiating Unity, but that's kind of neither here nor there.
It has a RC version of a Cosmos and Change which is tagged Junity for Juno Unity,
which is our submitted fix for the Cosmos and Pseudo bug that we found.
But it's not a security, no, there's no security element of that. But obviously,
we did do the security patch last week, which I think a lot of people didn't even notice we did,
which was Cosmos and Security Bug. That's different.
Okay. This proposal ends on the 29th. We still have a couple of days, right?
And then I think the upgrade block is farther out, right? We're early May or something similar
to that, if I remember doing the math. Yeah, yeah. We were also talking separately that
obviously there's some controls here around, is that enough time obviously to do undeligations
and those types of things? There are some questions about what might happen to anything
that's liquid or like be rewards and those type of outstanding rewards, but that's just
it is what it is, right? There's no controls over that and there's nothing somewhere to be able to
do that between now by the time that upgrade block hits, which looks like it's just about a week,
six days and 17 hours from now. Yeah. So because I mean, there's a lot of things that are going on
with this upgrade, particularly Harry, particularly controversial, I guess.
But the key thing is that, yeah, is essentially having enough time that we knew we could
get everything in place to actually do the upgrade and also crucially have enough time to back out
of the upgrade if we needed to. So something that isn't very often done as a cancel software upgrade
proposal, which can stop an upgrade from happening, that means you can just know up it.
And although we've obviously tested it, we're also rolling uni forward because,
you know, obviously that's test net that should go forward before main net does.
And so if there is something discovered on uni that is a showstopper, we need enough time to roll
to basically stop what we're doing on main net and then move forward in a more measured way.
Because we're running a very complicated piece of software here with the Juno chain.
Although we've obviously tested it in a variety of different scenarios,
it's only really when you use the kind of wireless 2020 25 validator set,
uni test net with lots of big smart contracts on it like JunoSwap and all this kind of stuff,
we can be 100% sure that everything is behaving as it should be.
So that's the reason we thought we'd build it in a couple of days and obviously Roar's going live
on the 1st of May. So we were like, well, we can't do it on the 1st of May. So we just said,
well, three extra days, probably fine. It's not a security patch. So there isn't a particular
reason to target a particular day. Okay. I guess we'll talk about it next week, right?
Yeah. So I was wondering, we might do it, we might attempt an upgrade cast, right?
Which might be the last one where some people will obviously remember when we did the upgrade
to, well, when we restarted the chain off the cyber attack, we did a game of nodes for that.
And then everybody got stuck processing invariance for anywhere between two hours.
Right. Yeah.
And this, the, go ahead, Noel.
Sorry, actually, just on that, with our forward planning here, we have potentially,
so two podcasts next Wednesday.
Or we just combined. On the same day. Maybe we just do it earlier and work it around the
upgrade block and see what time it happens. Yeah, we have potential guests next week.
So. Oh, maybe we do it twice. Can you have too much game of nodes? I don't know.
Probably not. I think the audience can be the judge of that one. That'd be a good poll.
Yeah, do we have too much damage? It's four hours too much.
Please stop and revert to one per month. Exactly. Hold what you're doing.
And this, the contract runs, so just so I understand from a smart contract perspective,
and the actions are going to happen when this upgrade block occurs. This will happen in the block
after, or I guess it's buried into the upgrade itself. So does that contract run as soon as
all the validators, we have consensus on the upgrade, everything else, then the contract executes?
Or how does that work when it comes to when the contract initiates?
So the contract is completely separate to the unity upgrade. Maybe that is something worth
clarifying. Yeah. I obviously said we need to cast, did a bunch of dev work. Other people always
see other devs and Juno contributed to all of the many parts there. Somebody needs to deploy
the Unity smart contract. I don't know who that would be, presumably somebody in core.
It's just a contract that gets deployed without an admin. So once it's been deployed,
it can't be changed. And then what the Unity handler does is unsteak funds and move them into
the control of the contract. So ideally, that would have happened consensually because what it
essentially does, what the handler does, the upgrade handler, amongst other things, obviously,
one of them is bump a couple of versions of things, but it bypasses the need to wait 28 days to
unsteak and then also have those funds at risk of being sold as liquid funds. So it just makes that
this block, I am saying the unbonding period is done. The contract. And then that's the
balance is then held there in the smart contract. So whoever instantiates the contract is essentially
just instantiating the contract with a representative wallet from CCN that they're known to hold.
Because obviously this contract can actually be used for any escrow situation, which has very,
very low trust in this situation. I guess we're talking about CCN, right? So they would be the
admin of that contract. So not the there's an admin that allows you to migrate the contract and
change the code. That's not what we're talking about here. There is a this smart contract has a
concept of the owner of the funds. And the owner of the funds in this case would be CCN. Right.
As we described in the episode previously, we did on what the Unity smart contract was and
how my work, they basically can trigger a withdrawal with a 28 days delay to get their liquid
funds back. And then on the other side of the coin, the community can send X amount of the funds,
send all of the funds or burn the funds. So it is essentially the staking part of the Juno
or the Cosmos SDK, if you like, the unbonding bit, the timer, but set to a Juno length.
But without the without the stake, without the needing to be staked in the first place, right?
Because from a developer's perspective, and I think from a lot of the technical people involved
in the project, the thing that concerns us most about CCN essentially having way more voting power
than it's having way more voting power. Sorry, here's the problem, I think for a lot of devs
and validates, look at that as no entity should have this much power on chain. Right. So that
solves that sort of straight away. As for the rest of it, it's a bit more complicated.
The way that works is that there is a special entry point to the contract that can be called by
the governance module of the SDK. So you can submit props that pass a message. If the message is
understandable by the contract, it will run. But the contract has to exist before that prop can
happen. And incidentally, this is also why off the top of my head, so Daes labs who are very,
very involved in Cosmism, very lovely people, very smart, do loads of great stuff running RPCs and
just loads of great stuff. Also Juno tools, which is a great piece of the year, heavily involved
in that. So Daes labs, I think like Needlecast had a similar opinion of prop 19, which is that
if the Unity smart contract existed, prop 19 could have actually been done as a Gov prop calling
the smart contract. And sort of like the community would have had to agree to it. Essentially,
them saying we want to be able to distribute from a Merkle drop contract to KYC people,
which is a reasonable thing to say. If they had gone away, KYC their customers demonstrate exist.
So using say one of the services that we as validators have had to go through for some chains,
which are quite thorough enough for you. For example, we had to provide a whole bunch of
documentation to prove that we were not a scam company because who's ever heard of the United
Kingdom anyway? Yeah, American KYC. So they were just like, what are these things? And you're like,
you know, this is like a normal business anyway, whatever. So they did KYC their clients,
there was a big list of them. And then they went, okay, these are the addresses, these are the people
who want a Merkle drop to we've instantiated contract, here's the contract address, they could
then just submit a Gov prop that says send funds here or send X amount funds here. And people could
vote on it. If it passed, boom, funds go. So this is why in short, some people who are more
experienced with the smart contract side of things like needlecast, like desk labs, a couple of others
said that we would have actually voted yes on prop 19, if it had come after unity prop, because
they would have had to implement the code for it, the KYC. And then, yeah, those are real people
fine. Like from our point of view, that's exactly this exercise. Can you take a second to clarify
what a Merkle drop is? I've heard that thrown around quite a bit, but I don't know, is Merkle a person?
So Merkle tree is the name of the underlying data structure, essentially that powers like Git,
a bunch of other things, blockchains, Tom Merkle. Is it Tom? I should, other than that, he was not
the he or they were an academic. Sure. So why is it called a Merkle drop then?
I think it's to do with the way the, so rather than it just being like a massive boom, here's all
the addresses, I think it's about the way that the data structure is encoded that represents the
addresses that will be dropped to. Yeah, uses a Merkle proof to store the addresses so that it
doesn't take up so much data on chain. Fascinating. Real-time follow up named after Ralph Merkle.
Thanks Ralph. Thanks for saving us that sweet data.
Yeah, went to Berkeley. What do you know?
There you go. That's all comes through Berkeley. That's right. Yeah.
So I've just seen this question. Does the drawer address in the own generator proposal?
They have permission to withdraw. What's that mean? Oh, right. So this is talking about the Unity
smart contract. Yeah. No, it doesn't generate a proposal. It's just a normal instruction to a
smart contract. So the only entity that can interact with the smart contract via the normal
execute endpoint is the withdrawal as they're called, which is not a very elegant name, but
naming stuff is hard. And yeah, so that's that. That's how the Unity smart contract works. And
like I said, we have an entire episode about this one before the smart contract had actually been
written really, discussing how it might hypothetically work if you're interested in it.
From a few weeks ago. Episode four, I say three, episode two, maybe two and a half.
So we're on. We need a list of like our episodes off to the side here so we can name drop them
for various reasons. Just to call us a few more. Which ones are which?
Sort of. Now that we've got more than two, it's hard to remember.
Yeah, well, yes and no. So the question, the thing that's just come up on screen is that
those are not real people. I assume that means that refers to the clients of CCN.
So I think the reason that we were talking about KAIC before is because like,
I mean, we will had to do it as validators for some chains. And
whilst not perfect, if you do KYC, it's pretty hard to actually game it effectively. It will take,
it's a little bit like the moon landings, right? You know, like the Occam's razor as it happened,
because if when you do statistical analysis, it turns out that in order to do the fake moon landings,
you need nearly as many people as did the real moon landings. And they'd all have to agree to lie
about it. So you can do a mathematical model to work out how long it would take on our
average based on conspiracies we know about that have been revealed that somebody would blab.
And it turns out for the moon landings, it's huge. It's like two years maximum that somebody
would just blab on their deathbed or something like that. And some economists or mathematicians,
they did this for a variety of conspiracy theories and then did a bunch of real conspiracy
theories that came out like abducting people and giving them LSD and all that sort of stuff that
did happen in the 60s and how long that took to came out, which wasn't very long spoiler.
So with regards to there being like potentially 30,000 clients that may or may not be owed this,
do you know, like being that they are real people and could be KYC'd Occam's razor is if you KYC
30,000 people and they turn out to be real people, then the orchestration work to fake 30,000 people
is much more work than just them being real people probably. And I know there's a lot of money involved,
but quite hard to organize that and then make sure nobody blabs out of 30,000 people. So
again, Occam's razor, I think comes into play there. I think that you just gave the longest
winded explanation of Occam's razor I've ever heard. Like normally it's just the simplest
solution is probably the correct one. And you really went out for it. Well, yeah, yeah, it's
because it's because like the simplest solution is the correct one. But but also when conspiracies
kind of thrown up, you can also say, well, look, there is also quite sophisticated mathematical
modeling that's been done on conspiracies and whether or not they do conform to Occam's razor.
And it turns out conspiracy theories and conspiracies are disprovable with Occam's razor for that
reason if you care to do the maths, which obviously I'm not mathematician, but I read a very interesting
article in The Economist about this and I'm passing it off as my own thing. We'll find that as well.
Yeah. I mean, honestly, I thought it was really interesting. I didn't realize that so much effort
had gone into kind of like prove the moon landing in such a way. Like if you think about you kind
of like, if it were fake, then why didn't anybody that was like filming it say it was fake kind of
a deal. But I didn't realize that there was actually, you know, models put to proving that it was in fact real.
Yeah. Yeah, exactly. And there's a whole bunch of there's a whole bunch of other examples of that,
which they did like 911 and blah, blah, blah. The earth is still definitively flat though.
Yeah, but I couldn't prove that the earth was round. At the end of the article, they're like
there they're like, but but moon landing stuff happened and wasn't inside job, but earth flat,
not flat. Don't know. It's a fucking dinner plate. Is climate change real though? That's I think
that's the real question. Oh, good. I'm just going to hedge and buy some carbon credits on
region anyway, just in case. Degen to region. Degen to region. There you go. So before
cheap at the moment, speaking of Degen, before we move on to today's topic of the day,
Frey, you had something in here about your Degen minute. Oh, yeah. Okay. So
probably set a stopwatch for this because I don't want to spend more than a minute on it,
which I can do with a smartphone, I think with the power of the stopwatch thing. So
I'm going to give you the keys to the Lamborghini.
That's a really old reference. Anyway, so some people will know that it's a name job,
a name service that we've been working on for a while. It's doing a name drop, which is for the
governance now. And we're basically been talking about when we're going to do the snapshot on the
team who's working on that. It's a little bit separate because there's lots of people involved.
So that's happening on May 4th. And the reason that we've said snap job and not name job is
there's actually two different components. There's the name job, which we've already talked about,
which is to users of the service. And there's also going to be a separate snapshot on the fourth,
which is a more general but smaller one, I guess, which is because we are launching a project
called howl. And they're both going to be brought together into the same project called the howl
dial. And howl dial, I love it, with a token called howl. And that is kind of where the
decentralized name service project is going. And that's one minute. And there will be a public
alpha, hopefully in June. And so that is the keys to the Lamborghini section. And that's my DGN
minute. Awesome. A minute of DGN from the fray. One minute of DGNning because we've had so,
so, so, so, so, so many questions about the various drops and whatnot. So now I can just point people
to this and go. So this episode and say, find the minute, guys. Maybe in the description of this,
we can put like just one bookmark is like that. Yeah, we'll put some links in there for sure.
DJ, that's exciting, man. That's great. Okay. So today's topic, we want to talk a little bit around,
I think, something that came up from a little bit of previous conversation, I think, within
the Game of Nodes extended crew. And then it came up in a couple other episodes here and there.
And I'm sure maybe I think this might have actually started out a little bit out of price
conversation, maybe is where this began, which I don't think we want to be able to talk about. But
the topic came around about the number of maybe recent or 2022 chain halts. And so we've had halts
on Juno and, you know, Ebose. And we've seen it yesterday in Akash and other types of things that
have come up for a variety of different reasons, whether that be upgrade based or maybe something
slightly somewhat malicious or, you know, XYZ, a bunch of different areas. And I think,
I think that the idea was that we could have a little bit of a conversation to say,
is this, do we think as a group in this conversation and extended, is this something that,
is this, should we be expecting this? Is this a sign of instability? Is this really a technical
problem or is this more of a branding problem, you know, in terms of halts? Does it actually matter?
Is it create or reduce confidence? You know, so I think that was kind of the idea is,
we see it and it really doesn't really affect, you know, as a validator, it doesn't really affect me
personally. I know that there's a very complicated system. And from my perspective,
there is a tremendous number of moving parts here, more than almost any other system that I've worked
on that usually is built around code with a very limited number of interactions. And here,
when you talk about, you know, more and more distributed systems with, that are global with
the amount of interaction and it's just the amount of development that's occurring.
And also, I think the amount of openness and really public pen testing that's occurring,
I think it's part of that, from my perspective, it's just part of growth. But I don't know what
you guys are all thinking about. And, you know, is it, is it just a branding problem? And should
how should we, should we somehow address that branding problem? I think for me, it's worth
dividing up the chain halts into like two different categories, right? It's like, you have a chain
hall during an upgrade, and then you have a chain hall due to like some sort of exploit. I think
those are very significantly different things. The question of a chain hall during an upgrade,
is that a big deal? In my opinion, I don't, I don't really consider it a big deal.
Wrangling validators like wrangling cats, especially the type of work global. And so,
whenever a chain goes forward within, let's say, half an hour, that's like, all of us have a cyber
leaf of like, wow, this was like the best thing ever. Whenever it takes two hours, not unexpected.
When it takes 12 hours, a little bit more unexpected. So in my mind, when a chain
halts during an upgrade, fine, moving on. That's okay. Whenever halts due to the exploit such as
what halted June the last time, that's, that's something I don't, I don't know about. I think
that's something that's inevitably going to happen. I think that there are a couple of
attack vectors that we haven't really addressed yet for chain halting.
Well, I guess I have a question maybe, which is that, how often do you hear about
improved work chains halting in that style, be it from an attack vector or from any other,
for any other reason, you know?
So I think fact is to consider when you, when you're pondering that though, is that
for the mobile, for in the side of Bitcoin, like Bitcoin doesn't do anything other than,
you know, make blocks with transaction data. It doesn't have any, anything more,
you know, that can make it more unreliable. And, but certainly for the Ethereum side of things,
but it doesn't have finality either for like certain amount of blocks. So I think there's trade-offs
for like, you know, that you're trading off like different things, but yeah,
it's still worth the conversation.
Well, so I guess my question is, if there's a chain hall on proof of work, that effectively
means a fork, right? Whereas proof of stick, it doesn't really mean a fork because we're all
like coordinated. So I guess that's my thought of like, well, the proof of work, it would just
keep going one way or another. And then for proof of stake, that just means everything
holds for a while. That's an interesting topic that I haven't quite thought about too much yet
and can't really speak of.
So I mean, I don't, I don't validate on any other types of networks like polka dot or
whatever other ones are out there, Ethereum. So I really don't know if they suffer the same
types of issues when you're, you know, getting random holds, whether it be like,
I think a lot of, I'm not sure exactly what the cash one was because I'm not a validator on a cash,
but I mean, we're sort of building custom, well, these, these networks are building custom
app modules, which sort of interacts with the, the tendermint as well. So it's, it's not just
smart contracts. It's also like a different methodology of making functionality on the
chain. So I mean, every time you do something in that fashion, there's a chance that you
bork something within tendermint or any of the other Cosmos SDK modules. So because it's all,
it's a different setup. It's modularized and people can sort of mess with things on an
individual basis. And it's got like a less, in a lot of cases, like, you know, the chain has got
a less of a dev network around it. Whereas if you look at Ethereum or polka dot, you know,
Ethereum's probably got thousands of devs looking at the code before it goes out.
Yeah. And I guess also they're more tolerant of forks, right? Because if you chew, if the new
software isn't adopted, then that like Schultz he was saying is a fork by default, right? But the,
I suppose like the need to coordinate is itself something of an, it has its pros and its cons,
right? Because it is something of an attack vector in of itself. And at the end of the day,
we're dealing with a black box of software, which the halting problem applies, right? So we don't
100% know what's going to happen with the code. I think I want to talk just a second about,
like, why is it that proof of work forks where proof of stake doesn't. So the reason why a
proof of work just basically forks instead of halts is because for proof of work, like,
it has infinitely reducing consensus basically, that can continue a chain going. So basically what
happens is let's say that half of the chain upgrades and half of the chain doesn't affect
like what happens is the habit chain that did upgrade, they will keep going off in their direction
and keep processing. And then the chain that didn't upgrade, they will also keep processing.
The problem is there, what they're doing is providing no value. And odds are that their
network isn't being used. And so then they have every incentive to immediately figure out what
went wrong and get onto the correct chain because they're not making any funds. These
these these computers aren't making funds anymore. Proof of stake, on the other hand,
you need that consensus. If that consensus isn't reached, then the chain isn't just isn't moving
forward full stop as a chain halt is. So with the proof of work model, if you don't, so say,
you know, 60% of the network or even 40% changes to a new binary. And then so it forks off,
like, which is how do you know which is the main fork and which is not the main fork.
So, so basically what happens with proof of work generally is apps will agree to switch over to
the correct fork at a given time. Right. So like, if they choose not to, that's how Ethereum classic
starts to exist. Right. Because people or whatever apps were like, well, I want to continue using
this. And so then clear Ethereum classic became something. So once those apps moved to the new
upgrade, that's whenever it's accepted as a new one. So is that is that proof of work in general?
Like does Ethereum currently work the same way? I don't know that it's proof of work in general.
That's my understanding of Ethereum specifically. Okay. Yeah. Well, it sounds like that's probably
how Bitcoin works as well. Yeah. I think Bitcoin's like a lot, a lot more simple in terms of like,
yes, you know, functionality. We're also exposing the fact that we're not Jen, we're not really
protocol engineers. So it's, there's, there's a little bit of black box behavior going on here at
some level. The thing I'm really interested to read actually, which I saw appear the other day,
but I've not read yet, is a paper by Dr. Martin Kleptman about conflict resolution data types
and Byzantine fault tolerance. Using those as a foundational building block, which I very,
very much look forward to not understanding until I've read it four times. But I mean,
that's, I guess that's more about data modeling in some ways than it is about,
I guess, chain architecture. Although, you know, there is a question. So,
you know, conflict resolution data types are kind of distributed systems primitive for sort of
resolving data coming from heterogeneous sources, but generally employed in a kind of trusted
environment. But it's, but I mean, if you, if they're sort of now writing about Byzantine fault
tolerance, that means that they're thinking about the application of that technology in
trustless environments, which I think is very interesting.
Is a kind of like different primitive to blocks, right? Because blocks are block hash,
next block hash. Whereas CRDTs are more, I don't know what program I was about to say,
they kind of feel like an ML type thing. But I actually don't know what programming heritage
they come from as a theory, other than a lot of people who are involved in the theory behind them
seem to have been involved in large distributed systems for, you know, the web giants, basically,
you know, sort of stuff like a patch capture and storm and SAMSR and blah, blah, blah, blah, blah,
you know, all this kinds of like streaming architecture, you know, heavily sharded,
web scale database type tech. So that's kind of interesting.
So bring it back, bring it back to the cosmos. Like if we get away from the POWPOS type of thing,
within within the cosmos, do you guys feel like these halts, like even the upgrade halt,
I think it wasn't so much like just a small halt for the, for, you know, waiting for validators to,
you know, build consensus around a new version or something similar to that. The Akash, when I
think the Akash issue yesterday and great, so I hope I don't ruin this, but my understanding of it
is the upgrade went fine. They understood that there was a defect in the inflation module that
did not show up on testnet. So they paused everything again, because the inflation was like
ridiculous. It was like, I don't know, somebody posted it was like 9000%.
They paused it again. And then during that pause, I'm not sure what happened, but they ended up
coming up with another defect associated to a cosmos SDK issue that had some sort of memory
type of problem that was in there, some sort of, some sort of incorrect point or something out of
memory issues, which was, I guess, a known cosmos SDK issue, but that was only within the last five
days, which, which means that was identified and there was already maybe a patch already there or
something similar to that. So they had to basically do up another kind of PR that was sitting on top
of that. And so I think that came up maybe, maybe it was at the same time as that. So as, as people
are restarting after this defect, then it paused again, please shut your notes down, all the kind
of stuff. And I think the, the outage wasn't that long. It was only, I think it was only like three
or four hours, if I remember correctly, maybe, maybe a little bit longer than that. And then,
you know, back to the Juno halt, that was purely, that's a permissionless contract structure, right?
So you, you have a lot of actors that are looking for ways to poke holes in wasm and wasm was moving
very quickly, right? Yeah, it's moving very quickly, right? So, so even those, you're not, not like
you guys are looking at, I don't, I'm not sure how far by that release goes, but I don't think
you're looking at releases that are six months ago that are extremely stable, right? We're looking
at things that are just being built because we want to take advantage of functionality.
So is that, it's worth bearing in mind, cosmoism is still not fully version one,
like it is version one, beta eight currently. What's a beta?
Beta, beta, like alpha kind of thing. Sorry. I think it's poking fun at your
annunciation. Honestly, Australian culture and UK pronunciation is all we have in the show.
That's it. Well, to be fair, I think, I think like if you'd asked me 15 minutes ago, 15,
15 minutes ago, 15 years ago, I probably said alpha and beta. Wait, alpha and beta, whatever.
It would have been closer to the American pronunciation, but I've been living in the
north for a long time. So everything gets sharpened. Everything's Eep, Eep, Eep, Eep, Eep,
Eep pronunciation. Sorry, I cut you off. Go ahead, please.
Yeah, I know. So I just wanted to say that the thing worth bearing in mind about cosmoism is
it's moving very, very quickly because a lot of this stuff is coming out as a result of
chains that are using it in the wild or audits or whatever. And it's moving fast necessarily
because it is new software that is still pre-stable in some ways. And when there is a version one,
that is like super gold standard, maybe that will be like an LTS or something. I mean,
I can't speak to their exact schedule and how they're going to, how Confio and Co are going to
actually run that. But maybe that's what they have planned, like kind of minting an LTS type
maintenance release. And then it will go from there. But the fact that there is
beta, alpha software and production is also part of the reason why there are so many
critical fixes coming in, I think, particularly on the cosmoism front. But it's also hard to see with
so many low-level dependencies of these projects that once you add together dependencies plus
transitive dependencies, it's a very large surface area. It's kind of hard to see how
it will slow down even when cosmoism stabilizes. So I guess my question is,
is it that we are seeing more chain halts and more of this fun stuff happening? Or is it just that
there are more chains in the cosmos now that are being more widely used and so it's more visible?
Yeah, I mean, I would definitely say that there are certainly more chains in the cosmos than the
last even six months. They've just been exploding out. But I wanted to circle back around to your
point that this is all beta software. That point really can't be emphasized enough. Like,
I had a meeting with Entertainment Foundation on a Thursday about entertainment accounts.
And they're like, you know, there's this problem where entertainment accounts, right now there isn't
really a good use case for them because right now, a relay would need to add a new channel for every
single account that was made interchange effectively. So right now, anytime there's a channel between
like, Evmos and Osmosis, there's a new channel for that. So every single user would need a new
channel. That was a Thursday. On Monday, a PR came out that fixed it. So literally in three days,
a new thing came out that resolved it that the person I had talked to, like, had assumed that we
were a long ways out from. And things like that happen a lot where, like, you go into conversation
and you both have the same understanding of like, okay, so this is just cool feature, but it's not
quite ready for prime time. Three days later, that prime time has been reached. And so that just,
as things happen like that, and since we're moving in such a breakneck pace, that's just the way it goes.
You know, estate, St. Wazem, like prime example, we're like that feature has been blocked,
that piece of work has been blocked or unfinished or still discussed for like a year.
I know Juno was talking about it when Juno went main there. And then it's kind of
poohled along a little bit here and there. And then like what they actually love about it,
and then like what literally last week, like Asaf was tweeting, oh yeah, no, no, no, we were pretty
much there. And you're like, oh yeah, so that one was like, that was like someone figured out that
you just, you know, sort of had a fix to manually make it work. And I think everyone just muddled
along with that for six months and like, ah, that'll do. And then, you know, someone, the big brain,
a wrinkle brain was in the background actually making it work, probably. I mean, it probably wasn't like
the three, you know, because I like, I kind of saw somebody was working on it and then
saw that the fix was announced like a few days later. So the likelihood is that it was actually
only announced the thing that people were working on when it was nearly done. Yeah, it probably had
been worked on for longer. But you know, there are a lot of things like that where you're like,
oh yeah, that isn't going to be ready for a while. And then somebody will be like, dude,
it was released yesterday. And you're like, what? So not only are there that, you know, the main line
is beta, specifically with Wasm, but you've also got forks of beta. So like Luna, for example,
which is like, you know, their own sort of bastardization of beta. So, well, it's pre,
it's pre version one. So it's not even one of the beta releases. It's not 16, I think, of
Cosmos. So, but I think they've sort of implemented some functionality to do with like interaction with
1.0, right? Like it's still, I mean, like case and point, I don't know. Like I know that there are
a few things about their model that are fundamentally different, like migrations,
like a few other things that do not function in the same way. So I know that there are points
where the code is just incompatible. And there are obviously things, there are things that are
incompatible between beta six and beta eight of Cosmos and version one. So I think it's probably
more stable, more unstable on the Cosmos and contract side than it is on the actual chain
level. Like some of that stuff will be able to interact, but you won't be able to deploy a
contract from one to the other. If you know what I mean, you probably have to write two versions
of the same contract and then have them do sort of, you know, do like receive and stuff if you
wanted them to interact. Yeah, but just like so it has to be specifically written to interact with
that other contract. Yeah, I think that's my understanding is anyway, because you know,
essentially you're just, when you do, when you're messing around with that kind of stuff,
you're basically just implementing a protocol on you to do the receive and send. Well, again,
that's actually, again, I'm kind of going to show my ignorance here because it's actually
slightly different, isn't it, depending on what you're interacting with over IBC? Because if you,
for example, use ICS 20, you just can send it straight to the bank module of the other chain.
So you don't need another contract to receive from the other one, but there are examples like
IBC Reflect. You can have, is it IBC Reflect? I'm thinking of, I think IBC Reflect is the name
of the thing I'm thinking of where you can see how you can interact with a thing that's on a
different chain and maybe like those kinds of examples. Chain halts. So yeah, man, I was like,
you're getting into the weeds here, dude. Are we kind of done on chain halts? Like they happen
there sort of because of fast development. They tend to get fixed reasonably quickly,
except for in the case of Edmonds, which took a little bit longer, but I think they had some more
fundamental things and they didn't want to rush it. Well, they are fundamental to proof of stake as
well. I mean, I think we've always had that too, which, right, isn't it? And like, and arguably,
it's a flaw of proof of stake, right? Because like, like, so if we want to talk about that,
there's something that there's something final interesting, which is I think worth
alluding to, right? Which is that so BitTorrent worked because the way it encoded incentives
was such that you were always incentivized to send the data, right? So all the clients fundamentally
would send the data because the way the economic consent, well, it wasn't economic because you
didn't, insofar as you've got something back, which is, I guess, still, right? You were incentivized
as a client to send data to another party so that other parties would send you data, right? But it
was correct. And it was an optimistic solution to the repeated prisoners dilemma, right?
Those kind of incentives have been core to how all peer-to-peer stuff has worked since and they're
in code. Those ideas are also encoding to Bitcoin and to some extent to Byzantine fault tolerant
solutions like the ones that govern proof of stake, right? Back to Schultz's point with the example of,
like, these validates, these nodes upgrade, these ones don't, these ones aren't making any money,
so they are incentivized to get back onto this version. That's really, really,
really core because the way in a trustless system those incentives play out is basically
the whole ball game. And you see that there's a very, very, very strong incentive to get,
essentially trustless nodes to act in the collective interest of a decentralized chain
continuing to mint blocks. Whereas how that's done in proof of stake is entirely a social dynamic,
which is itself an interest, like that's completely different because it's not done via arbitrary,
economic belief and essentially a mathematical model. It's done via social dynamics of 100 odd
validators agreeing to do a thing. And as soon as you have something that is explicitly socially
engineered, it can be attacked. And it can be manipulated. And it has that is a built and
vulnerability system. I think that that is an interesting thing, which we see by accident
with chainhalls, right? Yeah. But it's not impossible that it could be either an attack
vector or a response to governments in the future. I think that that's, yeah, I think that's a really
great point. One thing I want to touch on that is that because it's a social vector to stop it,
that means that a lot of the difficulty with chainhalls is they often happen due to
poor upgrade instructions. So like they'll say, you know, generally there's only a couple steps to
like do an upgrade. You check out the new branch, you install it, you move it by now or you restart,
right? These are the four steps. Now, sometimes you'll see like do an unsafe reset mid step,
and then this will happen with EdMos. This is when four or five validators
tombstone themselves, and then we have to roll back. Those upgrade instructions, someone that
writes good upgrade instructions is worth their worth, like, and they're worth basically infinite
value because they're keeping the chain going, right? There have been multiple chains where
like I'll go in, I'll start doing the upgrade, and I'm like, okay, so this is literally,
this is literally going to stop the entire chain, and I'll like put a small PR in that says,
well, we should do this instead. And then let's say I'd be like, oh, right, crap.
I don't mean to my own horn here, just the importance of like really good documentation
is just at the utmost importance. And because it's the social construct that you're talking about,
incredibly important. That's not even, that's not even like necessarily just for an upgrade
shield. So you like sometimes there's just chaff in the, in the installation instructions, or
it's just missing information, or there's like duplicate information, which conflict with each
other in the same document, because just care and attention hasn't been put into the,
the instructions, like I can't remember what chain it was, but the other day, I think it might have
only been a testnet to be fair, but I was trying to, like I was just going through their instructions
to look at it while I was booting up their testnet and see if there's anything different to the norm.
And I was a bit baffled by the service document for system D, because it had like
environment variables for cosmos in there. Sorry for a cosmovisor. And, but it wasn't for cosmovisor.
So it's, you know, people sometimes just copy, paste stuff and then don't even pick up these types
of things, which I don't know if it's because, you know, why, but it's not attention to detail
that they're putting into it anyway, or they don't know. I'm not sure. But anyway, paste for the
validators and the people who are, you know, involved with these networks to definitely review DOCS,
put some brain power into it when you're reading it instead of just blindly copy, pasting stuff
and submit PRs to fix stuff because it affects everybody and your responsibility as a validator.
Yeah. I do think that overall the documentation is getting better, though. I think there's a
lot of copy pasta going on across different chains. Like I think people still go back to
DOCS.Juno network or whatever it is and pull out of stuff that you've built and all and others.
And I still see a lot of that showing up everywhere.
Well, I think Meow wrote... Stargaze came after.
Either Meow or Jay Hernandez wrote some particularly good DOCS for one of the Stargaze test
nets. Yeah. Juno DOCS for the first Juno test net came out of that. And then they've just gradually
been shipped away out since then. So I think the Juno DOCS being good, you can trace back to the
Stargaze DOCS being good. Wow. In other words, you don't get that credit.
You bring a lot of delegations. Well, I think the phrase also done a lot of work on those DOCS as
well. Like we sort of chip away at them, I think, when either of us get time and update them and
fix them. And I don't know, rearrange those a lot out of the phrase. Certainly, I'm not sure if
you're referring to the actual setup DOCS that are actually in the GitHub or the actual DOCS.Juno
network.io. The good book stuff. Yeah, the good book stuff I think has shown up other places.
I see that being pasted around a little bit. And some chains, I think some chains also try to
for better or for worse, this is not an opinion, just of what I've seen. I think other chains also
give very little information on documentation to try to weed out validators that might not know
exactly what's there. Like I don't need the DOCS, all I need, give me a genesis and a binary.
What version are you running? What's a chain ID? That's all I need, right? And maybe a peer or
maybe a seed or something else are standing up a seed or something similar to that. But
and then you can see, you get kind of a different variety of validators that get
involved depending on the quality of that DOCS. And I don't think that's necessarily a good thing
or a bad thing. I'm just saying it might be different based on projects, based on what the
project's trying to do. And I can understand that, right? Yeah, so like, incentivize test
net that might be like test one, can the validators actually stand up a validator without
replacing their way to glory? Right. Yeah. And then I think some of those, you know, we see in
some of these other types of chains, which is just how fast can I write a script to spam the
faucet? Like that stuff, like, who are we benefiting with this crap? Right? That's a different thing.
But somehow we got off the idea of chainhults. But that's, yeah, back to documentation.
I mean, close up the, go ahead. Sorry, I was just saying, it is interesting as well, just in the
last year, how I mean, and certainly I remember like this time last year, it was, you know, a test
net was launching with spaces in the, you know, incentivize test nets were launching with spaces
in the test net set, because you could not get enough people, even with documentation, you could
quite readily pasta, you know, your way to glory as null put it. Whereas like now, you know, you
can you see some test nets, which get like 1000 Gen TXs or whatever. So again, it's just another
thing when we're talking about how, you know, how things are moving fast and how things are
falling over and all this stuff, you know, it's worth remembering that they were like, what seven
live chains in cosmos at that point, right? It's like 20 now. I mean, I joined secret because they
couldn't get enough people to validate for it. When I joined, there was like 40 of us. And then
there was another month or two before they ended up hitting 50 and that was back in, I don't know,
mid last year sometime. So if these have moved quick, as soon as the chain comes up, it's already
it's already a max every time now. Yeah. Yeah, I mean, and it's the floor is lava thing we talk
about a lot. The lava comes up fast. So I have a couple of other things to add in just randomly. So
weeding out, you know, poor or, you know, validators who don't particularly know what they're doing.
I suspect that sometimes people put the wrong instructions into a set up on a test net on
purpose and actually, you know, make purposeful mistakes to see like, you know, either to make
sure that people are helping people or to, you know, weed out people who don't know what they're
doing. But the other thing is like something I noticed for Stargaze, you're mentioning the
phrase that during that, some of those test nets, like they were putting bonuses like into
upgrade code, I think it was then and if you like picked it up by actually reviewing the code
before you just blindly run it. Yeah, it said don't run this upgrade. If you read this, yeah,
like, if you read this, don't do the upgrade out to Jay, I think. And I think nobody spotted it
either. Was that in the contract when they, oh, when they did the contract reviews before they
published before they govern in the, no, early test net, there was like an upgrade. And in the
in the upgrade handler was a no op and it said something like, if you read this line privately
contact Jay Hernandez, and if this happened in the real world, don't ask the code. Right.
And I think maybe nobody did that they were, I think one person did one person did.
Yeah, I mean, I don't remember who it was. I think consensus was definitely reached on that
handler, as in it, the handler got 67% or whatever. And so everybody got a little bit of the riot
attack reading to them. That's cool. I think we are, I think we are and Jay, we're like,
you know, had a little bit of fun throwing Easter eggs and stuff all over the place in
those test nets. I don't think it was just that one thing. I think they did like a couple of things.
To sort of, you know, see if people are awake or not. And some of us were not like,
I guess it was like, it depends, it depends like what, at what point of June is first few months
that was as to whether or not I would have been awake at that point versus very much not awake
and trying to be like, what is all this SDK shit? Which was definitely where I was last June.
It's been like, the hands up. It's just a lot of it. Like if you're trying to stupidly eat the
elephant and do the SDK and Cosmos and everything at once because you need to write documentation
and actually write code and blah, blah, blah. And it's just like, it's a lot of it.
So we have some questions and ask game of notes. Should we deal with them before we wrap up?
Yeah. Yeah, for sure. Yeah, for sure. So the first one, no, would you like to take this one?
Yeah, I'm not looking at the. Okay, well, I'm on my laptop. So the question we have on this
is this is ask game of notes. Why do validators dump? Why do validators dump? That is the question
for you now. Well, I'll start out on that because I've been called out for selling rewards as well.
So I think a lot of people get some heat at some point in their career for selling rewards.
So I think what the general public need to realize in some instances, like,
you know, our company is a for profit company. I think a lot of them are. But also,
we're running businesses and a lot of us aren't just anonymous people. We have companies and we
have tax to pay. And we have to think about the long term viability of the companies. We have to
have, you know, war chests for potential legal issues. We have to be able to cover, you know,
our server costs so that we make sure that we're running for the long term. So, you know, we could
be coming in right now to a lull in the market and who knows how long that'll last. And even
people are a long way up the the validator list. Their incomes are going to, you know,
evaporate pretty quickly. And certainly, I think when the when the price is a lot of validators
want to try not to sell as well. So it makes more sense to sell them while they're high.
And whether that's high going up or high coming down, like, you don't know where the high is or
where the lows might be. So I think in our case, we have other bags that we just constantly compound.
But it's more or less our policy that when the prices are rising to sell rewards.
And then when the prices are on the way down, we sort of sell them to a point that we think it's,
you know, diminishing returns and that it's actually hurting the market. And then we sort
of stop. And then we just roll them back into the validators or add them to the other wallets.
And then just keep compounding and hope for, you know, heydays again, nice sunny days and
you make our bails and then, you know, try and save it up for the bad days. But I guess as a
business, the long term goal is to be able to provide your service, rain, hail or shine,
and be able to cover any, you know, costs that might come up in the meantime, whether it be,
you know, unforeseen hardware or labor or, you know, legal expenses or anything like that.
You don't just run on a shoestring in the hope that everything will be okay. And then you don't
want to be forced to sell stuff when the price is really low. And it works different for tax in
different countries as well. Like I know the way our company taxes is incredibly different to a lot
of the ways like I think in the UK, the company taxes fundamentally different to how we operate.
And again, completely different in the US. And whether you're an individual or a company,
like each validator has their own set of circumstances. So I think we're quite fortunate
here where we actually benefit a lot when the prices go down, because it reduces our tax bill
overall if the end of financial year coincides with the low market. But, you know, that hurts a lot
of other validators because of their tax laws. They might be incurring
tax liability during the year. And if they don't sell their rewards or at least half of them to
cover that tax, they're actually, they're making income tax liability, which means that if the
token price goes down, their liability doesn't go down. They still owe tax on those high prices.
So to summarize that point for somebody who's not an expert in tax. So if you
work in this space for any other time, and certainly one of the core skills of a validator
is learning about the tax law and your jurisdiction, because it will determine whether or not you
go bust in your first year or not by accident. In a lot of jurisdictions, the second you claim
rewards, you owe taxes straight away. So if not everyone, but it's relatively common that something
like that happens, something is owed. So if you then restake those or do something different with
them that doesn't involve turning them into dollars or pounds or euro or yen, you might end up in the
situation at the end of the year when the government comes knocking for more money than you have,
and then you are bankrupt as a business. And that means you can't run a business for the next
five years, which would be very bad. So there's a whole bunch of things going on there. So
quite often, when you see validators dumping in any way, shape or form, it's because of,
it's kind of a function of that sort of thing, which is as Noel was sort of alluding to, if you
anticipate the market is to some extent volatile, and you don't want to be not supporting the market.
So validators have more tokens rewards as a set than your average sort of user of the network.
So what you ideally want is when there's an economic downturn, you want the validator set to
compound, not sell, because that provides price support. And it also signals it's an economic
incentive that there's strength in the network, right? And so in order to do that, you have to
make sure that you're covering whatever the value of those would be in the downturn when the times
are good. Like Noel said, on the way up or the way down, it doesn't really matter which.
So that or both. Yeah. Which is why you see a lot of people, both investors and engineers,
validators, whatnot, people who get paid in crypto cash it out on the day they get it,
or immediately restake it. It's the same phenomenon at play. And just for folks in the US, so in the
US, which might be the same elsewhere as well, I'm guessing it is. But for us, when we pull those
rewards, we are treating those rewards as income, and the commissions as income at the time, and
also at the price when that happens. So like what Noel was saying before is, if I have a slide like
we're dealing with right now, where Juneau has gone from $45 to $13 and a half dollars over this
time, I'm paying income tax on a coin at $45 at that time that I earned that amount, which was only
a few months ago. And so that is income at that amount. But the current value is $13. And when
I if I do sell that, sell that at some point in the future, then I can deal with capital gains,
which could be a loss or it could be a gain or whatever they'll say is. But it's extremely,
there's a lot of high risk for a validator in terms of the effect of price on if this business
makes sense or not. Because for things like this, when you're dealing with income that's coming in
with that way, it could be extremely, extremely negative associated to that, right? That's point
one. But one other point on this is that of the mint, so let's focus on Chihuahua, because Chihuahua
I think has been there's been a lot of talk around validators dumping on Chihuahua and those types
of things, which, which, and that's the reason that's driven down to 0.0003 cents from whatever
it was of the total mint that's occurring, right? Validator commissions, I think when I look across
the commission structure on all these fit out validators, I would say probably the there's a
lot at 5%, a little bit, a couple of 10 or 9% in terms of commission, right? So if we if we average
it out at 6% and say of the total mint that's occurring in the total total increase in terms
of coin, 6% is going to validators. If all those validators are dumping, from my understanding,
if I'm correct here, that over a period of time, 6% is going to validators, if validators are dumping,
there's still 94% of mint that's occurring, right? So something to piggyback on that is that
there's a lot of accounts that people don't see because they're not validators, right? So,
sure. You know, from that, from that drop, there's no, there's no cap in that airdrop on
Chihuahua. And there are wallets with hundreds and hundreds of millions of Chihuahua that are getting
the full reward, and they're getting more rewards than the top validator. And there's more than one
of those wallets. And then a lot of them are just selling the entire time that Chihuahua has been
which way I was being live. So the validators, they validate is selling on that network is,
like you say, a small percentage of what is actually being sold. But the thing is that people can
see the validators, right? Right. They can click on, they can click on King Nodes or Rhino or Lavender
or Bloody Needle Cast, and they can go to their public wallet. And they can see that these guys
sold and we're hurting at the moment because the price is down. We're going to get on Twitter and
flame the shit out of them. Right. And if you want to do them, fine. But yeah, just know that it
hurts those validators. I mean, look at the top 10. I mean, even the self-bonded percentage, and
even self-bonded is not really a good metric here because sometimes validators might have personal
wallets as well as wallets as well as a validator wallet, right? But nobody uses the validator wallet.
But like King Nodes, the most staking percent is like the biggest red herring imaginable because
everybody, the only time that is ever accurate is if you didn't incentivize Testnet and they say,
oh, whatever you used for that one is what we're going to airdrop to no exceptions. And so you end
up with your validator address just having all the locked tokens, whatever they're locked for 10
years, 12 years, whatever. Right. In almost every other situation, I don't know about you guys, but
certainly we as a business have, it goes straight in a cold key and goes into a bank vault. And we
never touch it basically, especially if you've invested long term. I would say that I have the
highest, and I haven't checked, but I would say that I have the highest self-stake in Chihuahua, I think.
No, no, no. I'll throw it down. No, you don't. No, you don't. Come on.
You dare, Reagan? I have more than you, and I'm 42 on this list. Yeah. I think...
When I say self-stake, I mean like my wallet is...
All your personal bags.
No, no, no. My validator wallet is staking 45 million, I think. I've been turning the rewards
back in, but it's not necessarily self-stake to my validator because I...
Oh, you're right. My bad.
You're right. I stake all the way down as well.
You're right. That's correct. You are better... See, this is a good example of the right thing to do.
You have 50 million Chihuahua that's in there and 20 to you and another 30 to others across the group.
That's noble. Provalidator has 10 total Chihuahua stake.
Yeah, it looks like...
Yeah, two.
Out of top 10, you are... It's no contest. You are obviously the number one in there.
Maybe even the top 20.
But I think the point of this question and why do they dump...
I think I asked this question when this came up. I forget who was
shitposting on Twitter about this topic, but so many. I can't keep track. But the idea is like
there has to be some rewards taken to be able to pay for what's going on.
Running these services are not cheap. Whether that is cloud-based services or
all you do from a colo in a mix of cloud or... They're not cheap.
I think that's come up, especially if you make bad decisions like using AWS for all your hosting,
things like that.
I was wondering how long my storage choices were going to take to come up.
An hour and 20, I guess. But they're not cheap.
I think Todd from BlockPlan just called this out as well, which folks who are listening to this,
you have attorney's fees, you have hosting, you have payroll, you might have accountants,
you have taxes, you have insurance, it's a business, right?
I'm really glad you did.
And so that overhead is... Sorry, but say it one more time. There you go.
So that overhead is not insignificant. And when you start adding these types of things up and
being able to pay yourself or you have a small team or other types of things or outside consultancy,
you might be able to use for media or other types of things, there's a significant amount there.
So when something like Chihuahua dumps from.008, I think is what it was, if I remember,
to literally 4% of that, yeah, you're going to see a ton of dumping because those bills
are not going away. Those bills don't scale based on what the price is.
The taxes of what is coming from income from six months ago, we're still paying for, right?
So those types of things, I think, are important components that I think everybody,
or some folks, I think, miss. And it's an important piece. So I think it's difficult to look at just
a one wallet or look at a validator or look something on one chain and say, oh, this person
is what's driving that cost down. I would think any validator, even all validators dumping,
would not have a huge impact on price compared to what's going on from a commission perspective.
There's one thing I want to add here. I think as far as your validators concerned,
you want them to be selling basically weekly, maybe even more frequently, because that means
it's very relatively small amounts that are constant. Whereas if they're building up for
two months and they dump it all in one stretch, that's going to dump the price for a moment and
might recover mostly, but it's going to have a noticeable effect. I've seen this several times
on secret where it's coming for some reason over there for people to hold it for a long time and
then to do one huge sell where you can see because they're not selling, they're selling on the market.
So if you're validators selling, you know, pulling, let's say every Monday, selling that Monday,
and they keep doing that every week, it's basically just going to remain
in this constant pressure versus one huge event. You want that reliable system rather than a big
event to occur. Yeah. From a validator perspective, that's tough to do because I think Nall said
this earlier, like when prices are going down, man, I'm holding onto this stuff because
it's just painful to be able to sell. Stargazing example of that, I think that is so incredibly
undervalued. Even getting rid of any of that seems horrible. That's not a mistake. I'm going to
ask. That's not investment advice. I know, right? I know. It's a horrible approach to it, isn't it?
It's a horrible approach. Don't listen to me. That's a bad advice. You should. So, but I think,
I think in others, I think some of you guys are much more, like you said, Schultz is much more
structured around that, which is, I don't care. I'm not going to look at the price. I'm converting
it at this date. And so every Sunday, I'm going to do it and whatever it is it is. I think that's
probably a much smarter way of being able to think about it over the long term. I think some
people's strategy is to just look at what I do and do the same thing. That's my hope. I know
at least a couple of people that what Nall does filters down. That's the Rhino tagline.
But you go back to Blockpain's comment that you highlighted a minute ago as well. I think
there's one final thing about that that's worth drawing out, which I think people don't often
think about, which is in amongst the obvious things the hosting and the less obvious, the
legal fees and insurance. Like, damn, nobody thinks about insurance, but that shit is expensive.
There was also like, maybe employee benefits and more of a US thing, because health insurance,
whatnot. But payroll is, engineers are really expensive and convincing people to come and work
in crypto is really hard because, so to give you an idea, like my previous life, I worked as a team
lead, managed teams still wrote code, but did a lot of hiring and whatnot. And for a big engineering
department, 60 people, 80 people, you're backfilling like a quarter of your department at almost all
times with contractors because you can't hire enough people. And you're paying above market,
you're doing everything you can to hire. And you would have four or five senior engineers,
principal engineers doing two to four hours of recruitment a week, like calls, first interview,
second interviews, tech tests, reviewing tech tests to hire nine people a year. When you're a
validator and you're doing this as independence, that stability of not only saying we are a stable
enough operation with stable enough income streams that you can trust us to pay your salary next month
is super, super, super hard. And you have to be credible as somebody who could say that with
a straight face and convince somebody to leave what may be a very stable situation for them
to come and work in this space. And I think all of us have had the situation where we've looked
at our cash flow for the next year. And for some of us, the tax year is ending now. So you start to
do projections for the next year. And you go, okay, well, can we bring more people into the
operation for, you know, there are other projects, there are, there is a lot of stuff involved in
running a validator, maybe there are more chains that you want to move into and or teams that want
you to help out with X, Y or Z. And that thing where I think a lot of independent validators
basically do everything with one, two, three people tops, maybe the odd contract hour here and
there from people they know that will do the odd bit of contract work. But they're running on quite
a shoestring in terms of development hours, you know, or, or any hours, right, infrastructure
monitoring this that I mean, I think that's one of the other reasons that way, I think this
GON started, which was, which was just validators helping out because whether that is being able
to be informed about something or snapshots or helping with this or I need a snapshot. Somebody
helped me with X, Y, Z, right? Like, yeah, it's very difficult as a small kind of, I guess,
business owner, if you want to say that, to be able to do it all yourself or even with a small
team because it truly takes more than that, right? Well, I think from memory, the original thing that
we all started chatting about was letting each other know when our nodes were down, right? Because
sometimes you screw up, you know, especially before we all got onto block panes, tender duty,
some of the stuff, I mean, particularly Grafana, I'm looking at you like was quite flaky in terms of
actual good uptime notifications. I'm slowly changing everyone to Zebix.
I have a double tap notification system now where I have my own homebrew one and the other stuff,
so that I get two. If I know if two happen, then it's real, it's 100% real, it's on.
That's how paranoid I am. But like that was how I think this conversation started happening, which
was, you know, yeah, essentially having snapshots and things to recover and also because we were
already talking to each other, going, dude, your notes down. But we found the communities
that sort of help us out with that. But like, I think all of us, you know, look into the future
and go like, oh, wouldn't it be nice to have a couple of keen young interns or whatever, you know?
And that's a kind of part of the, you know, like
when you talk about building tech organizations, like we're obviously
unrelated organizations or the validators, but there is like, you know, there's a social set
of everybody. And everybody is trying to professionalize to a lesser or greater extent
or at least stabilize their operations. And that's like very common in all organizational growth.
You have sort of like a pioneer period, then you have like, town planners come in and then you
build a city there. And then that's a stable technology. And we're like way off in the
wild, wild West, where we're still pioneering. And all of us are just trying to work out a way
of getting to the point that we can actually start like putting a well down
or whatever it is you guys did out West in America. And to be fair, I guess,
rather than recently in Australia, at least in terms of putting in,
you know, more modern infrastructure.
That's right. Hey, our internet can get a solid 50 megabits per second on a good day
with fiber, which is honestly, which is amazing. It's incredible. When you actually think about
the global component of that, like if you actually think about how much data is being moved there
and the location and like it's, I mean, it's maybe maybe it's my age showing. That's fascinating me.
It's very carried on camel across the desert. Yeah, I used to live in Chicago. I remember
having a single and dual. I remember going from single channel ISDN to dual channel ISDN,
thinking that was unbelievable at 256k a second, right? Like I remember those days of like actually
getting when the phone rang, you went from 256 to 128. I was like, shit, somebody called me. I'm
losing like downloads and those types of things or even 300. I go way back. Yeah, call way. We
fuck up. Exactly. That used to happen. Yeah, you see, I do that. I forget what the star number was.
It was 56k to bro bomb was like a revelation, wasn't it?
56k to ISDN was because I remember moving from 300 to 1200 to 24 to 196 to 56k to then to ISDN.
And then the first time I actually was on broadband, it was like, it was unbelievable.
I remember having a T1 in the office. I thought that was incredible. I think I jumped straight
from 56k to broadband. So it was just like, it was like, can't load a page to like light speed.
And I don't think I've ever, it's never been that much of a jump.
She'll see it. She'll see it. No idea what the fuck I'm talking about right now.
Oh, I know what you're talking about. Okay. I mean, the irony here is that like,
I think that a lot of torrenty came about because they needed some sort of technology for like,
how do you continue download over it ends, right? And so now we're kind of coming from a circle
that where blockchain is basically, it's basically torrenting really. If like,
if you want to extrapolate it far enough out, that's basically what it is.
And so now we're using it in some sort of global scale to resolve something that we thought was
resolved 20 years ago. And so while there recently becomes a long that makes the blockchain
redundant again, I don't know. We'll see. I don't know. We'll see.
Yeah. And Roman just talked about Starlink as well. Starlink, like just watching that kind of go
where it's going has also been pretty amazing. Like that, that type of technology then opens up to
say, where do I want to live? Right. And then other types of things that are important to me,
which is unfortunately bandwidth is important to me. But having some stable bandwidth and those
types of things, that means you'd be able to open up anywhere you want to live. And you put
middle of Montana with a, you know, 200 megabit. What the heck, man, it's crazy.
So I have two things that particularly around Australia, one thing is that our population
density isn't very big and our country is incredibly fucking huge. So the cost of our network is
enormous per capita. Right. And which makes our internet not as cheap as it could be. But I think
like, you know, the government's just taken a pretty big L on that because our broadband
networks owned by the government here because no private company in their right mind had built it
because they would never be able to profit from it. But I think one day our publicly owned network
will probably get privatized out of discount. So someone can profit off it at some point.
Yeah, that's kind of the way, isn't it? Yeah, someone makes the money at the end of the day,
who's probably buying an official lunches right now to get in their pocket. Anyway,
right. So on Starlink, I just pulled a Starlink out of the box yesterday
and plugged it in business Starlink. And my God, I was just astonished. So latency isn't that good,
obviously, but it's better than a normal satellite by a long shot. So I had like,
this is in Armadale in New South Wales. It's in the middle of nowhere. And I had just sitting on
top of my car with the satellite dish, a latency of 40, a download speed of 220 megabits and an
upload of about 25 megabits at that point. And it was cloudy. 40 milliseconds to where?
So that was from a SpaceX point, I think, to the speed test on the phone.
So I think that that demarks back to somewhere within Australia, right? It doesn't like automatically
demark to California or something like that, right? Well, it probably, so they go to the lowest
latency public server, but it would have been on the surface, but probably in America somewhere.
Yeah, would have been the endpoint from SpaceX. But I mean, it's it doesn't particularly matter
so much. I don't think where the point is because they've got stations all over the world. But in
terms of like the round trip would have been like 40 milliseconds for a satellite because of the
low orbit is low earth orbit is like, amazing. Like we have, yeah, like we have technical issues
trying to get internet out in the bush. And that is a revelation for us because we there are no
other options. Other than if the if the cell network doesn't reach there, and there's a lot
of black spots on the cell network that aren't on public roads or in cities, you go out into the
bush somewhere and you need like even for emergency services, you need some sort of deployment for
your emergency services. Unless you're getting like iridium go satellite from our main carrier
and paying a shitload of money to get it there on a trailer. Like you've got no options. The
bloody starlink I could plug into my inverter in my car, because the anywhere on the roof,
yeah, and have local Wi-Fi for anyone that's there, whether they're an emergency service or
what have you. So like it's pretty revolutionary for us. That's pretty cool. I know there's some
SSH clients out there as well that are I think built around that were because SSH obviously is
every keystroke right where you might feel that latency. But I think it just brings it all local
then whenever you hit enter, it goes right. So you can have some structure where it does some
work where you don't really so that 40 milliseconds might not feel like 40 milliseconds, even if you're
going to Hetsner or something else in the US or whatever else just because you know, I'm trying
to trick it a little bit. So I'm sure even with that, like you would feel like you're right next
to it, right? Pretty cool. To be fair, I'm not SSHing over a satellite. I'm watching YouTube and
Okay, so after I extended return to the bush and Noel having gone walk about for a little bit there,
shall we do what you most excited about this week and wrap up? Yeah, absolutely.
Yep, I'll go. I feel like I've just just insulted the whole of Australia. And if I ever go back
there, it all comes back to Australia. We like that episode Simpsons where the Prime Minister
of Australia bends over Bart Simpson and gives him the kick in in the ass.
On a live broadcast, you took the piss out of Australia and therefore you're going to pay.
So you said, what are you most excited about this week? I look forward to dumping all my
Cerberus and Chihuahua and everything else just selling all rewards across all chains just to
try to dump the price as much as I can with my small bags. No, I think this week it's great
that I've most started back up today. I think that's great. I know there's still work to do there
in terms of enabling ledger support and inflation will come back and those types of things. I think
that's great. Kind of on the heels of some of the stuff related to dumping, we're getting ready,
as it is more of a personal thing, but from a Rhino perspective, we're getting ready to
announce something associated to doing some kind of large scale donations from the rewards that we
earn. It's something that we've been kind of working on for a little bit of time. So we're
getting ready to relaunch a new website and some other stuff to relate to how, why it's named Rhino
and what we think about that and those types of things. So that's all coming in the next week or
so. So that's pretty exciting as well. And I think that's it. I see no and defund finance and
Althina and all these other types of things that are the next wave. I think this early 2022 wave of
test nets are happening, which is also pretty exciting to see some of these projects moving
forward. So that's kind of what's up for me. Yeah, she'll see hit it. Yeah, I go next.
Let's do the cosmos itself, but I'm actually really excited about, I'm gonna start playing
up Horcrux this next week. I'm gonna try doing some more redundancy type missions, if you will.
And it scares the absolute hell out of me. And so naturally, I'm pretty excited for it.
So that's what I'm looking forward to next week.
I'm actually on a, yeah, not completely cosmos thing, but more of on a personal company thing.
I'm excited about our company evolving, especially in this lower red period in the token price,
like a lot of the heat comes off in terms of like, you know, the fast pacing of things and like,
you know, users tend to weigh in a little bit when the when the prices are low, and that gives us
time as a company to work on things like website and also our company is working on policies,
which I want to be a big part of our strategy moving forward. I want people to be able to come
to our website, see our strategy in our policies and know exactly what we're about. So that'll
include things like, you know, our token selling policy to support our business, our network
selection policy, our governance policy, how you can expect that we'll vote, and what our,
you know, goals are for governance voting and things like that. So we'll have a dedicated
policies page with all of our policies listed out. And I really want to see that live, but
it's a little bit of work left in that before we can get that out. But our new website's almost
completed. So I'm pretty happy to get that out. And thanks to our validated friends who pointed
me in the right direction to help me get development for that done as well.
I look forward to plagiarizing all that content for my own use.
I was just thinking exactly the same thing. Well, typical great ideas organizationally,
I wish we had thought of that. We've talked about this, but I think we all have a pretty
similar approach to what's important as an independent validator. And I know you're working
on it. No, I'd like they'd be good for us to, I want to be able to collaborate with that too.
I know it's kind of a personal thing, but also kind of a community thing. But I don't want to
be able to produce something that follows too much to what you're saying, because I know that we
agree in terms of how we approach it. So it's kind of a challenge.
So to me, I want people to be able to, it's important to have it on the website. It's
important that when people go there, they can go and see what you're about. And I think it spaces
you from directionless, at least you can know what you're going to get when, if you come and
stake with King Nodes, you know you're going to get this in governance. You know you're going to get
these going to be selling at these particular times or in this market condition. You know why
those types. I just think it's important for the delegators to know what they're getting into.
As an aside, somebody asked earlier, and I'm going to put this in our notes for
Ask Game of Nodes for next time. But you know, whether or not it makes validates uncomfortable
when community asks them to show their reasoning on particularly like proposals, but I guess like
broader anything. And I think this is a good example. And like I say, we're wrapping up now,
so we'll maybe come back to this in Ask Game of Nodes next week. But it is, I think, you know,
we were talking about this in Relation to Prop 16 in the past, and we've talked about it on several
shows as this kind of, if we're playing around with social consensus here, then one of the things
that I think it probably is important for validators to do is think about these things and present
their reasoning. Because like you were saying, just then you said like, when one validated presents
their reasoning, it tends to be that we then have to talk about it. And that might be as simple as
something which is like policies, which is a good thing for us to be doing, or it could be what
happened with Prop 16 where, you know, Shortsy wrote a really good write up and Polkutu wrote
a write up. And like, you know, a lot of the validators actually wrote up their thoughts
and then talked about them. And that was hugely important to people even decide, because there
was a point, I think, where we were looking, things were looking much more like a really
you know, adversarial situation, a hard fork, something like that. And I think everybody
being able to come back around the table was very, very largely helped by that
people showing their reasoning for any given thing. And I think if your validator is necessarily
public, right, so part of what we do is have to represent the network. That's part of why you get
paid as a validator, right, is to be doxed and public and have to share your reasoning to some
extent, right? Maybe I think I think ultimately is a validator though, like, we're motivated by
what's best for the network ultimately, because the network surviving and being successful is in
our interest, right? So, yeah, I guess ideologically, like, it's, of course, insensitive. Yeah, I mean,
some, I mean, and, you know, some validators don't even care, you know, that much about the
economic incentive over the ideology of a blockchain and others are more economically
driven, but regardless of which camp you're in, or a mixture of the both, ultimately,
your validator wants the network to survive and they want it to succeed, because it's
in their best interest. And I think if I could just say one thing, like, as we close out with
the Juno props that are going on at the moment, no matter what your opinion,
please don't shout it at people, let the let other people just come to their own conclusion
and make their own decision. So, I get people like DM me asking me which way I'm going to vote on
things and then trying to convince me one way or the other, like, that's not the way to do it.
Have your own opinion, make your own vote, but let people arrive at their own decisions and don't
shout it at them what they should be doing and make them feel obligated to do one thing or the
other. Yep, totally agree. Yeah, but then that's also a function of the fact that you're Australian
and Australians don't like being told what to do, right? Yeah, except, you know, if you
contrast that with our voting over here, we have to vote or we get fined.
So, but we have anonymous voting here for governance, I'm talking about it with the
government, federal and all of our elections. If you don't go and make a vote, you get a fine in
the mail. So, you have to go and vote.