0:00
and then we thought W basically all right you did actually you did put it live
0:12
[Music]
0:21
hello we don't have any viewers [Laughter] that's a good good professional way to
0:28
start the podcast this week hello and welcome to game of nodes a weekly podcast on the cosmos from independent
0:34
validated teams and Welcome to our handful of very dedicated uh listeners
0:40
who are very definitely like us not at Cosmo verse um I don't know presumably there's some
0:47
sexy party although it's like isn't it like one o'clock in the afternoon there right now so probably not a sexy party probably uh gonna just predict it's
0:56
going to be a panel on something where a lot of people are
1:01
bullish and everybody on the panel is male
1:07
engineer yeah no okay so I'm gonna stop trolling because these are literally our friends but
1:13
we can we it's yeah last customer first was good obviously just jealous but not that um so uh We've we've got a break in our
1:23
very very heavily well organized uh run of shows with guests uh do we want to
1:29
see who we got on for the next couple of weeks since we got it planned yeah of course yeah
1:34
yeah or should we save it to the end we can save it today well let's say let's save it to the end so this this week
1:39
this this week we basically we're just we're just gonna we're just gonna charge it so um we've got the we had uh what's it
1:46
called we had the Juno upgrade today there was also Yumi as halted uh we had
1:52
evermark upgrade qg upgrade was there an avos upgrade as well or did I imagine that
1:59
um yeah there was a couple days ago
2:05
yeah I mean it's busy night for upgrade [ __ ] up all night for upgrades I had to set my alarm like three times to get
2:11
up and [ __ ] do upgrades it's been like it feels like even we're obviously not on on avmos but it feels
2:18
like or or osmosis is typical where there's just [ __ ] tons of um gov props and stuff all the time but I feel like
2:24
the the calp attack um notification bot has just been going off constantly this week so like [ __ ]
2:31
Fuji man they've got like five props a day [ __ ] that's annoying it's the new uh it's the new osmosis
2:39
well that's progress for you I guess uh the Dex world is just
2:45
yeah you know that you know the meme where it's like all those soldiers holding up the town and then there's
2:50
like all the blood and there's like the family having a picnic like over the beach like that Meme where
2:56
it's just like validated voting on props for the delegators that's that's how I caption that meme
3:05
[Laughter]
3:15
when there's an appropriate time later I'll see if I can find it in my memes folder which I definitely have on my on
3:22
my laptop and uh oh my God okay I have a meme folder that's version
3:30
controlled in git just because like you know when you see like a good meme you're just like am I
3:36
not like here today gone tomorrow right meme the meme economy is is fast right
3:42
yeah Easy Come Easy Go you're not gonna you're not gonna use it again but like in a couple of years you might
3:47
be like yeah what was that Meme format that was like quite hot in 2017 with like you know the trombone kid following
3:54
the other kid or something like that and like you just need to like reference what the format was so you could like
4:00
you can remember it in your head to like describe to somebody so yeah people like a commit description when you commit a
4:06
commit a meme um well as anybody who's ever worked with me knows my commits even on
4:14
business critical code are usually pretty roping update update typo
4:23
update stuff well they're either like they're either very eccentric or they're just kind of
4:29
trolling like you could in in the just before Hal Alpha went live when we were
4:35
bug fixing there's like the lyrics to A Whole verse of Another One Bites the Dust by Queen but just in chunks like
4:42
one line at a time per commit um it's pretty good yeah because you have to do it backwards as well so that
4:48
it actually shows correctly in the UI uh sorry in the in the yeah well anyway you know this is the kind of [ __ ] idea and
4:54
then people are like well what actually was in what what actually is this Branch because all the commits are trash uh
5:01
Fray and you're like oh yeah that is the problem yeah yeah I'm that guy I'm that
5:07
guy that hides asciard source code and then eventually somebody finds it and they're just like why did you do this this isn't even that right I don't even
5:14
remember doing it this just makes it bigger and you're like well clearly you just get the compiler to remove comments
5:19
and that doesn't make you bigger anyway uh I actually created a spicy meme this week did you create it now yeah but like
5:28
no one understood it so I didn't distribute
5:35
[Music] yeah no you said you said you wanted to talk about some
5:40
so what's your what's your take what's your take on atom 2.0 from from cosmiverse I didn't say I wanted to talk
5:47
about Adam well you just did and you put it on the spreadsheet yeah
5:52
number one he's never put anything on the spreadsheet so you knew that was wrong as soon as you said it hey
5:58
this week oh yeah yeah I did yeah it's on there
6:04
yeah I think about Adam 2.0 question mark was my it was my addition to the
6:09
spreadsheet is the question mark should we talk about it or is that your review
6:17
I'll explain that name right so it's my understanding that atom 2.0 is
6:24
like trying to change the inflation so it's from this constant whatever it is to like taper out to zero like like
6:31
minus one percent a year and I thought that was funny because whilst the
6:36
inflation's been constant all these all this time the people who have been like just
6:44
getting money from the ICF delegations that are crap node operators have been raking in the the atom and now
6:51
like just as they changed to a decent delegation program they like snatch away the inflation
6:57
so I thought it was a funny meme but uh you
7:02
know maybe I'm just not a funny gag that's not me yeah yeah I mean I needed to explain to
7:09
me but that's also because I don't obviously understand anything as soon as there's like you you weren't exactly
7:15
missed the current events on that one you're like what's what's what's that of 2.0 I mean
7:22
like I'm obviously aware it's a thing but I I hadn't like had time to read
7:27
like we're trying to launch a protocol at the moment and and you know there was this Juno upgrade you might have heard of that was uh that all came
7:36
that came closer to the wire than than I think would have ideally liked I think
7:42
uh we try to have our cake and eat it was the problem with that upgrade but but we
7:50
did that upgrade in uni right yeah so we we obviously have the uni one
7:57
which then uh had the little little wibbly wobbly problem with the cache
8:03
which meant right out of it so the problem there was like we we sort of we
8:09
the problem was we basically engaged in wishful thinking and if you engage in wishful thinking you get stung right and
8:14
we we knew that there was a new X wasm due out almost exactly when we wanted to
8:20
release so we were like oh well if we just push back like three or four days we can kill two birds with one stone
8:26
whereas what we should have done was just shipped what we already had and instead we tried to use the new version
8:32
and then basically had to then back out of that change um which it kind of ended up being okay
8:38
because we were spinning up uni 4 at the time anyway but obviously uni 4 R.I.P
8:43
um one block yeah it didn't make it didn't make it right um right and we learned something very
8:50
fun in the process but you know it didn't make it um
8:56
only having two days left so give me the give me the tldr on my uh uni4 didn't
9:01
make it because I just sat and forget and went to sleep so right so my my understanding
9:08
my understanding is basically State sync was set on a number of nodes
9:13
um right right and so I think there was several things there was several different there were only about eight or
9:21
nine nodes in Genesis maybe a couple more um and several were misconfigured
9:26
because people were reusing boxes um and essentially somebody one of the
9:33
nodes must have successfully broadcast um
9:38
essentially like a valid uh a valid block ahead of where we were and then everybody just app hashed
9:45
um and at that point you're like well we can't recover from a snapshot because we're on block one two sorry we're on
9:52
block two uh we've got past the Genesis block and then we've all that patched like there's no place to recover from
9:58
other than starting from a new genesis so uni 5 was literally the same Genesis
10:04
why can't you recover from uh one block snapshot well because block two was had
10:10
already been committed somehow we don't know how though
10:17
yeah so well I think the pro the problem was it actually wasn't block two that had been
10:22
committed I think it was actually like block one million two hundred thousand or whatever
10:30
um yeah but that's the thing that I'm not
10:35
clear it's because basically the final the final log that I saw I mean unfortunately I guess I've lost the logs
10:41
because we we recycled that box as well but the final log I saw was very weird
10:46
it was like it was like evidence received of block a lot
10:53
in this network but but they should be distinct because they are different networks but those there was obviously
11:00
something screw you going on but then you think that that could have also been maybe I'd locally misconfigure something as well which is why that node was
11:07
reporting a problem I think there were quite a few mistakes on I wonder if it's a remote
11:13
sign a thing like I wonder if if someone just changed the um
11:18
the network ID but still had the same state file on their remote signer
11:24
but that still has that still has the chain idea as well right when it has chain ID yeah yeah because that way you
11:30
can change it in your in your signup yeah
11:35
so that's I mean it's possible actually because I know quite a few people obviously are experimenting around with their setups on testnet which is the
11:42
place to do it um but I guess the difference between like testnet and mainnet is
11:47
on mainnet if you mess around with your remote sign of the only node at risk is you whereas I guess um there is I guess
11:54
a small chance on a test net of nine nodes that you might be the proposer I propose a completely mad block
12:02
I think it'd be like some sort of check and balance to see that you're incrementing
12:08
yeah well so this is the thing I think I think the
12:13
way tenement works now forgive me if I'm wrong somebody who knows the fundamentals of the state machine
12:20
better can chip in but I think as long as you broadcast evidence of a block
12:26
ahead of now other and it's accepted as a as a
12:31
proposal um I think you're all good basically like as in all the other nodes will go well I have evidence of block five
12:38
you've just given me 900. okay I mean I guess I didn't know about
12:44
the intervening blocks I guess the problem would be that you then wouldn't have any like there would be a
12:49
disconnect between the the Merkle route right and then it would blow up
12:55
you'll definitely disconnect yeah yeah because obviously those two histories are completely unrelated so that's why I
13:01
think you would in that case you would just fail to reach consensus right yeah and I guess it's just to be fair
13:07
that actually was what happened everybody was dialing and we lost consensus so maybe what maybe what happened was
13:13
simply that there were two mercaroons and everybody was like don't know everybody
13:20
right half of them had one and and I guess the proposer might have had State sank or that's making sense though
13:26
because States the three nodes with State sync on that shouldn't hit on but
13:32
then there's no data yeah but there's no data because the data would not be there because the node wouldn't start on uni4
13:38
right so if the data was in there and statesync was on all it would do is try to reach out to an RPC to try to understand yeah we're just trying to get
13:44
uni 3 State wouldn't it well it would probably try to get uniforce state from the RPC node that is
13:51
configured and it would start dialing for that but but I guess if it was also the proposer I don't know how that would work
13:57
um I don't know I don't know but it wouldn't be the proposal because there's no there's no State there so I don't know exactly so something very something
14:04
very bizarre happened but yeah the solution was essentially once the three
14:10
nodes that were configured uh with States and con were fixed the network
14:15
started fine so it looks like that was The Smoking Gun
14:21
but I also know anecdotally that a couple of people had issues with
14:26
um remote signers uh we we actually had problem with our
14:31
remote sign up but we actually fixed it the night before uh and then snuck it out in in the final
14:38
change to the Genesis because uh that's the privilege of being the person that compiles the Genesis
14:44
um but that was that was a really dumb a dumb mistake and you two I think I I
14:49
sent you a message being like hang on a minute have I done this right and I think both of you are like you've done
14:55
that wrong here's why uh so for those for those listening what what we had done was we'd recycled a box
15:04
um and forgotten obviously with a remote signing you get rid of your keys from that box um that you can use for your remote
15:10
signer when when the process re-initializes it obviously creates a new private vowel key for you
15:17
um now that vowel key will be the one that's used to create the Gen TX for a
15:23
network if you then run um a gen TX command with whatever key
15:29
you want to use from that box so we we use like just a a burner key for for
15:36
each test net we run um and we tend to not always but quite often either run it on we we've got a
15:42
couple of Linux boxes just knock it around for doing stuff on um but the uni uh this time we just ran
15:49
it on uh the the unibox we had which obviously the valky was there
15:56
um but then completely forgotten of course that it won't be the same valky as the private as the as the signer
16:04
um and so the Gen TX looked kind of weird and it was like hmm and then it was only about it was only the night
16:10
before where I kind of realized that something must be up sense check with these Two Fellas and they're like yeah
16:15
you must be the right wrong valky think about it um so deep right I gotta tell you the
16:20
only reason I knew is because I've done it before there you go so it's obvious once you realize it
16:27
though that obviously the the key matter of your the key matter of the signing key that submits the Gen TX or the
16:33
create validator command and the pro valve key obviously completely independent of one another the only
16:38
reason I had the realization was because we're doing stuff on uh on Aptos where again they have loads and loads of keys
16:44
but the fundamental concept is the same right which is that you have wallets over here and you have
16:50
validates private Keys over here and one attaches to the other but they're they're created a different process to
16:57
one another right they they are they are independent so well the reason why I thought mine would work right because
17:03
what I did was this was like a post-genesis creation and so I was like it was a test
17:11
net I was like I'm gonna like connect the signer first
17:16
and so I took a a key just the the random key
17:22
and I chucked it on the signer and I connected it to the node um because you can like have a remote
17:28
signer that will connect to the node just to boot it up and it doesn't sign any blocks it just you know you just get
17:34
no pre-commits all the way through but um I I thought that being connected to
17:41
tendermint that when you ran that tenement show um you know node I'd uh show validator
17:48
or whatever it is that it would actually funnel through the one from the like the
17:53
pub key from the remote sign up because that would make sense but it just picks up the local print
17:58
validator key that's in the local config diet so um
18:04
that's exactly the mistake I made as well because the the private listen address
18:09
is obviously getting The Pub Key what what sorry when you're
18:15
using TMK mass at least a threshold sign and probably not well
18:20
yeah right by the rough leader yeah exactly yeah so it will be the same Pub key so intuitively I was like well the
18:27
the yeah the pub key must be a memory right now uh
18:33
uh yeah yeah because the sign is up then it must be a memory but no it's over it
18:39
it's completely right off disk it's completely right off disk as far as I go yes so I think what we might be
18:46
you know dense node operator [ __ ] doing is
18:52
maybe it's like that the there's remote signers actually do sign from there
18:59
and injecting the network I don't know it's they do yeah they do and they're they're passing yeah they're passing
19:05
that back to the the note the note just turns into a relayer at that point yeah the signature yeah yeah so maybe that's
19:12
because maybe this doesn't send the key at all right so that's that's the conclusion isn't it is that
19:18
it's not sending the pub key because right right that also makes sense
19:23
because if you think about if you're using a remote signer not a threshold signer but a remote signer with a
19:29
Hardware security module then obviously it's going to sign against the Hardware security module on that node and then
19:36
just broadcast that's right and it's using the node as the broadcast method right it's like yeah exactly it's that's a broadcasting
19:42
it's almost just like having a private RPC that's on a different box if you like that's that's exactly what it is
19:47
right that's yeah so like even in the horcrux you've had two or three nodes running they're just nodes and people
19:53
call them validators they're not they're not validators they're just nodes right there's yeah burned keys on them and if
19:58
if the if the listener isn't working those nodes won't start right like they will they will halt um because they
20:05
don't there's they're not doing anything other than relaying and so if there's nothing on the relay side they will not start
20:11
exactly it's just like a private it's just a way of privately peering um an RPC that can't be connected to if
20:18
you like so the basic result is we're all dumb shits and didn't really know how it worked but now we do pick because of test days
20:24
but this is like this is kind of the joy I think of test Nets and this is also like why
20:30
I'm like it's not like everybody needs to be in every test net like like we
20:36
don't run on absolutely every test note on every chain we do um but I think it is
20:43
I I'm always a bit like dubious about validators that don't run on any test net like you should you should have a
20:50
Sandbox where you can learn these things and make these mistakes because we literally all do we've been doing this for 18 months or longer in some cases
20:56
and you still like learn things where you're like huh okay that's particularly with the states thing like I didn't
21:02
think that would be so catastrophic but evidently it was I know not only with
21:08
that but also running multiple nodes on test Nets because usually I think where people kind of [ __ ] the pooch that's a
21:13
weird thing well what we usually [ __ ] up is when
21:18
they're juggling multiple keys or they're juggling multiple nodes or there's a halt it's not just starting it
21:24
and letting it run like it's usually when something happens and so something happens on test Nets pretty often and uh
21:31
and when it where somebody you know like we just had a situation or not us but we you know we
21:36
know somebody got I don't want to steal the Thunder for later on the whole the new graphic how do I I don't want to
21:44
talk about that yet but somebody somebody had an issue because I think they're running out of drive space or
21:50
something and so while they were while they were going to get ready to prune one they started up they had another one
21:56
already running as a backup and so they started them both at the same time and they moved the keys across things like that like that that's just the
22:02
fundamental not understanding of what's happening there uh but that's also just like the way why not be down
22:08
yeah yeah 13 hours you've got 13 hours to play with I know like just be down
22:15
for an hour like it's not a big deal like the prune process doesn't take that I mean we just we just build a new box
22:21
and even on AWS like building a box throw away the old one yeah uh yeah
22:27
that's the way of doing it or you know Paul could choose snapshots again always shout out to the main man the poker
22:33
massive
22:39
I was just thinking do you remember that do you remember in Home Alone when when
22:45
uh the mother's trying to get back to the house and then John Candy's there
22:50
yeah like the Midwest poker bad and he's like you must have heard a song
22:59
foreign
23:25
and then you can be back up yeah no I agree even that or like we stay safe we State sick nodes all the
23:31
time now even if that takes a while I mean there's there's a lot of ways I think there there is a there is a um
23:39
I I don't have an issue with with like boards or other types of tools that that
23:45
look long-term around Miss blocks like I think I'm overall okay with that like that I
23:51
there's a place for that as well I I do think that that some for there is a there is a um
23:57
for some people think that if if it's you know if you have that much Miss then and it means it's a bad operator and
24:02
that doesn't necessarily that it doesn't necessarily mean there's blocks of there's blocks of
24:07
misses and then there's you just [ __ ] missing half of the [ __ ] all the time like there's yeah yeah you know there's
24:14
different different kinds of missing but um cause less harmier says don't you get a gold star for 100 uptime I think so
24:25
that's how it works isn't it I'm still waiting no I operate yeah yeah well that's probably that's I mean that's one
24:30
of the benefits of offer note signing I mean that's really what that gives you is it allows you to separate the node
24:35
from the sign and so you can you could increase your signing percentage much higher yeah well that's that's where
24:41
we're at now it's like we've been using tmkms for like last six months and that's cool it works fine works really
24:49
well um but I think we we've been talking here for a while about like running
24:55
horcrux but the there's a lot of different things about I guess like the CPU consumption and
25:02
like how to I guess like essentially what it comes down to is I don't think tendermen is a
25:10
complicated enough piece of software like keeping it up and signing and consensus is complicated all those
25:16
things that are more like an application Level but like the fundamental core of the stack
25:21
it feels like having a raft of signers is
25:27
working on a threat model we don't yet face I don't think it's actually threat it's
25:34
it's I mean you at AWS I think you have a lesser chance than maybe others but if
25:40
if whatever node you're using to run TM tmkms or if it's a single or Crux node
25:45
or whatever else if that's unavailable you're not signing right so that that would be the that for and two three
25:57
power cable out so I I don't agree with what
26:02
well I mean yeah so what are you back yeah I'm back what the hell
26:08
so what usurper was saying right um you know he's right if you're if your
26:14
signer goes down then you're not signing but like for me I operate everything in business on a
26:21
risk management basis right so risk reward and complexity and and stuff like
26:27
that so um I know you you were dying to mention it The Fray uh I run what I call d-gen
26:34
horcrux which is where I don't use so I like to
26:40
use the benefit of horcrux in that you can connect to multiple nodes and
26:45
I for me like I don't see the risk reward
26:50
of using um a more complex system like the raft signer so
26:58
um in my opinion if something happens on my signer or my
27:04
signers um or the sign up for the network and say it gets cut off and I start missing
27:10
on a network or multiple networks or whatever like I've got it on
27:16
pretty high nine servers and the likelihood of have been down for a long
27:21
time is pretty low and like we say it's okay to miss a block of signatures right so there's other
27:28
people that'll pick up the slack you don't have to have 100 uptime and as
27:33
long as you have a backup plan um in a you know a documented Playbook
27:41
of how you're going to handle the situation I think it's fine so I have like
27:47
a backup of my horcrux cyanurai
27:53
and so all I have to do if my main signer goes down I just spin up the
28:00
other one on a different place on a different service provider
28:05
and then I just change all of my firewall settings to block the other
28:11
signer and allow the new signer and then it just continues so
28:17
I think like downtime can be pretty well mitigated at a pretty low risk and I
28:22
think for the additional complexity of running a raft at this stage it's just
28:28
not really worth it so the result is that I can run two or three centuries
28:34
off a um a single horcrux signer and I'm able to
28:41
get really good uptime because I don't have all the latency issues of talking to the raft
28:46
um so as long as there's at least one of my um uh
28:51
centuries that are in the same region um I get really good up time everywhere
28:57
now so even like on secret and evmos which most people are in the hundreds or
29:03
thousands on an ongoing basis of Misses I'm in the you know teens most of the
29:08
time so yeah I think it's a good compromise
29:14
the camera is also I mean the complexity of the like we run two or three not an old chainsaw just because we started and
29:21
we had what we started to run into is was some of the challenges of running your own Hardware stack and your own
29:27
data center which is that Hardware gets older but now we're buying more so it's kind of changing now again but that two
29:33
of three um the only issue I have with it is that just the complexity like it just has the
29:38
it has the um once you set it up and you're you know we made that changeover and
29:44
everything else it's running which is fine and what happens with me is I have decent documentation on it but when I
29:49
need to go back into something that I haven't looked at in two or three months like if there's a issue or something else I start to get freeze right like
29:55
you have to go relearn what you've learned in the past and like it's not and it's a complicated structure in
30:01
terms of what's going on there in some cases in the in the config files and everything else so even if it's well
30:06
documented it's still that is approachable it's like a you know a single either signing on the Node or you
30:11
know a single single horcrux node that's that you know dealing with thumb dealing with what which boxes one two and three
30:17
and I have you know running a bunch of those on these different boxes and so that that can get a little bit complicated quick which is complication
30:23
this type of stuff is never a good thing yeah I think that's the kind of that's the thing that always sort of again like
30:30
whenever I have to go about something that I haven't done a lot of maintenance on for like let's say a week or a month or something
30:36
like that you're always just like yeah you're like let's say looking at file the firewall rule set you're like
30:43
[ __ ] I mean I've written down what all these things are but what if the day
30:48
because it's like it's that thing of like you're like it's only as good as a documentation what if I mixed up those
30:54
two yep right yeah how like I can the only way the only way I the only way I
30:59
can know that I didn't screw it up now is by manually going and checking all that source and then suddenly like a
31:05
five ten minute job is like an hour two hour job like going through everything and double checking your work and like
31:11
and I mean I'm paranoid so so I do do that and then something that should have been quick takes up like half a day but
31:18
that's again like that's but it's back to like even if my node was down I would
31:24
check because it's just it's like like the risk of double assigning tomb standings
31:30
forever you know um or it should be but um not really
31:35
right some people can come back from the dead like that same day
31:41
it's fine do you remember that Meme from a few years ago where it's like there's like a truck driving down the motorway
31:48
and somebody's somebody basically is like what's going on over there and like
31:53
there's like a single coffin on the truck and the the the coffin lead bangs open and a guy
31:59
just gets out and starts doing the Michael Jackson Thriller dance yeah yeah yeah that that is some people are able to do
32:06
that when they get tombstoned and I fear we would not be able to so
32:12
yeah like just missing blocks it's not a big deal miss an hour missed two hours you know whatever
32:18
um but then I guess most of the major like disasters we've had to deal with have been at chain level anyway when the
32:24
entire Chain's down when the entire Chain's down doesn't matter if you're not signing does it really doesn't matter no um
32:29
which Speaking of which is Yumi backup you you're on well no no no you're on
32:35
you me aren't you I am not on you me no I thought you were a new me
32:40
Knowing Me Knowing You me uh-huh [Music]
32:48
yeah so just sorry I was uh you probably heard all my Caravan noises um I was uh making a cup of coffee which
32:56
is a complex assignment in a caravan um so uh yeah I think it's the the only
33:02
thing that I find so just just segwaying back to the the horcrux thing
33:09
um so the only thing I find that is a risk is that you don't have a sharded
33:15
key so if someone wants to get onto your horcrux sign a single signer you've got
33:21
an Uncharted key there that um you don't they need to go find the other shards to reconstruct but in saying that
33:29
um you know there is no firewall rules allowing any connections in
33:37
to my horcrux so it's um that anyone's going to be able to go
33:44
find the other node to get the the wire guard VPN and then
33:51
be able to like know the path to the horcrux signer because you don't need to
33:56
have incoming connections on on Warcraft you only need to have it so
34:02
there's nothing open unless there was some serious vulnerability in the service provider
34:09
it's very unlikely that anyone's ever going to be able to get in there so I mean what is the risk profile it's
34:16
pretty [ __ ] low and even if somebody did get the validated private key
34:22
funds are safe through anyway so yeah
34:28
I still don't understand so much [ __ ] effort just to double sign you especially as it turns out the
34:35
reputational damage from a double sign is [ __ ] cool so that's a good segue into double
34:41
signing that people will
34:47
suffer a double sign on evemos of 10 and then [ __ ] red delegate to the same
34:53
operator it just boggles the mind okay twice we've got a new segment we've got
35:01
Tombstone of the week so we're into Tombstone of the week foreign
35:12
that took me 25 minutes to make because I had to watch loads and loads of loads of traffic fail videos on YouTube that I
35:19
found a truck getting tipped over uh so Alex I suggest that in the future you'll
35:25
keep that rolling and you actually put a little face on the on the truck that is
35:31
like the validator they've got Tombstone so we can go Tombstone of the week
35:37
okay okay I'll take requests like we can we can we can work it out
35:43
um yeah I guess most validates have a logo we could make this even more interactive like if we're going to take escalate the uh escalate the ship
35:50
posting to the next level um so yeah so who was it so who was it they got that they got Tombstone this
35:57
time so space this way yeah space boy IO so it's interesting because they have
36:04
like all this other stuff around right like they've got their the app stuff that they're building and um
36:12
yeah it's just it's so you said but we were you hanging
36:17
around during the um in the in that chat or if you go on and reviewed the chat from that period
36:22
uh when they're trying to figure out that they got Tombstone it just it it was after upgrade right I think so so
36:30
what happened I think was when uh let me go back and get the right versions but two upgrades ago uh there was an issue
36:37
with the um with the states think I'm sorry with pruning um with the version for some
36:43
reason it didn't show up in test net nevos ended up showing up in mainnet and nodes would not be able to stay on chain
36:50
um if they had printing enabled so this was I think when the 810 upgrade which was back on the 15th so a couple weeks
36:57
ago everybody went to prune nothing and so prune nothing and kind of it like you get you know drives actually grow pretty
37:04
quick right um and so I think for folks who had 100 you know depending on how much size they
37:10
had um it was it was eating up a decent amount of disk space a day and so when
37:15
this went through this was when was this I forget when this oh so this was off
37:21
chain upgrade it's not a proposal right so there was a version that was updated to add the block height in there and it
37:27
also had kind of Cosmos Cosmo visor indications within that binary then it
37:33
went to 820 that allowed pruning and so I think what my story is on this one is that based on what I saw in chat was the
37:40
the thing was out of space they had a separate a separate node that was going this thing was obvious the thing was
37:46
obvious the thing was obvious the thing was obvious the thing was obvious the thing was obvious the thing was obvious the thing was obvious the thing was obvious the things the thing was obvious
37:53
the thing was obvious the things so
38:00
how do we explain this all right
38:08
they can stop they could stop this is Sensational I'm on a phone how does it not happen to me
38:14
anyway so they brought up a second node I think they were moving Keys over um whether it was past upgrade they they
38:21
put the validator key on the second node and at some point within there they were both running at the same time and they
38:26
got boom Tombstone so that was it I think they didn't understand I think there was
38:32
some confusion at least what I saw on channel there's some confusion exactly how that worked and and what tombstone mean and there's a lot of that you know
38:38
how do I unjail myself which is always like the first question right hey how do I in jail myself and then somebody goes
38:43
and look and say well I guess be fair if you missed the log line where it says evidence of what what it says
38:50
yeah and that it does from memory because I've seen a double sign I've double signed myself on a test net
38:57
before that um right probably the same one you did as well right the estati one where we back in the day we killed the
39:05
network there was like a rogue smart contract test or something like that Network and then Todd was like should we
39:13
try seeing what happens if we will delete our state file and the answer was you double side right
39:19
right and yeah I think I think a bunch of us did right yeah yeah it creates so you won't want it when it cycles and
39:26
then you all double sign is what happens yeah and it says in all capitals something like evidence of double-sided
39:33
detected exclamation mark exclamation mark exclamation mark that's it and then when you're right the log goes like it
39:38
just keeps going past right yeah and then it just starts dialing again doesn't it yeah yeah it looks like it's dialing I mean well it is dialing I
39:45
think um it's just converted itself it's like the quickest way ever to convert your
39:50
violator into an RPC right correct that's right that's right I'm sorry uh
39:56
so unless you've got that Porto but it's not really benefiting anybody that is so I think that team did not understand the
40:03
structure of it which which you know I get this is complicated stuff but whatever um well then it was clear on this a
40:09
clear misunderstanding of yes where the first start where the
40:14
pre-validator key was so that could have been catastrophic in itself yes and also
40:20
um you know even a basic understanding of tombstoning and double signing so
40:26
yeah I mean they they really could have benefited from some time in in some um test Nets and breaking stuff yeah yeah
40:35
um so if anyone's interested uh in sorry about my uh I could still hear you guys just fine but
40:42
um yeah you said it was just an infinite Loop coming out I have a clock right and when I'm using
40:50
my dock it will just at some point freeze everything so all of the like the
40:57
monitor will still work but all of the peripherals were just like stop but hold their state somehow and
41:04
then it's just I have to unplug it and plug it back in for it to work again yeah we'll do cause cause less harm has
41:11
said uh can we get a remix we'll do it we'll do we'll do a jungle remix what just happened
41:17
uh when we get when we when we pull the audio off for the podcast
41:22
or yes yeah yeah yeah it was just a kind of locked a locked three second Loop or
41:29
one second Loop or something like that uh sorry for all of you listening on
41:34
podcast players it's part of the joy it's part of the
41:40
joy um so ding ding ding double sign double
41:45
sign of the week uh one one day inshallah it will never be one of us but
41:51
you know you've maybe one day we will be we will be saying goodbye
41:57
to one of the fellas floating off down the night wait no it's not Nile is it what which which which one is it where
42:03
they burn the boat and then is that the Vikings where they burn the boat and then they yeah or whatever
42:10
well I guess it's going to is it going to Valhalla like how's that work okay like if you know about Viking folklore
42:15
and you know to what I'm referring and you know what the thing actually is and how it works
42:20
put in the chat um they're they're like happy places Valhalla yeah I think you burn you burn
42:28
you put the boat in a put the body in a boat and then burn it and then they'll Hollow for [ __ ] this up
42:38
[Laughter] none of the Ganges you you get burnt
42:43
first sorry so Ben Davis has said in the chat are you mixing the Vikings and the river Ganges I was thinking Ganges when
42:50
you said it I was thinking no you get burnt first you get cremated and then you get scared whereas like the the I
42:57
think the Longboat thing is like you you they you get pushed out on fire yeah yeah yeah
43:04
um there's do you remember do you know who the actor David Tennant is he was Doctor Who for a while
43:09
um he's been in a bunch of other stuff he was in the um uh do you remember good oh good Omens
43:17
was it called good Omens it's like a new there's a Neil Diamond book there's like an Amazon Prime thing of it a few years
43:23
ago I think it's called good Omens he was in that was he in that maybe he was in that
43:28
I can't remember was was he burned and dumped in the Ganges no but he is in a he was in an
43:34
independent film a few years ago where oh it was somebody really famous played
43:40
the grandfather it wasn't like John Cleese but it was somebody like that played the grandfather and he just
43:46
quietly dies during the course of the film but while he's taken the kids to
43:51
the beach and these little kids without the adults are just like what do we do with this dead body and they're like
43:56
four and five years old and so they put him in his little row boat and they [ __ ] set fire to him to give him like
44:03
a viking funeral they push him off into this lock yeah and then because because he's
44:09
because he had said to the kids like don't tell your parents but how I wanted to be like it was just like a running
44:15
joke that he had told the kids several times like when I went on debt just like burned me and pushed me out to the lock
44:20
like and they're just like aha Grandad and then they actually do it and then obviously the adults arrive just as this
44:25
boat like like you add this stop fire and they're like oh my God have you made it Granddad
44:31
they're like no it's okay he was already dead and they're very dead like what um I can't remember what I think it was
44:38
like what did you the film was called something like what did you do last weekend or so it was some like where and
44:43
it also had that um Rosamund Pike in it who's been in loads of stuff since and yeah
44:49
it's a very weird film anyway but quite good quite Bleak you know dark dark comedy there's only like one or two very
44:56
funny moments and obviously I've just ruined one of them because you're like oh yeah the kids are really gonna burn that
45:03
I'm gonna burn their ground that is quite funny but also black do we want to talk about the cosmos now or
45:09
why so here so let's so let's talk about let's let us talk about liquid staking
45:17
how about that so there's a couple of liquid staking operators that have um popped up in a couple of
45:24
people liquid staking operators that want to pop up so there's currently uh
45:31
yeah uh Commonwealth um thread for uh staffy staffy
45:39
so that's terribly close to space fight which I was like getting my wires crossed with a
45:45
little while ago but um anyway so there's so maybe someone can tell me who there is
45:51
operating because I know that there is stake easy and I know that space fire wants to come on so these are both
45:58
um uh like contract Solutions right and then
46:04
there was another contract-based liquid staking protocol
46:09
I'm sure and then a couple of uh chain based on so can someone give me
46:15
the the list the full list no
46:21
strides Quicksilver uh steak easy
46:26
X Juno external right yeah the one that was like um raw dower
46:33
looking to acquire that's actually you know right yep so she's riding Quicksilver uh the chain based ones bro
46:48
we're going to validate Quicksilver I think just because we're trying to we'll see just because
46:55
obviously like we have to do the opposite thing from Rhino's business plan
47:01
do the opposite it'll be better so I mean to me they both seem like something they can just be done with the contract
47:07
so I just didn't join either well that's that's the thing so
47:14
the the staffy Hub staffy staffy starfi
47:21
um I guess they have a Comic Sans logo so maybe it's supposed to be starfi
47:27
as in Star When You Wish Upon A comics and star they do not have a comic sense
47:33
oh yeah they do no way yes what's the Euro
47:39
um I don't know they screwed it up on their Commonwealth proposal so
47:46
um but yeah it's I I screen grabbed them put on Twitter it's definitely Comics ads
47:53
but uh but yeah they they they asked for two percent of all Juno
47:59
um to be used as uh liquidity incentives so how exactly what liquidity incentive
48:05
so they just want the community pool to hand over two percent of all the Juno yeah for for their for the Arduino token
48:13
I guess and Juno like through liquidity or liquidity incentives because that I
48:18
mean it's [ __ ] stupid I think it's I think it's done to well actually I think it was liquidity pool actually
48:25
I mean that's still stupid I mean it's almost two percent it's like what wherever that sign me up
48:32
bro meme has got to just that sign me up bro
48:38
um the funny thing about building liquidity is right that so these dexes are a little bit funny
48:45
economically in that the more liquidity you have in a DEX right the harder it is to shift the
48:52
price so if you have a lot of liquidity in a pool and the price is higher then you've got
48:58
a lot of exit liquidity right for when the price is going down but if
49:04
you have if you stuff a whole heap of liquidity into pools when the price is low then it makes it harder to push the
49:10
price up you need a lot more um incoming money and a lot more volume on that pool on
49:17
and buy pressure to actually be able to push it up than if it had low liquidity so you kind of hurt the ability for the
49:24
price to rise in the long term without a lot of incoming Capital but you protect it
49:32
somewhat from Sharp drops so I wonder if people realize that
49:38
building liquidity right now could make life a little bit difficult to in the future to be able to bring the
49:45
price up well because it privileges inherent stability at the current price level
49:51
yeah because you've got a lot more liquidity in the pool so you know that
49:57
that X Plus y equals zero becomes harder to move because of math and [ __ ]
50:03
yeah I mean you've already got two variables in there so I'm I'm kind of lost because I only did math till like
50:09
age 14 or whatever so yeah never never used a variable in programming
50:15
never heard of it your honor uh no no
50:21
um obviously I do understand what variable is but uh I only use constants because mutability is bad
50:27
um yeah that's the thing that like Niche joke I don't know constant
50:33
uh so yeah but well it's not like it's not like we're limited on memory these days is it it's not like we've got like
50:38
only two kilobytes of memory to play with so oh gotta make everything mutable so that we can you know optimize our
50:44
memory it's like I think we've got we've got enough memory at this point Lads that's it's memory is not your problem
50:52
um early versions of uh evmos would beg to differ
50:58
well that's good well but then is all the tenement trains are basically single threaded right that's the problem
51:10
would have considered like is that something that you think people would consider okay
51:15
no the the like increasing liquidity at a low point
51:21
in the market do you think people don't give a [ __ ] they're just like more liquidity
51:29
well so we're increasing liquidity in what sense you mean because because that
51:34
this is liquidity going specifically to like a pool and attacks it's not like just generic liquidity although I guess
51:41
the state well are you saying the liquid staking itself isn't increasing liquidity I suppose it is isn't it even
51:47
as a derivative no sorry I kind of drifted off the the liquid staking I was
51:52
big just because of their liquidity incentives um and you know there seems to be like a
51:58
drive all the time to increase liquidity in um dexes so like at the moment there's
52:06
um talk about uh what is that what is that Decks that they have there that the
52:12
I can't remember do you remember a circle the one that um held all the the
52:17
coins across that Bloody um bridge that got wrecked it was so mad
52:23
well that was the bridge but what's the um hang on so the text is diffusion
52:32
diffusions um got a proposal in um Commonwealth I think to
52:38
uh get a whole heap of the community pool to increase um you know incentives to try and bring
52:45
some liquidity back to diffusion um and there's you know already
52:50
incentives on osmosis and it always seems to be that you know chains well
52:56
people are always pushing for incentives to increase the liquidity in these dexes
53:01
but do you think they consider the economic environment when they're pushing for increases in
53:08
in uh you know liquidity or is it just like you just want more liquidity no matter
53:13
what right honestly I think the majority of the time the liquidity is to allow the
53:20
um allow the founders to exit no like let's let's be real
53:28
um I think somebody on Twitter made this joke where they were like I I know what this is and they literally linked our
53:33
first episode and it was just I think the point where I think shortly said something like well
53:40
you know the thing is all this is just liquidity for the founders to exit or something and I was like
53:45
it's already you see shoes he's already said it there you go someone needs to start like a
53:51
game of nodes shorts chat I think it'll get a lot more views than like our Channel well just because
53:57
like our single uh a single sentence [ __ ] posts are actually yeah dumb [ __ ] we say on here be like oh this
54:05
thing and then you can just take that completely out of context and make everyone mad on Twitter it's like that
54:10
uh the line from spinal tap right where he's like it's a fine line between stupid and
54:16
clever well that's just it he just he just pinwheels doesn't he and then somebody
54:22
has to say clever and he's like yeah clever all right that's it yeah
54:30
um what was that on spinal attack
54:36
I don't want you to maybe you're not getting a [ __ ] on it
54:45
I mean if you're as badly as you did to heat then maybe you shouldn't watch it because it is it is probably the
54:53
funniest film I've one of the funniest films ever made and I'll be really sad Ben Davis says liquidity going up is
55:00
great if you assume that tvl is the best way to measure a deck's success
55:07
I mean well how do you measure success I think
55:12
the odors the the founders measure success in terms of exit right
55:18
um that bit seems to be very similar to when I worked in startups
55:25
um but it is I mean the problem is it is always like we are it is another
55:30
reminder though that like [ __ ] posting aside we are early enough that we don't even
55:35
have a good I I would say a good agreed set of metrics for what
55:41
defines a good project and a bad project right I mean that's why like I you know people like Rama and stuff who go and do
55:49
doodle on stuff and try and come up with a set of metrics whether a project is like good or bad or whatever
55:55
look they have to do a lot of leg work and then quite a lot of context setting to even like
56:02
explain their findings in such a way that you can get a grasp of the earliest
56:08
like you know [ __ ] idiots like me can get a grasp of like what's actually being said because you know a lot of the metrics are quite
56:15
meaningless in the abstract I think because they're still it's all back to a lot of what we've talked about the past
56:20
where it's a lot of it's based on kind of I guess like what sort of economic
56:27
utility but like in a very spot fashion like quite a short-termist like it's a
56:32
slice of it's a slice of time it doesn't really have any it doesn't have a lot of bearing on like where that will be
56:43
doing doing due diligence is a portal to making Maxi boys angry
56:50
[Laughter] you want engagement through some due diligence
56:55
yeah I mean how many how many I I'm sure Rama's got more Twitter followers than
57:01
this podcast so that's yeah we need to do deal segment where we just uh we just absolutely
57:07
Sledge some projects that would that would help
57:13
during the week it'll be like going to do deal yeah destroyed
57:20
yeah put it on like the boring hat from previous jobs and actually doing yeah one of those green translucent
57:27
brim things oh yeah like in a like an accountant's hat yeah
57:33
100 um
57:39
the space flight validator are you still reading the conversation back well I was
57:44
actually I was going back I wanted to see I wanted to see if the Airbus Foundation was a delegator into that because they I think they had like four
57:50
or five hundred thousand utmost which would be somewhere around uh they're probably in the mid 40s
57:58
schultzy's got 459. we're number 105 on this 104 in this list with 157 right now
58:04
they're at 107 with 153 000 after after
58:10
they restarted the new validator since they were slashed so they they ended up
58:15
I think that slash was about 40 or 50 000 ofmos um and they came back so that they
58:21
wanted to be able to repay that 10 percent a month over 10 months I thought is what they
58:27
said I I don't know I mean it depends if they get back up in the high enough in the set to do that but
58:35
um yeah I mean uh 4 000 of most a month is
58:40
is a pretty decent chunk to be able to pull out of commissions to uh to try to
58:45
pay that back so we'll see if that if that's yeah but I mean yeah these are just very they're
58:52
also very big numbers they are yeah I mean it could have been a lot larger too actually
58:58
yeah So like um this is we're definitely into idiot
59:03
question time but the double sign penalty on fmos is pretty high huh it's
59:09
10 it's 10 I think it kind of matches everybody else I thought mice change is like five percent
59:15
oh is it I don't know I thought Juno was 10 as well was it uh I mean I should
59:20
[ __ ] know given the number of times that I've read and reread and rewrote Rewritten parts of that Genesis file but
59:26
I can't actually tell you off the top of my head oh yeah Juno's five percent I think that's one of the lower ones though I
59:32
thought I felt like it was five percent but I didn't want to 100 commit to it in case it turned out to be like 20 and
59:39
they'd be like you're just way off Fray what the [ __ ] um but yeah um as a complete aside I
59:45
looked up what that film was it was called what we did on our holiday it's from 2014
59:50
British small budget black comedy and it was been it was Billy Connolly oh okay he plays the the grandfather and
59:59
uh and yeah his his grandchildren go with his final wish as expressed to them
1:00:05
and they cremate him Viking style much to the horror of the entire extended family
1:00:11
it's a very dark like there's a lot of other stuff going on in the film in the
1:00:16
background as well like it's very like like it when it's funny it's very funny
1:00:21
but it's also a very sad film um it's one of those anyway um
1:00:27
most of these are five percent oh it's like it was always ten percent in my head but you're right most of these are five yeah I felt I felt like that their
1:00:34
slash was that was a bit higher than than the norm but even though that's like it I mean it's kind of like there is a
1:00:41
question as well at that point where you go like should given the complexity and like also the
1:00:46
continuing risk involved with getting back into that and then trying to repay that yeah well this should they even
1:00:54
bother like should they just say instead like we'll just step away we [ __ ] up sorry Fun's not safu
1:01:02
you know yeah this kind of goes back to my My
1:01:07
overall challenge with distributed proof mistake and Cosmos like this
1:01:14
this is there's a lot of different projects that are coming in that are running validators that that like in
1:01:19
this case where we saw that before with stick like model and there's been some other ones in there where where not saying they can't run it I'm just
1:01:26
saying that there's a there's a pretty high learning curve in some of these areas in terms of being safe and it's not just double signing but there's also
1:01:33
security uh there's there's other types of things that go into running an infrastructure security structure around something like
1:01:40
this and so I understand the idea of you want to be able to bring projects into ecosystems and you want to find ways as
1:01:47
project owners or delegations or as foundations to be able to incentivize those projects to come in right like nft
1:01:53
projects or anything else like you want you want to find something to do that is this the way is there a better way to be
1:01:59
able to incentivize somebody like that to bring projects in than just running
1:02:04
validators and having you know a 10 slash right so like there's like it
1:02:10
seems I don't know why like like this is my issue is that like it's not their role like I understand that there
1:02:16
there's there's other extra money to be made there right so there's that's that's kind of the rubbing here is that they can find ways to be able to nft
1:02:23
drops based off that or something else that gives them extra Revenue coming in but but there's risk to this too and
1:02:28
it's not made I don't know it seems like it seems like it's a Mis a miss here in
1:02:34
terms of um what these sets are going to look like in a couple years yeah it was I
1:02:39
guess it's coming back to that Moultrie uh the point Moultrie made a few episodes ago where I like I think we've
1:02:46
been very like you should run your own infra if you're a validity you shouldn't
1:02:51
really be wild I mean I understand people do white label I'm not a big fan of it to me it's like does seem like
1:02:58
it's is putting your eggs in one basket as a network and it's sort of a low-key Sybil
1:03:04
and also it's the concentration of power in the hands of people who know how to run nodes which is like the opposite of
1:03:09
decentralization but then you kind of go well if people aren't
1:03:14
able to run nodes then what are you going to do and then you go well okay so
1:03:22
I mean maybe that does come back again to like the thing that we've been talking about before which is like if it is it all just decentralization theater
1:03:29
and should we just not worried like why labels are fine yeah because actually the fundamental problem is that we are
1:03:35
in a proof of stake system which privileges um fast finality over liveness so we
1:03:41
have a small vowel site and maybe maybe what we're all doing here is kind of collectively going well
1:03:48
the income model where you just spin up a validator state to yourself and then Robert's your father's brother is is all
1:03:55
very well but only if you have like a much larger vowel set and maybe no slashing
1:04:01
so that you can accrue those rewards but not at the not with any risk to the network and not with potentially even
1:04:08
any rewards to delegates because not all networks not all proof of stake models have slashing
1:04:14
uh right I have opinions
1:04:20
please does anyone want to hear them please I
1:04:25
I recently have like thought on it a bit and I don't think it really matters I think
1:04:33
if people want a white label they should white label at the end of the day
1:04:38
I'd prefer it if like say for example there was an influencer who was you know
1:04:45
got a community and they want to get some income from
1:04:50
validating whether it be a small amount or a shitload there's different influences out there at different levels
1:04:55
you get different levels of income from their validators right personally I would prefer if they teamed
1:05:02
with somebody to and created a team with like another devops guy to be out
1:05:10
but stake like Mo literally did that yeah yeah yeah so hear me yeah the
1:05:16
person white labeling for them double site like and there was one of the other double signs was a big validator team
1:05:22
white labeling they took down two notes like which network was that a few weeks ago they took down two nodes
1:05:35
it was blocked demon and they were running their node and somebody else's so it was white label again
1:05:40
and then you know not naming any names but also there are quite a few other influencers that I can think of in there
1:05:47
they have other people running their notes for them yeah yeah
1:05:52
it's it's not it's not so there's different things you're talking about there right you're
1:05:59
talking about white labeling and you're also talking about them having their own
1:06:04
devops team right so if you take stake cedo and friends they
1:06:10
teamed up with devops people they don't white label if you take steak like Mo he
1:06:17
white labeled with someone who is PR has nodes for a bunch of other people and that's the same with like block Damon or
1:06:24
whatever they they were like in a group of white labeled nodes right in their operator
1:06:31
um double sign them all so there's different things we're talking about that I would prefer that you know
1:06:39
if an influencer wanted to starter node to get to generate revenue
1:06:44
from it that they would do it with their own devops team they like you know find some people and they team up and then
1:06:50
that's what they do but you know ultimately if there's a trend of
1:06:57
um influencers getting uh you know Tombstone then people will eventually
1:07:05
notice and people will just stop delegating to influence it'll just be like a
1:07:11
self-regulating thing so that's the risk people take and if they want to delegate
1:07:18
to their influencer because they like them and that's their prerogative I
1:07:24
would prefer that people delegate to about career validators because we know
1:07:30
the [ __ ] inside out and we are very good at it without like having a big head but
1:07:36
we spend a lot of time in test Nets we spend a lot of time breaking [ __ ] we spend a lot of time thinking about you
1:07:43
know risk um and how we set up our systems so you know that's the
1:07:49
difference someone else who's just a devops getting a weekly pay from their influencer pal probably
1:07:57
doesn't spend that amount of time in there and it's evident when it comes down to the result right
1:08:02
so if whether it be like at the same time
1:08:08
like whether it be an influencer wants to run a node whether it be like you
1:08:13
know Commonwealth Bank of Australia wants to run a node give it a go it's it's like a
1:08:18
competitive landscape right so right but I'm going to put it out there that like I I think I think your point about like
1:08:25
people not having skin in the game being maybe a bit more likely to make a mistake or not having like loads of skin in the
1:08:32
game because you know being paid to run a node whatever whatever I can I can probably buy that like it
1:08:37
it's very different when you are representing the company that is running the nodes and you really
1:08:44
you don't want to be made a fool of in public and you want to provide you know the standard of service and also you
1:08:51
know you've got your own self-stake you know that's I don't know about how it works in Australia but
1:08:57
if we get ourselves slashed we're still going to pay tax on the full amount before we are slashed
1:09:03
so that's a really great way of going into loss for the year um so if you were running like your node
1:09:10
as a individual you could just take that as a
1:09:15
loss against your income I imagine it should be the same there so it would come out in the wash but as a business
1:09:22
it just comes out in the wash anyway because we only take our totals at the end of the year so
1:09:28
right yeah I mean I think it would be a bit I think it would on the tax system
1:09:33
here depend on when you compounded but you'd obviously have to pay tax on it at the point of compounding so you could
1:09:40
potentially end up slashing yourself for them losing money like like losing a lot of money if the based on where the price
1:09:47
was but yeah I mean I think I think I think the thing I'm trying to get to is like I
1:09:53
think that like the market decides is like a fair enough it like it's an argument it's one of it's one of the arguments but I think the the thing I
1:10:00
guess I'm a little bit cautious I I feel I think we all feel you know joking sorry we all do feel some some sympathy
1:10:06
for node operators that are in that position because nobody's immune from a mistake right
1:10:11
that's the thing if I've learned one thing in the last 10 years like yeah
1:10:16
I've I I'll put my hand up and say I've pulled down a production system minutes before like a key demo to a government
1:10:22
before um you know sometimes these things happen and then when you do the five
1:10:28
whys of like how it happened you're like well in hindsight
1:10:35
in Hinds I mean like like uni uh uni4 like failing on the second block like in
1:10:40
hindsight is obvious like what happened and it's obvious that we you know should have
1:10:47
been more clear about like the configuration that we expected on the startup of the nose and stuff but you know we didn't know about that thing at
1:10:53
the time forgivable mistake whatever you know and a lot of these mistakes are forgivable
1:10:58
in hindsight but the damage is already done right um and that's the problem with running
1:11:03
Alpha software and production and and the thing is like you know like Magnuson
1:11:09
um and Barry were saying from skip the other day like if you you dig
1:11:14
around in the stack March and you start seeing to do's they're not big ones but
1:11:20
like you know it's got that sense of like you know we we still are only like a year and a bit into IBC right
1:11:27
um and sometimes we do also forget that a little bit because we are running this like a production system and it is a
1:11:36
significant part of the revenue of our companies if not like our full-time job right to run the stuff
1:11:42
so that's kind of wild given how alpha a lot of this stuff is right
1:11:49
um so it is sometimes I am just like well for all the through all the test Nets
1:11:54
you run and whatever like you only got to make one mistake you you only signed twice right that was the the episode
1:11:59
name we had a few weeks ago yeah so it is very like there but for the
1:12:05
greater God go out you know what I mean yeah and and even like I'm but I'm still
1:12:11
going back to the main point which I I agree with what Noel said but the the Forgiveness uh delegation side around
1:12:17
those types of mistakes doesn't you might as well make the slashing thing zero like if nobody cares
1:12:23
then there is that that free that that idea that that Noel was saying around like like those validators will end up
1:12:32
going away over time or over the long period of time which which that's proven not to be true right like I think some
1:12:38
of these some of these have actually come back stronger after um than they had before the slash which
1:12:45
is just or before the tombstone which is like crazy to me maybe someone should make like uh an an ongoing chart and
1:12:53
categorize validators as like you know corporations
1:12:59
um career validators influencer validators and have like a tombstone count
1:13:05
oh yeah I do like block do you like block pains whatever it was the yeah the taxes
1:13:12
validators and then just put put who's yeah I mean but then the thing is I
1:13:17
think you'd I I don't know I I bet you would find
1:13:23
the the ones who are small businesses and all Engineers would probably have the lowest Tombstone count and they
1:13:30
would also have at this point the highest blood pressure of their entire lives
1:13:35
that is I don't think you need to have high blood pressure like it's
1:13:41
for me it's I don't know it's pretty cruisy for me as long as I'm on top of upgrades
1:13:47
then it's my infrastructure I have very little problems with it um I can spin up and spin down nodes
1:13:55
with ease like my my I know that my signing strategy is solid I've been
1:14:01
through lots of [ __ ] up test Nets and always been signing at the head it's I
1:14:06
think that the setup I'm using is solid enough that I don't have to have high blood pressure
1:14:14
and you know occasionally there's a [ __ ] up upgrade but
1:14:19
very like I've never it's you know I'm not in the percentage that's the cause of it
1:14:25
because of the way we're set up and we've always got multiple nodes and um if it's like a high risk in my opinion
1:14:33
um upgrade I'll actually just not upgrade one I'll start the Daemon manually so it just
1:14:39
goes off when the upgrade height and then have another one like automatically upgrade with Cosmo visor and if there's
1:14:45
a [ __ ] up then I've got like a clean database I think like
1:14:50
you can mitigate your risk to like bring your your blood pressure down
1:14:55
um but yeah that's fair enough I I would like to like have a look at some of the comments here so going to like Ben Davis
1:15:03
all the way back here and sort of jumping back a little bit I didn't understand it he said no context gamer
1:15:08
nodes is the new context turn I Googled her can anyone explain what
1:15:14
Hearn is because the Google definition I still couldn't [ __ ] understand it
1:15:19
the definition is past participle of here
1:15:25
so explain yourself Ben Davis I don't know
1:15:30
what the [ __ ] you're talking about man so coin Waters
1:15:36
uh was from from a chain perspective it's not fair to those that stay up all night and
1:15:43
spend time in uh us and spend and spend and full time maintain the change should
1:15:50
reward based on this look that'd be nice but you know in a free market economy like
1:15:57
there's just nothing we can do about that you can't force a decentralized thing to reward a
1:16:04
certain people and you know whilst you would like to recognize well what would you like to
1:16:11
have the the broader Community recognize the efforts that are put in by like certain groups
1:16:18
you can't they're just they're going to be influenced by whoever's screaming allowed us at them
1:16:24
um and whether that's from like lukewarm engagement or causing anger on mass or
1:16:32
you know being good at communicating your efforts you know there's all different ways and
1:16:39
the Market's just going to choose who they're going to choose and I think inherently like a lot of operators are
1:16:45
now forced to become influence of some description so we have a podcast whilst we don't
1:16:52
touch on news as much like we don't really have that the right reach for user base or reach but you know we we do
1:17:00
this we all have a Twitter account and we all try to engage on some level to
1:17:05
have people notice us to come and stake with us right that's we've been forced
1:17:10
into that and from a practical perspective I had deleted all my socials before
1:17:16
I had to recreate new social media accounts because of running validators
1:17:23
because people were like where are you and we were losing delegations like because of it and
1:17:30
yeah yeah it is it is a weird but then this is this is sort of back to
1:17:37
you know you can't it's very very hard to make incentives with algorithms that
1:17:44
actually work in the real world and like people have tried a lot in the past like
1:17:49
this is not a new area in economics we're just able to iterate on it incredibly fast in crypto for like the
1:17:56
first time you can literally design an incentive system deploy it change it a few days later to redeploy it that is
1:18:03
new but this is stuff that people have been playing around with for well I mean for thousands of years but
1:18:09
like with a with an academic basis behind it for like the last 40 or whatever you know it's yeah
1:18:15
it's pretty well established at this point so you go well you know I think what we saw with prop
1:18:21
16 and a lot of that stuff that was the first time it became really really clear that
1:18:26
um the incentive structure and what I've been put in place for governance for for
1:18:33
these dowels for like very for Cosmos Maybe
1:18:38
um had had grown scaled past the original design of that module of that
1:18:44
system right and and it no longer functioned um and what filled the Gap was social
1:18:50
consensus came in and just went do this thing it's completely like your
1:18:55
options are a b or c and we want you to do f right do that and it and and what
1:19:01
happened was everybody wanted well a majority wanted F and they actually got
1:19:08
l or something like some some completely fifth thing some sick thing whatever it
1:19:13
was an eventual compromise that was arrived at completely off chain
1:19:19
and so like now you look at slashing and like usurper says it's mad it's failed like slashing the hard slash has now
1:19:27
happened enough times that we've seen it doesn't it isn't significant enough slash
1:19:34
to actually affect um to actually do it do its original
1:19:41
intended effect right so it splashes without reimbursement and
1:19:46
people go back to the same [ __ ] operator like you would have to be yeah yeah so but like what's this is the
1:19:52
thing like you look at that and you go okay so the number so there are two things going on right one there are some
1:19:58
people that it could be a hundred percent and they would go back but that's not everybody the majority of
1:20:04
people are price elastic to some degree in terms of that penalty they've taken so number one you go well what is that
1:20:11
number and number two you go if that number is too high that the chances of
1:20:18
you know just penalizing the network or changing the way people think about the risk structure to the point being like
1:20:24
I'm not going to run it risk is too high if you bought it and if you had to buy in a hundred thousand hundred hundred
1:20:30
fifty thousand dollars would be different the slash was a hundred percent yeah as a even if you are a career
1:20:36
engineer you would look at that and you'd go that is high risk to me I'm not going to do that right so I'm gonna I'm
1:20:41
gonna train myself to make sure that I'm not in that position right yeah you're gonna run one network at most and you
1:20:47
know right now in a bear if you're running one network even with a decent self stake you're probably not making enough to to break even so you're back
1:20:53
to a situation where that economic structure is going to cause people who understand the risk I.E the node
1:21:00
operators who are thoughtful and experienced enough to recognize what they're seeing to not run so you're like
1:21:05
okay so what we've done is we've successfully showed this group of people that if you get slashed you should leave
1:21:11
and they will now do that but a side effect of that is that experienced and potentially desirable node operators
1:21:18
will also recognize the flip side of that and possibly not run so then you go okay well what's the Other Extreme and
1:21:24
the other extremists have no slashing and I I don't know I haven't really thought about that one enough to
1:21:29
step through what are potential consequence of that but I assume you know there will be another set of consequences which
1:21:36
so so this is the question right if if smashing is broken as an incentive mechanism then what can we replace it
1:21:42
with I don't know somebody's smart could chip in or or the percentage is
1:21:49
too low or something else yeah yeah I mean and then you have like the flip
1:21:56
side of that is osmosis and they just don't want a slashing at all well that's that's their downtime but do they have double sign slashing I would assume so I
1:22:03
can check real quick I assume it's probably five percent the the slashing percentage on
1:22:09
Atmos uh evmos is like the highest drive any other ten percentage
1:22:17
Yeah we were talking about that before when you were off for a second um yeah they're they're all defaulting
1:22:22
the five percent except for everyone since that time yeah but it but again that and we've seen more double signs on everyone's
1:22:29
than any other network so it's not it's not it's not and and I think to the
1:22:34
other piece is that those operators didn't even know that existed so not not only is it not an incentive
1:22:39
but there's no warning within maybe that maybe that's a tender warning on create vowel like maybe that
1:22:46
should be added as a as a before that create Val command comes to say when you
1:22:51
do you understand what this means like if this is a double sign then this is blah blah blah because I don't think even I don't think there's any the
1:22:58
barrier's so low to be able to run and the Very solo to be able to get in the set in some in some of these areas
1:23:04
um that I don't think people understand what that means Barry is only getting [ __ ] lower as
1:23:09
well like we're making it so easy even other validators are making it so easy for people who have no [ __ ] idea what
1:23:15
they're doing to run a validator and they can float under the radar for long enough to to gather a lot of
1:23:21
delegations as well so yeah I don't know I I think like you said
1:23:28
before it is a free market and it may be you know at five percent
1:23:33
um it's just not enough I think people get more that's part I mean people get pissed about that but the other piece is
1:23:38
like if they've just delegated in that they can't redelegate for the you know for the unbonding period with 14 or you
1:23:44
know 28 days or 21 days or whatever and sometimes they get more angry about that about missing on rewards on being bonded
1:23:51
to a jail validator uh than they do on the double sign right so I think getting been missing well I mean yeah so there
1:23:58
seems to be like more emphasis that they've they're missing out on rewards than what they are missing some of their
1:24:04
capital yeah exactly the the uh well I mean some some of them
1:24:10
refund it so maybe it's a quick way to unborn get slashed and get a reimbursement
1:24:15
there you go um so what what's Bonkers to me
1:24:21
reading their their Twitters right um so I saw in in the recent slashing that
1:24:27
um in their Twitter post people were saying in there that oh you
1:24:32
know thanks for the transparency like they went [ __ ] transparent they
1:24:37
didn't say that they were incompetent and that's why they double signed they made up some [ __ ] so yeah yeah
1:24:45
and then stay tuned earlier yesterday it's got jailed or something similar to that right but I think they're they were trying to relate it back to State like
1:24:50
Mo which I think was in a little bit of denial about it um well they said this thing so they said
1:24:57
the thing about about um you know having to to do pruning I think
1:25:03
was in there right but they didn't say that they didn't know where the private validator key was and they copied it
1:25:09
across to a new node and started them both at the same time like right there's nothing about that
1:25:15
right who's just like oh it's an accident whereas no it wasn't an accident it was that you didn't know
1:25:21
what you were doing yes it's an it was an avoidable accident for sure absolutely and you know but
1:25:29
people are just Bamboozled by [ __ ] so yeah and like I said we've started with before like there's a there's a
1:25:35
really high bar for um you'll you know you'll get it next time and you know we all make mistakes
1:25:44
good luck next time I'll re-delegate to your new thing and exactly well I could
1:25:50
just know when I can do it boggling mind-boggling yep but um
1:25:56
it's nuts this is uh I'm so what's what's coin water
1:26:02
saying uh I'm sure we'll get cut soon this is
1:26:07
awesome topic and time timely crashing yeah you know who knows when the uh when
1:26:13
the rug's coming um
1:26:19
cause less harm says great episode gents thanks mate um I think like I like to just talk about
1:26:26
random [ __ ] too by the way um and not necessarily always have a guest yeah I think that's what it should be we
1:26:32
can tease out a lot of stuff when we talk about this I I also get that like you know
1:26:38
many people who listen to this might just get angry about [ __ ] that we say because it's been quite truthful and
1:26:45
like you know I'm not trying to Sledge anyone but I think that people should like
1:26:52
be more transparent in in what happens and um it seems that for the most part
1:26:58
they're not and look something could happen to us one day something could happen to needlecast
1:27:04
or Rhino and I'm sure people are going to say [ __ ] about us but yeah um
1:27:12
there's no magic um Shield up there that makes your career validator immune to mistakes
1:27:18
um but I think we're a better set to to mitigate them
1:27:23
um as you know in a more uh probably reliable way than people who don't have a [ __ ] clue what they're
1:27:29
doing so well I think I think the you know the flip side of that though is like even
1:27:35
all the things that we do like you know threshold signing or TMK mass or you
1:27:40
know running in an enclave or a or or a VPS in a certain way with a certain set
1:27:46
of you know firewalls and whatnot however you do this stuff at some point you're building
1:27:53
on your at some point you're running somebody's binary You're Building somebody's code You're Building TMK Mass
1:27:59
right have I stepped through every line of code and TMK Ms no would I understand
1:28:05
if I did no because my background there's graph data
1:28:10
streaming data functional Programming type some bit of type Theory
1:28:17
dmkms could be reaching out and sending your private key to someone yeah I I and
1:28:23
right so number one it could be doing that I don't think it is uh based on what it seems to be doing traffic wise
1:28:29
but but this is the other thing like pure cryptography a lot of the cryptography going involved on in crypto is
1:28:36
fundamentally complicated and you have to have a CS degree or a masters really in cryptography and maths to
1:28:44
really like a a meaningful level grock that stuff you know like fundamentally
1:28:49
grasp it and and I think that's the tricky thing I guess is that as soon as you and that's where I'm always a bit cautious
1:28:55
because I'm like until you it's such a deep and complicated stack so we're working with here that like it
1:29:02
is a bit risky to to feel like you are out of the woods you know yeah
1:29:09
um because that's just too much to know there's just too much to know to to ever really truly sleep peacefully at night
1:29:18
laughs said the paranoid guy on that note