0:00
probably 40 overnight that's pretty hot is that when you had that mullet
0:13
[Music]
0:21
hello and we're you're listening to game of nodes a weekly podcast on the cosmos from uh
0:28
independent validated teams and it is very very very hot in europe it is
0:34
probably very cold in australia it's probably very normal temperature where usurper is
0:41
so different levels are bothered probably from our from our host this week
0:47
um no we were just talking about uh his amazing mullet we found out that he has
0:53
an alternate alternate alias as a pro golfer
0:59
usurper is very convinced on this fact might be some relationship there that cameron smith that's all i'm saying i'm
1:05
saying these they look exactly like i'm just saying there's some there's some similarities man i'm going with like something to say
1:12
that people need to look into it you know just have you have a little bit of background in the same room have you
1:18
ever seen them in the same room same time for folks on the live stream we have we have some video going on here of
1:25
of someone who looks somewhat similar to king notes that's all i'm saying
1:30
yeah that that picture that picture i'm not saying it to be fair but some of the other ones yeah i agree that one sounds
1:36
great i mean there is i really just need to grow out that ginger mullet that
1:42
mullet is really american is that an australian thing because that's that's like that's like usa right there right yeah it's like cletus mcfarland eat your
1:49
heart out drag car [ __ ] he looks like that he looks like that uh tiger king lad
1:55
doesn't he he does that kind of kind of little strange multi thing at the back there um
2:01
okay so i've i've been gone for two weeks what what have i missed lance i mean this is obviously purely for show
2:06
because i have actually watched the episodes so i did do know what was talked about but what i've missed
2:13
well you've missed drama i guess
2:19
well i asked on twitter i'd missed and it was like some paid shill some some people got paid to do
2:25
marketing yeah so they disclosed that they were doing marketing and that was
2:31
that was natural not on game of thrones i can tell you that so
2:38
nobody in this channel is making money off marketing i don't think yeah man like we need shills though we
2:43
need like a team of shills hey green uh
2:48
you know we're not very good at marketing or production
2:54
or anything like that really really selling you know when it comes for us it's time to sell out we
3:00
can use this our prospectus and say like well we're not good at marketing we're not good at self-promotion
3:08
we're tolerably good at social media and that's why you should partner with us at
3:13
game of nodes um for whatever your i don't know what do people partner on
3:19
making money that's another thing you can do your security solution
3:24
looking for are you simply safe in the toilet that's vpns isn't it it's always vpns
3:30
like always vpns sponsored by blur vpn sponsored by grizzly bear vpa is that
3:36
actually a vpn that sounds like it could actually be a vpn it is yeah we can turn it into one
3:41
sponsored by vpn you did miss out on good conversation we
3:46
had a great conversation with uh with ghost and with thy borg on some governance type i think we've been
3:52
talking about governance a lot because it's it's so fun um but we it was two weeks ago and then last week obviously
3:58
um we had the great highlander and and um and simon which i'm not going to try to
4:03
pronounce the actual name that was a great conversation well but just about what's going on within within
4:09
the new highlander nodes i guess and uh just what we're you know what they're trying to build and those types of things we talked again a little bit more
4:15
about governance in those types of areas but those were good episodes i nearly doxed highlander there just
4:20
there because i just because obviously they live in the uk so um i've met them and i was just like oh yeah of course
4:27
you know that was good conversation with uh but yeah i know that the thai ball was very good as well i thought that was
4:33
really really interesting episode but barry what yep tom
4:39
what's his name [Laughter]
4:50
yeah if we just uh if you come on the the show we kind of reserve the right to just dox you into the dark dimension
4:55
whenever we see fit let's see just realize what your real name is normally it's not it's meow doing that
5:00
to me to be fair but um i realized also that a while ago actually i'm gonna say that because that talks me further but um whatever i i
5:08
found another way yesterday while i was going through some stuff that i'm doxed and i was like is there any
5:13
is there any way is there any platform on which i'm not doxxed at this point um ridiculous so
5:20
um we have a couple of follow-ups i think um usurper which i've got the spreadsheet up here
5:27
um and so there was a there was a lot of talk about horcrux uh there's always a
5:33
lot of thought to talk about horcrux on the show but obviously um i know from speaking to schultzy and null that we're
5:41
now you know when and you're you're horcruxing too so it's it's now the hottest flavor
5:47
in the game of nodes treehouse um but horcrux
5:52
uh apparently had uh some role to play in the evmas double sign that happened
5:59
the other day which is probably is that the biggest drama of the week i'd maybe the last 48 hours yeah that
6:05
was that was stake like mo right um which i don't really i don't really know those guys at all so i'm just talking
6:11
about based on what i know so i believe that they is a partnership with somebody who's very who seems to be
6:17
quite good honestly from a from an operator perspective that this this happened to them and again we'll talk about a little bit more and somebody
6:23
more on the marketing side which i think is cryptigmo is that right i think that's correct um and they were in the atmos set and i
6:31
think doing somewhat well um and i think there's i don't know if they're not sure if that if there's some incentives there
6:38
within that validator or what's going on but the end result is is that they double
6:44
sign and i don't i don't really this is not a horcrux blame situation i think what the story is that came out
6:51
is that they're they were signing about 97 percent of blocks um and honestly in
6:56
etmos is not the end of the world it's a fast block time it has a decent amount of transactions there's a lot of failed
7:03
transactions in the mempool um we run like uh 59.50 or whatever it is on the amd side
7:10
to be able to handle that chain and still we still miss blocks honestly um and so it's it's not the end of the
7:17
world to miss a little bit of blocks in there and honestly anything that that that you're signing um
7:23
you know at a decent percentage it's not worth messing with um or just wait until maybe there's a co-change or maybe you
7:29
know there's maybe just a single node and bump the hardware a little bit those types of things anyway uh the idea was
7:35
that i think they were writing a note that they felt was a little bit underpowered and so instead of bumping that up they ran a second node that was
7:41
similar in size and wanted to put horcrux behind it the idea would be that i have two nodes in front i'm not going
7:46
to miss which i don't necessarily think is a valid i don't think they'll make a difference honestly because that would
7:52
i think it's i think that's an absorb commit issue right like it's just the node trying to absorb the tren like
7:58
i don't know you guys probably know no better than i do but i don't think it's not it's not really a signing issue in terms of how fast the machine is signing
8:04
like i think it's more of a i think it's more of a drive absorb commit type structure anyway
8:11
so in the in the midst of that i answered my own question i'm not going to wait for an answer from both of you since you're both on mute uh but in the
8:17
midst of that structure they were setting up horcrux and then they were had two notes next to each other they
8:22
were changing the nodes and and um changing the ladder address to be able to
8:28
point that signing back at the horcrux um you know be able to be able to basically map
8:34
from the horcrux swarm out to those nodes and in that structure the i i guess the
8:40
issue was instead of having one ssh session to each one of those box they had two ssh sections ss h
8:47
sessions pointed to the same box and when they restarted the other box was still have the validator key on it
8:54
and so horcrux was signing and the original validator was signing and so they realized that 15 seconds
9:00
later it's already tombstoned and 10 slash for all the delegations and
9:06
yada yada yeah so i think like part of the major drama on the twitter verse yesterday was
9:13
the payback um i think like you know there was just
9:19
they were probably getting i'm not sure if they were getting a hard time by other validators or their their delegators or or what it was but it
9:27
seemed like there was a conversation about whether or not their delegators would be paid back from the slashing
9:33
so i think that was like a big thing with the the twitter side of things yesterday as the fallout from that whole
9:39
situation that you just described um but on the technical side
9:45
uh guys so what do you guys do when you are
9:52
uh transitioning a signer i know what i do to make sure i got a [ __ ]
9:57
checklist [Music] but uh like what are you guys strategy to ensure that you don't double sign
10:03
like what are you doing what are you guys doing do you want to go first yeah i i mean my
10:10
away from horcrux or anything else with that my my first rule of thumb is to delete the private
10:16
validator file so though no matter what that gets regenerated from the node if it's not the validator it's there's no
10:23
issues in missing blocks like if you want to miss the blocks for 20 minutes and figure out what the hell is going on miss blocks for 20 minutes figure out
10:28
the hell's going on it doesn't matter it really doesn't um so the the idea is number one if you're
10:35
like in the past um even away from horcrux or anything similar to that used to move validator keys between nodes
10:41
right and and it's pretty easy and they're all named the same you can open them up and cat them but you're not like you're like oh yeah this is the key like
10:47
you're gonna remember across 25 chains what exactly what this key looks like in this hash like it's not really easy to
10:52
understand right so what i always do is i always keep keys local and they're on a veracrypt container and those types of things so i
10:58
always have obviously a number of backup of what i know is is proper and then number two is my scripts what i run i
11:04
delete those keys because i don't give a [ __ ] about them because if they get re if next time it starts it'll if it's not
11:10
there i'll create a new one if i'm not signing i know something's wrong so if i'm if i have three or four nodes then i
11:15
know i'm moving a key that first thing delete goes into those ssh sessions and it deletes deletes all those keys um
11:21
because i know that no matter what it's going to regenerate that key is never there so if if i'm restarting five or
11:26
six maybe not five or six three or four nodes i know that only one of those has a key on it and the rest of them are
11:32
blank um and i don't have any checks on that that's just basically the script i'm running and have some structure
11:37
around that but it's one way to ensure that no matter what when you restart um
11:43
hell you can even build you can even build into the brick and service file if you wanted to on non-validating nodes to be able to do that but the idea would be
11:49
that it doesn't matter so just delete the key and move on um and and if you're not if you're not using a you know a
11:54
team kms or like a horcrux or something like that where you have all that set and everything else going on it's a real issue and this situation i think the
12:01
only thing that they obviously could have done better is there's no reason to not go back and delete those keys off those notes like
12:09
and they might have he might have done that on the one server if he was actually ssh into the same server twice
12:15
but i think you would have figured that out because you obviously started the service on the other server that wasn't
12:21
shelled into so there's something wrong there right because you would have obviously realized what was going on
12:27
because the other servers should have been stopped my guess is that they're just moving too fast right and you're worried about
12:33
missing blocks and there's like this real weird thing in validators like if you missed like 100 blocks everybody looks
12:38
at you like who cares like you can get missed for five hours it doesn't matter right like just
12:44
make sure you're not doing this which is far worse yeah but you don't want to miss your phone yeah all right so four hours
12:51
the the first thing we do is we disable we turn off cosmic visor which is what we
12:57
use on on pretty much everything we stop service um and then we quite often disable the
13:03
system file as well um because i don't want to accidentally tab up through
13:10
recent things i've done and just by sheer luck that's that's i'm very paranoid people know this i think
13:17
i don't want to accidentally hit the keyboard twice and let's say the second or third command above is
13:23
pseudosystem control start the custom advisor start right and then let's imagine i have a moment of madness
13:30
i hit enter don't want that to happen so added paranoia sometimes disable
13:37
the system process or it until i'm done but regardless stopping it is absolutely
13:43
the first thing second thing any of the private state files um get
13:49
immediately renamed with a dot old on the end of the file name um i just have a script that grabbed
13:56
that just pushed just mv sudo mv both files to json.old so there
14:02
i can see them if i need to um but so that we're talking also this is about a when you have the keys on the
14:09
box obviously we use tmk-ms so we don't have that anymore um but this is
14:14
hypothetically what we do um now obviously we're just shuffling centuries so we just
14:21
who cares really um but those are those are good uh sense
14:26
checking things to do it's just always if you're gonna mess around with something that could cause you a lot of problems just
14:33
rename the json file to hold on the end and you can just instantly see oh this is a thing i might want to keep keep it
14:38
around until i'm done with this operation and then you can delete it at the end and then at the end of that only then you know do you restart stuff
14:46
when we moved from when we moved from our previous setup
14:52
which was a traditional century architecture with some additional steps but we weren't using a remote signer
14:59
we took an entire day the entire working day booked out
15:06
no slack no telegram nothing to do it and left all that time
15:13
so that any time there was a thing that took a little bit longer than expected or something that didn't work first time
15:18
or whatever you know an ssh session that hung or terminated or if you're doing longer or
15:24
more complicated operations if you're spinning up cloud infrastructure which obviously we use a lot of aws type stuff
15:29
so you can you know if you're using automation to bring up a box or something like that sometimes sometimes things go wrong and then you have to go
15:35
and fix it in the console or whatever same for hexner as it is for aws at the end of the day because you're still so
15:42
some somebody is not plugging in something somewhere it's all automated um
15:47
you don't want to be in a position where you're stressed out or you're in a hurry you want to just be like it's fine i've got five hours left in the working day
15:54
whatever and uh and you know you know like i said also there's a
15:59
period of time where we fully down i think you could probably look back in the in the record and see us missing blocks
16:06
when we shift our juno because i actually made a mistake in the in the ladder configuration
16:13
um and it took an hour to debug it we missed a whole hour of blocks on junior
16:18
and then had to catch up and do you know what happened nothing yeah absolutely nothing happened
16:23
because the cosmos sdk is designed um as we've had cosmos ogs on here a number of
16:30
times um telling us that you're allowed to miss 13 hours of blocks there's a reason for
16:35
that it's because you've got you've got somewhere between let's say 74 and maybe
16:41
125 or maybe even more validators who are also picking up the slack for
16:47
you so yeah you being down for a short period of time is not as important as you
16:52
protect your key matter um which is why and and then the second thing is obviously like like usurper says you
16:58
know um have your keys backed up somewhere very
17:03
secure so that in the worst case scenario where you do delete stuff
17:08
um you can be in the situation where the worst thing happens is not you delete your key it's that you essentially bring
17:15
up the service early or whatever that is so you know we actually could do a similar
17:21
thing to usurper and just delete our keys entirely because we have a hard drive and a bank vault with our keys on
17:27
and in the worst case scenario we have seven day a week access to that so
17:32
we if we really screwed up bad enough and it was 10 o'clock at night we'd go to bed sleep for seven hours eight hours
17:39
like the cosmos sdk has assumed you will be able to and then take our laptops to the bank the next day and do it
17:45
better that's what we do and that's all a very convoluted and unlikely scenario but
17:52
it's better than the opposite right like i mean the i mean the point is valid where where that stress around
17:59
around trying to be worried about missing blocks is really a manufactured one they really
18:05
should not be there right and i asked somebody pointing out um the idea around doing pointing and
18:10
calling which is i think is a japanese occupational safety thing where like conductors of japanese trains have to
18:17
based on things that are coming they actually point at the sign and be able to read it out out loud and shout it um
18:23
so like in in those situations like your brain kind of gets out of the loop and like has to kind of focus very
18:28
succinctly on what's going on i'm somebody pointing to that because in this types of situations especially this situation too which is which i think is
18:35
probably the most delicate uh and most obviously probably other than halts
18:41
and goofy halts kind of like what happened with evmos and and bad information being shared this transition
18:46
of being able to change a signing authority is probably the second most common area that you're going to maybe double sign right
18:52
just running a validator it's not going to happen but this type of thing where you might have two sounding authorities or i'm moving between a and b or
18:58
you know i have a halt and somebody does an unset result you know unsafe reset next thing you know you're resigning blocks you already signed
19:05
things like that like those are the areas that that's going to happen and stopping and
19:10
making sure you're being very clear and what's happening is important um because it's really easy to i mean
19:16
again this like this this error sounds like it's a couple errors that maybe just two or three that that backed up on
19:22
each other that turn into this problem but it can it can happen right like i don't think any of us i mean these are
19:27
good processes that we're going through and noel's going to tell us his since we've been talking for 20 minutes without him
19:33
but the idea is that it can happen even with good operators right in the in the heat of the moment what's something
19:39
that's going on all right sorry go ahead yeah
19:44
absolutely is something that can happen um but obviously you can maximize your chances of it not happening by having
19:51
your process right so like you guys both have i mean mine's not going to be drastically different to what you guys
19:58
do i think like we're we're 100 kms now uh
20:05
horcrux now anyway so all our nodes are disposable they don't really have any keys on them anymore but
20:12
when we were doing migrations uh i actually just delete the users so i'd
20:19
start a new node um i would point the
20:26
um horcrux to it but before i did that i'd just stop and
20:32
disable the other node and keep it around just in case there was something wrong with the new setup
20:38
uh i'll start up the new one after i'd missed like maybe 10 or 15 blocks just to make sure it was a good good amount
20:44
of misses between you know starting and stopping the nodes uh so every time i do this i always have
20:52
um like ping.pub or minscan or both up next to me um and as well i might even
20:57
wait for like tender duty to uh give me an alert that i'm missing blocks
21:04
and then start up the new note make sure it's working properly and then stop all the services and delete the the old um
21:11
users because we run a couple of um networks on each box uh because they're plenty big boxes
21:18
and so um yeah then pretty much nuke that user
21:23
start up another new node on there resync it add it to the horcrux configuration and restart those
21:29
and then add more but the keys initially we just cap them out of the config file uh save
21:37
them onto our encrypted area and sync them so yeah we never really
21:45
once we um sort of get the key we never really use that box again um just to
21:51
just nuke it and then pretty much we have a configuration for each network that we just clone across
21:57
the new nodes so it's pretty easy to set up new ones you can get them up in like 10 or so minutes
22:03
um and now you know poker is so bloody organized with all of these state thinking and
22:09
snapshots that you can get i can literally get a note up in probably manually completely manually without the
22:15
pokemagic where you press a button and sit there and get a cup of coffee probably completely manually typing it
22:22
out i could probably get a note up for any of my networks in probably
22:27
10 to 15 minutes i guess right um from you know new user to synced so it's
22:35
pretty good these days um that was not the case previously we
22:41
like go back just a couple of months you're always looking for peers you're always having peering problems you're
22:46
always having trouble like finding something a snapshot that was close to the head
22:52
state sync was dog [ __ ] and didn't work and so like you know just a couple of
22:57
months ago it was hard to like raise new nodes quickly but now
23:03
it is very very easy it can be done really quick so i don't see why not just nuke the old node and start again
23:10
without keys added to it at all so you know that you don't have any old signing
23:16
material lying around on there one thing that actually i'm very interested in at the moment is
23:22
um having done like quite a bit of research into running um nodes on a cache and
23:29
obviously put i've put pen in that because you can't do private um you can't
23:35
selectively whitelist so it doesn't work at the moment um for
23:41
running with dm kms which we use um but obviously it's possible to put a
23:46
reverse proxy on a large box and then install docker and then you know
23:52
bosch if you were willing to go to the dark side you could also just use amazon managed
23:58
kubernetes service or indeed ecs which um i have tons of terraform
24:05
knocking around for for previous from previous jobs um to run omnibus images which are provided
24:12
by a friend of the show tom from ecostake those images really do work very well
24:18
for syncing an rpc node and obviously if your key matter is elsewhere you can
24:24
actually just use that then as your as your sentry so
24:30
that's something i'm quite interested in trying actually um aside from the possibility to move start to move stuff
24:36
on a cache um i'm quite interested actually to maybe try running a couple of big boxes
24:43
and essentially proxy to docker containers running the chains um on that and then pay for one really
24:49
big ssd to each of those boxes rather than paying for lots and lots of amazon storage
24:54
um so i think from memory from talking to ecostate tom
25:00
um tom from ecostate you could stay tom i keep switching it around um he his his strap for his validator nodes
25:08
is actually several large boxes that are all running multiple things on on docker
25:13
containers using omnibus so um and i think he uses a remote signing setup
25:18
of some description as well i'm not sure if it's tmk-ms i think he was talking about moving to either that horcrux
25:24
i think he might have a different solution that he's sort of worked out so
25:35
okay so i and i think that the next piece of this was we could move on from this topic but i think after that point as as
25:42
null pointed out before so they had this happened and then there was the realization of this happening and then
25:47
there was this like this is [ __ ] why why did 10 of our tokens get slashed for just a simple mistake which
25:54
hey not you know i can understand that right like this is not malicious this is something that just occurred and um
26:01
and you know the realization that there's 50 or 60 000 usd that was wiped away based off of five
26:07
seconds of of just a mistake um yeah i mean that's the immutable part of this
26:12
right um that's kind of what happens when you get involved in this type of thing and then there's some realization to say oh [ __ ] these are then i lost is
26:19
money for the people that are involved in here um and um and and oh [ __ ] like now i'm now can
26:27
i spin up a new validator and how am i gonna pay these individuals back and all these and then we have this other kind of stuff around you know just
26:33
understanding that the implications of being a validator um and so i think it's a good honestly
26:39
in my eyes it's a good lesson for both delegators not just theirs but overall
26:44
and validators to understand that there's real there's real money at play here right and and
26:50
just because ignorance or mis small mistakes or anything else fit into it
26:55
there's still ownership like this is still something that happened and you need to take ownership of it regardless
27:01
of the hey it wasn't malicious or anything else that's it doesn't matter right um and so i
27:07
think there's some realization of what's going on there and the uh i think it's like
27:12
now become evident that the slashings on uh evmos are pretty savage
27:18
so that's not really this 10 is i don't think i've seen that on any other
27:23
network and even the um the soft slash is like 0.5 which is
27:30
a lot more than what the usual 0.1 is on on most other networks so one of the
27:35
other validators got slashed the other day for a downtime um and they repaid their
27:40
uh their stake is they're quite high on the list it cost a lot of money
27:46
to repay them for that downtime slash so i mean it was because they assumed it was like you know their script was point
27:53
one like everything else usually is and then they um they upped it to like 0.5
27:58
after that everyone's like hey that wasn't it but then this is also part of the so if
28:04
you try and start a bank um validators aren't banks this is not financial advice blah blah blah but
28:13
if you try and start a bank one of the biggest things you have to do is ensure liquidity um
28:18
and well there's actually a whole bunch of things to pay your watches jurisdiction you're you're in but um
28:24
you you basically have like a multi-year time scale to becoming one where you're only allowed a certain number of users
28:32
essentially and it scales this is in the uk i've got a job offer for one a startup bank once
28:37
we have a lot of startup banks and things so um you start like year one as a bank and
28:43
you have like five customers or like 20 customers you're allowed to have like a tiny number and then it scales up quite
28:49
rapidly year on year but it's all about basically ensuring continuity of service to your delegators
28:57
essentially and and like obviously validators don't have quite the same model but this is also like we
29:03
we've gone around the houses a few times around you know when do you cash out how do you cash out why you cash out
29:10
all that sort of stuff and i think like as soon as you start looking at the slashing and business continuity and that sort of thing
29:15
it's a lot easier to look at those things of as okay well this was a screw-up
29:21
um but you know it's okay because we have the money in the bank because we already
29:27
cashed out according to a business continuity plan that said we should have enough in the bank to cover a soft slash
29:33
or a hard slash um as for a hard slash i mean i think there's really no
29:41
okay so spicy take here i think the hard slash should be your big number
29:47
yeah i don't know i think it should be a small one is anyone promising heart slash
29:53
fray i don't i don't know anybody is right promises it because it's so savage
29:58
i mean and even now like you have mo said a you know two bucks if this was i mean again depending on launch times
30:04
and crypto prices and all this kind of stuff if that was a twenty dollar coin you're talking
30:09
six seven hundred thousand dollars u.s like they're not you're not going back i think if you get i think if you get hard
30:15
slashed um in a situation where we're not regulated i think that is the end of your
30:20
validator operation on that chain i would think so i i think for us we would
30:25
i i don't honestly know what we do if we had the money maybe we would cover if we had the money we'll probably cover it
30:30
because that seems like the right thing to do for a mistake on our part if we didn't have the money then you know i think we
30:36
would just be saying i'm sorry and then and then that's the end of our validation operation on that chain
30:42
possibly the end of our validation operation full stop it's it's kind of hard to say isn't it because like like you alluded to earlier you can have a
30:48
chain hole where for complicated reasons that may in fact be developers originated or a buggy a
30:56
bug or uh you know whatever it is possible under certain situations to double suck
31:02
yeah sure and in that situation you had not to go back to that essentially but that situation you had even developers
31:08
giving bad information because they didn't understand the details of and just because you're
31:14
same thing i'm a validator i don't know [ __ ] about the development chain their development chain their developers on
31:19
these changes they don't know [ __ ] about the validating side because they're not the same skill sets right and
31:24
yeah they might not understand the details of the blockchain but they might understand obviously the contracts other types of things right so and there's
31:29
there's huge there's a huge amount of stuff to know like i mean you know even people are and i'll say
31:34
somebody who works on juno there are things about say the anti-handlers or the end blockers that i could easily
31:39
make a you know be like oh that's probably fine and then it wouldn't be and you know
31:45
actually a really good example would be things that are consensus breaking when you're looking at an upgrade and going
31:50
which bits were going to be consensus breaking which ones aren't going to be um i remember relatively recently having
31:55
a conversation with some developers from another chain and they were sort of arguing that
32:01
something would probably not be consensus breaking and i was like well i think it might be
32:06
and whenever i am on that it might be side of the fence i'm going to call it consensus breaking because the
32:12
autonomous alternative is bad and it turned out to be consensus breaking when
32:17
you maybe would have thought it wouldn't be but it was um so there is a you know there is a lot
32:23
to know in cosmos and you know it yeah like like the the first evmos hot as a prime example
32:30
where in that situation default doesn't really lie with the validator
32:35
really if you're given instructions you know possibly or maybe it does because maybe you should be maybe you should know enough about the
32:41
sdk to look at their instructions and go these instructions are wrong but the number of people that we have to spot
32:46
that straight off the bat you could probably fit in the back of a london bus you know it's probably 30 people in the
32:51
world yep i agree it's tough like it's just a
32:57
it's a there's huge risk to it and like this this idea that validating is easy i i think is just
33:04
there's a there's a heavy experience part of this and this is a very new industry right i mean i've been this for 18 months or whatever the heck it is and
33:10
so it's not like i'm drawing from 30 years of history here or 20 years or 10 years or five right um and even at even
33:17
at my level i think that i i
33:22
how do i say this not being an [ __ ] i i feel like even though i'm not the smartest person in this conference call
33:29
i do feel that in many of the test nets that i get involved in i'm in the top 10
33:36
of people understand actually what the hell's going on put it that way and um
33:41
and that that is that's that's not a bad thing like there's it's this is a growing industry but at the same point
33:48
there's real things at play here and people don't i don't think or really understand the the risks of what's involved in that um
33:55
and so both from a delegation perspective as well as a validator operator like yeah you could say i'm going to bring up a vm
34:01
and it's no big deal and blah blah blah but there's there's there's a lot of [ __ ] behind here right it's not it's more than just running commands yeah
34:08
so actually another another thing the final thing i i think say on this is that the other thing that besides
34:14
everything else that we were talking about about operational stuff you just hit on it usurper is that you
34:20
run a lot of test nets and uh
34:25
now from my memory you said because i was talking about this on twitter i think we got tagged in the thing saying
34:30
oh you should do more testing should be involved in test nets it might have even been you that was saying it now um
34:36
but i remembered you know uh block payne making the jokes about the astarte test nets and we did four of them in three
34:41
weeks and an insane number of upgrades and stuff and i think usurper you were in three of those maybe oh yeah or three or
34:49
four of the astarte test nets like yeah and those were i mean i i think we all
34:54
double signed on one of them because there was a problem with one of the anti-hand there was there was some um
35:01
co-change that we had screwed up and then the upgrade handler booked and i think
35:07
all of the like something like 20 out of 30 uh what would have been they're about 12 validators in the starting weren't there
35:13
so it was something like two-thirds of the validator set double signed but the time to do that is in a test net
35:20
because you know that's how you learn how these things happen right you go like oh yeah okay if
35:26
the chain holds and there is any kind of uh modification state at
35:33
that point there is a good chance you're gonna double side and that's why like there is there have been several juno
35:38
holds now um on upgrades on things that you know have not gone right and also we had to
35:44
restart the chain when we got cyber attacked but in those situations we also had people who had the experience of
35:49
these started test nets where they had double signed on a test net to go okay
35:54
don't worry turn off your validator stop your system and wait
36:00
right and then restart it once we're all ready to go there's things like that and you
36:05
know and you know to touch wood it won't we won't have a breakdown that will result in double time as a result of
36:13
developer coordination on juno but it's not impossible you know yeah
36:19
but we managed to navigate those problems i think because of that experience like um that is back to and again it's back to
36:24
that calm thing you know the point i think the more the more experience you
36:30
get in things going wrong and especially in like test net environment like the
36:35
more confident you become with the fixes to those things going wrong right so over time you develop like a mental
36:41
toolkit of how to fix things and then you can see
36:46
like even in test net environments when something goes wrong in the test net the [ __ ] discord is frantic with people
36:53
trying to like make the thing work right and no one's really sitting back and thinking about
36:58
how they can make it work without just asking questions and spamming the [ __ ] out of everything so
37:04
once you like you know if you've done a hundred test nets and you know three or four hundred upgrades
37:10
then you like your toolkit that you've got to handle these situations is
37:17
broad right and then you can you're you're in a situation there where you're not panicked you know what's
37:22
going wrong you know what to do you can just sit back determine what's wrong look at your logs
37:28
figure it out fix it sit out while you know and you know if you're if
37:34
you're using like local signing um matter then the answer might be to just turn your
37:40
[ __ ] off like the phrasing and just wait for the network to sign because your chances of double signing might have
37:46
gone up by you know 90 so um you know not so much an issue when
37:53
you're remote signing but these are all the things that you do in a test net you know they exist you know how to
37:59
handle them and then you can be you know a far better mainnet validator particularly in a pinch
38:05
so um you know it's like constant development right you're you're not just automatically
38:12
know all about validating it's it's a it's a war chest of knowledge that you build up over a
38:18
period of time and the more exposure you have to it the better your war chest
38:23
so while you guys know when and when you get stuck you just go ask pokechu right
38:28
yeah parkachu or or um schultzy right
38:34
or or code hands or or um george
38:39
that's right you're like hey man how about this thing so um while you guys were talking before
38:45
i i just did like a few random calculations right and so on fmos it looks like if you get hard slashed at
38:52
the current rate of rewards and you know the the standard five percent commission it would take you six months to pay it back
39:00
six months of rewards to pay that back so the the network hasn't even been running that long
39:06
i don't think that that's a really cool calculation that would actually be something really
39:12
interesting across chains to publish to say at the reward
39:17
rate what is the payback on a hard slash that's actually a really cool that's a nice metric i mean the numbers itself as
39:23
a as a it's an absolute kind of get lost in translation like in this example it's
39:29
you know 60 grand of rewards but but if i'm a small validator how many how long am i running operations to be able to do
39:34
that or how long should i have how long how how much money should i put away from the past to be able to pay for the
39:40
chance of the future type of idea that's actually yeah so i guess that's an ever increasing number too and it doesn't
39:46
matter it doesn't matter if you're like a big validator or a small validator the payback period would be similar based on
39:52
the rewards for that amount of good point yep good point because i actually just did it on like
39:58
percentage and i didn't work it out on dollar values um
40:04
yeah it's a pretty easy calculation too like don't don't rub it in just just
40:11
like i did it well you were talking it's not like he's building a bridge or anything easy it's an easy calculation
40:17
for you okay if you asked me yesterday what like this multiplication was it was like a [ __ ] man i use a calculator
40:24
[Laughter] i'm gonna go stick your tables [ __ ] on the back of the toilet wall because
40:29
you're always in there doing a [ __ ] so just do that read it all the time maybe it'll stick otherwise just use a
40:35
calculator perfect
40:40
[Laughter] you know you're so good at
40:45
math and [ __ ] okay yeah math while [ __ ] or math and [ __ ] um
40:51
the um the one last thing the other follow-up we had was just around the validator training and things like that which which kind of is just on the tail
40:58
end of what we were just talking about here it's is i was thinking more about this week because we had um there's just
41:03
a halt on the say test net on atlantic one because just of a code issue that happened when the max number of
41:08
validators went down and i guess there's two different modules within tendermint one of them was changed one of them wasn't under something somewhere that
41:14
you guys would probably know better than i do um but the two things one is i think my role in
41:20
that was trying to calm people down because there was a lot of conversation like what to do and stop notes or start
41:25
nodes and those types of things and the idea that that's good val it's thing just to wait let them let the node just
41:31
dial away it doesn't matter let it do its thing it's fine don't take any drastic actions those types of things um
41:37
the other area was that in this area in this instance the validator set was pretty inexperienced
41:43
because it it was again i don't mean to be an [ __ ] on this but just just based on how they're
41:48
doing kind of chunks uh there were in the validator names that were in there there's a lot of individuals which is cool like i think
41:53
it's great um and it's an incentivized test nut so you have a huge amount of participation but you don't necessarily have folks who have run this before and
42:00
so there's a lot of like basic linux questions and like how do i tail this out and like just simple things that
42:06
as somebody who i mean that face that you're giving right now fray is probably the same face that the say team was
42:11
giving which is oh [ __ ] i have 75 validators that i need to get going on a specific thing then we're doing a mid
42:18
we're doing a halt that we have and we need to do a binary change and what's the right change that we can do to
42:24
reduce the amount of [ __ ] upper e that can happen in the field right and how do i try to not only fix the
42:30
problem but also do it in a way that has the least amount of other kind of side effects and things like that um and this idea of like
42:36
and that's not the really the right place to do training right and this idea around finding a way to allow people to
42:43
be a part of these situations that are not on a project's structure and
42:48
you know kind of the idea around doing validator training in these types of situations like your point like like what was going back on with um with
42:55
juno like that was amazing training that might not ever happen again and the people who are involved in that learned a shitload but now you have 1200 other
43:03
validators that want to get going who don't have that opportunity right um and i don't know how we create
43:08
that i don't know how we create that situation but boy that's valuable um and it'd be cool
43:14
to be able to create a game again kind of sounds like say needs to hold a couple more times or maybe maybe it's a
43:19
game at your boot camp it's a gaming notes chain that just constantly [ __ ] us up and then we
43:28
just keep introducing bugs that [ __ ] it up that's easy yeah i think we could do that right we could find if it was a
43:34
[ __ ] up champion i know a guy that could they can write something i know what guys yeah like i mean i think for for
43:40
um i mean we did like seven gentexes in like six hours right like we did a it
43:46
was a shitload i remember um
43:51
we did one we did one where it was we were trying to resuscitate a chain uh we
43:56
would see actually very similar to what ended up happening with the junior cyber attack um yeah and it was a data point we used
44:04
with the genus cyber attack um was we tried to resuscitate a chain by
44:09
turning off all of its validators and then essentially yeah putting it back together with the
44:15
same id um and it choked because there was still uh there was one validator who was in a
44:21
different time zone and hadn't checked in and they were just that they had already pre-committed i think and uh we
44:29
all double signed um so then we had to redo gen tx's and bring up the chain from genesis at well
44:36
at my time it was um i think we we did something like at one o'clock in
44:42
the morning my time right it was like we're going to do gen tx's in 20 minutes time and then run
44:48
they're 30 minutes and then we're going to bring up the chain 25 minutes after that and amazingly it
44:56
all came together that's right chain came up and i went to bed that's right i think we're i was
45:01
commenting like why'd you wait 25 minutes we could have done this in five minutes like at this point like why did you push this out why is it gentex time
45:07
so far in the future i gotta wait 20 minutes for this ridiculous 20 minutes man what the hell man i'm gonna go to bed like what are we doing
45:14
yeah well i didn't participate in any of those because there was like 40 upgrades at
45:19
3am i was like [ __ ] no
45:25
it was intense although it was so late a lot of that stuff went wrong european time they probably would have
45:31
been actual day time for you and you would have been fine yeah could have
45:38
five or six of you guys that were doing that todd todd's just chimed in so yeah
45:43
yeah that should be a badge
45:49
honestly i've joked about it a few times uh about doing an nfc collection for the astarte validates i kind of think i
45:55
should because it was a very very very difficult few weeks
46:02
that's true oh dear um but yeah no
46:07
so do we finish follow-up after 50 minutes yeah well to be fair like the rest of it
46:14
it's all sort of follow-up there are like thousands of test nets at the moment
46:19
right are we like loving this or are we [ __ ] hating it [Laughter]
46:24
i'm just not involved in any of them i can't can't keep up with them at all there are lots man it is crazy crazy
46:32
crazy um i stopped trying to be in all of them just picking the ones that we think
46:38
might be good main nets so there's just so many and uh
46:44
there's not so many incentivized test nets though there's a couple floating around but but not [ __ ] loads but
46:50
those those incentivized test nets tend to just get completely flooded with people they do and a lot of them don't
46:56
have a [ __ ] clue what they're doing but it is a good testing ground and learning ground still
47:01
um and it's uh you know it's kind of good for the other validators too to be
47:06
able to stretch their legs and their minds and and help people with their issues dig into
47:13
their knowledge base and uh try and help other people with their issues is like a good exercise because you don't you know
47:20
when you've been doing it uh a while you don't always have these types of issues yourself um anymore and it's
47:27
sometimes good fun and good mental exercise to try and diagnose other people's problems
47:34
based on a very like vague and uh unhelpful description of
47:39
the issue [Laughter] well it's kind of about i think meow always says like
47:44
it's it's okay to be a noob because everybody was nude at some point but it's not okay to be a noob forever right
47:52
so that's that's how you kind of arrive at the kindness because you go well people
47:57
people helped all of us um like i know i certainly asked jorge
48:02
and meow a bunch of really stupid questions when i think as a stargate was my first test nap
48:09
back last february january something like that and i now asked a bunch of cheaper
48:15
questions like i really distinctly remember that they had very recently changed their name from stake bird which
48:21
was their original um name to stargaze and the
48:27
i had the wrong git commit i think of the docs and then
48:32
i was like these were out of date and they've re-changed it back to the wrong one and i was like this is ridiculous
48:38
this is obviously the wrong binary name and then submit a pr and then i think jorge was just like
48:43
um why have we done this like ah
48:51
um he was very patient he was like i though is the other way around it was correct and now you've broken it again
48:57
i'm like ah okay sorry about that somebody someone should create a
49:04
cosmos ecosystem trivia game because we're getting all these like random things coming up i think the original state of stargaze should be in one of
49:10
those questions that's a pretty good one so there there is like um
49:16
tell me it exists and tell me it's an nft and there's already a chain and i'm missing test net no no no no
49:22
uh there's there's um what does it [ __ ] sniff chain do like a
49:28
cosmos type trivia thing so dan b was telling me about it the other
49:34
day he was doing it one other night it's just like an online thing people hang out
49:39
and do like this trivia um and i think it's like you know you click on a window for
49:45
i'll i'll take sure you can know it's falling over for 100.
49:50
and then like it's got a question that you have to answer and you get a score and [ __ ] so i mean there are some things like that
49:57
around that would probably be pretty cool to have a look at and participate in with maybe you know like fear of getting
50:03
a [ __ ] score yep it's also unchained yeah yeah it's all like logged on jane
50:10
your scores two out of a thousand [ __ ] [ __ ] [ __ ] and it gives and it's an nft
50:17
and goes up next to your validator that's right [Laughter] direct link on min scan 100 uptime
50:24
very bad knowledge of mine yeah don't delegate
50:31
but there are a good number of test nets i mean i think uh obviously we talked about say right now defund is getting
50:37
ready to relaunch on the next test net uh stride is coming this weekend you guys have heard of stride i think we
50:42
talked about that a little bit stride i think is a um
50:49
liquidity staking type of competitor to quicksilver i believe
50:56
um i haven't really gotten involved in too much yet uh who else is who else is actually getting ready to launch a
51:02
a test net coming up cosmos sure yeah of course in cosmos um
51:11
i think defunct's probably the main one um but i i kind of got loose a little bit
51:16
involved in the last one but that that sinking was an absolute nightmare and the box the box i was
51:21
having to use was really big and i was kind of like because it's not incentivized and stuff
51:27
it was like this is actually throwing a thousand dollars into a whole per month in a bear market um
51:33
right with with some additional work required and
51:39
it's difficult isn't it because you're like i i think that's a good team with a good product but i think so um
51:46
you know it's one of those where it's it's like okay if if it were a case of putting together a validator rig for it
51:54
um or if there were specific test net things that i could kind of help with i think i think the problem i have with
52:01
time restriction stuff at the moment is that especially as needlecast we've sort of talked to by sometimes like oh yeah could you
52:08
get involved with this test now are you interested in getting involved in this test net and it's like well look if the test net is going to have
52:14
challenges like can you use our binary to submit a thing then no it's a waste of our time sorry
52:22
um but if you're like is there something we can help with
52:28
then if there's something you have in mind then you know it's almost more of a consultancy thing i guess because
52:35
my background is software consultancy and stuff architecture consultancy i guess and i'm kind of just in that space now where i'm like
52:41
i don't pretend to know anything everything by any means but very time limited and
52:47
for obvious reasons juno is kind of priority so it's like i would love to help out other chains but i can't
52:53
spend all my time running test nuts sure but i assume that even like even an hour with you on the phone on some of
52:59
these types of things again going back to different issues that have occurred and not just techno not just technical but
53:05
also running successful test nets and you know how to bring chains online those types of things i think would be valuable for anybody who's calling you
53:11
right yeah i mean and and like you know i'm not the only one from juneau who
53:17
reaches out to to reach out to other people other people when they see something that they can help with i i know schumer c for example is obviously
53:25
not directly for juno but is very active at helping up people with test nets
53:30
drawing on you know astarte and other things that he's done um and you know there are other people
53:36
within juno organization that actually reach out to other chains to say hey you might want to consider this because of
53:41
that and obviously the stargaze folks are also very very good for that um because there's a
53:47
lot of information obviously flown back and forth um over the last year and a half i guess between those two teams
53:53
since we're since we're on the topic kevin asked what has been everyone's favorite and smoothest test net experience historically
54:01
i'm gonna say star guys i would say so too yeah i think that was just the
54:06
every saga test net in the run-up to their launch was impeccable even even every upgrade right like
54:12
like everything just kind of moves i you know i think my second would be probably omniflix we've always had really good
54:17
sisler and team have always done a really good job with that but i think it also shows like those are teams that have just been in and around
54:24
cosmos for a very long time i mean stargazers being built for what two
54:29
years and then both shane and jorge were in the ecosystem before that so right
54:35
they have such a deep knowledge of the sdk that literally i mean almost
54:41
no teams can catch up with that depth of knowledge now at this point um but it is why they can make those
54:47
things so smooth because they they've got an extra three years on top of us in terms of their diagnostic
54:53
experience the thing that you were talking about now where you look at a thing with a vague description and you go
55:01
yeah jorge is really good at that i give him vague descriptions all the time i might just
55:07
do that um but i think like you know another thing
55:12
another way to look at it is not necessarily like you know like what's the best run and
55:17
most smoothest experience but you also get to look at like completely different scenarios right
55:24
because things happen in test nets first so everything's first in a test net so
55:29
when you're running one second blocks that's happened in a test net first when you're starting a network with 800 [ __ ]
55:37
validators that happened in a test net first for example the early um
55:43
omniflex test nets were like i think 800 strong was the first flix net flix net
55:48
one um and it wasn't a complete [ __ ] disaster surprisingly it actually worked
55:54
the blocks were enormous but the thing started uh i think the first time i saw
56:00
fast blocks was kuji um and then shortly after evmos
56:06
yeah but now say yeah man and now yeah everyone's getting on that fast block everybody's got
56:11
another blockchain we did actually get a question about fast blocks and i wanted to put it to you guys like do you
56:17
because it's obviously basically validated tuning and hardware as a combination of hardware and
56:24
it's reducing block time out right is the primary mechanism [Music]
56:30
what do you guys feel about do you have a feeling on on this on the fast block
56:36
craze all the kids are doing behind the bike sheds i think it's it's a trade-off and it
56:42
really needs to be specific to app chain stuff right i don't think it makes sense to try and do it on um
56:50
cosmosome chains because those chains you know people are building apps that suit that environment
56:56
right but when you go to like the app chains like um 2g it makes sense because you need
57:03
a fast block time for that type of order book environment but you know there's long-term implications
57:10
of that as well so these databases are going to be [ __ ] enormous they're going to be six times bigger than this
57:17
you know one second block is going to end up being six times bigger than a six second block over the same period of
57:23
time so they already get pretty big and i'm pretty sure no one is going to keep all that history like that's a lot of
57:30
dollars to um you look at that over a five or ten year period right so
57:36
the amount of ssd you'd need to keep the all of that history available to people
57:41
is enormous and i don't think i think it'll just become a trend of cost chains won't keep history there'll
57:48
be like a trailing history of like a few months maybe where there's available snapshots for that but then no one in
57:54
their right [ __ ] mind is going to have like 30 terabytes of bloody
58:00
um you know nvme to store the history of a one second
58:05
blockchain after a couple of years it's just probably not feasible so
58:11
yeah i think i think from that perspective we're going to lose basically not going to have history for
58:17
these genes because of that fast block time but also um
58:22
you know it's harder on the hardware the the faster they're going the more pricing processing cycles you have to do it's harder on your hard drives um
58:29
you're going to get that where a lot quicker on a one second block chain than what you do on a six second
58:35
um so they're all they're all factors that i don't think anyone thinks about and if
58:40
you pay for networking so our our second biggest cost after our hardware and we run
58:47
pretty big ec2 instances is that we have to pay
58:52
to be fair it's a very nominal amount for for network
58:57
um usage but it is actually our second biggest cost so if um
59:03
block times halved on juno we would double our monthly cost um
59:08
and the question is what does the end user see for that um
59:15
and and that's that's coming from the perspective of people launching a protocol that could use shorter block
59:21
times so i'm kind of in two minds about it like if we end up with a successful launch of
59:27
how you know three second block time is more desirable than six seconds but then this is also the question of read versus
59:33
right um it's okay to have slightly slower rights because we're doing a cryptographic
59:39
operation um i think um it's the read performance
59:44
that really concerns me the most i think majority of the time i think it's fair there's there's pros
59:50
and cons and just kind of wildly just saying hey we want to reduce it to for the sake of user experience there's some
59:57
there's some trade-offs with that right and maybe the the important part is that this doesn't
1:00:02
come for free right like so shane we just i was just watching um shane on stargaze because somebody um
1:00:10
he asked a question because he saw what we were talking about before with stake like mo uh had their issue with the double sign
1:00:17
and shane asked that did at mo switch to two second block times like yes they've been to two second block times for a few
1:00:23
weeks and shane just said hey like he said tenderness tendermint was designed for this range would be interesting to
1:00:29
to experiment with faster block times on test net within stargaze um and i i get that like i mean because the the
1:00:36
ultimate piece is that you want a better user experience and if that if that finality comes faster right especially for something like a mint or this or
1:00:42
that or the upside things that things happen instantly you don't realize it's a blockchain it's just just a great user experience right um but
1:00:49
difficult to work backwards from that and it's difficult to be able to have 150 validators with that i think i think
1:00:54
validator size also has a big impact on that right um and then the hardware increases and
1:00:59
all these other types of things come with it so they i'm not sure where that balance will work over time but it's not free
1:01:05
so you just touched on something else you set up there which has come up as a question which is about
1:01:12
um so there was a this this was in the couple of days i was off this is definitely something that completely
1:01:17
passed me by was that there was a text prop anonymously submitted on stargaze
1:01:23
for increasing validator size evaluated set size which passed
1:01:29
um and so that's now coming up um on stargaze and there's also a similar
1:01:35
there is a commonwealth prop on juno for the for the same thing so
1:01:42
uh this kind of seems like it ties kind of in a little bit to what you were talking about just then yeah in stargaze actually that text prop
1:01:49
just got changed to a real parameter change prop just today so there's an open prop right now on
1:01:55
stargaze to go to 120 120. that looks like so far that it's passing
1:02:02
but we'll see kind of what happens um but that's that yeah i think
1:02:08
and again i i mean there's a different piece of this which is does the actual validator said
1:02:13
increasing actually do anything for decentralization which i think we've all kind of hammered the drum a little bit
1:02:18
and the pass and says it really doesn't it really doesn't um in the grand scheme
1:02:24
of things but but again that but you also want to allow i mean it's not a bad thing you want to
1:02:30
allow new validators and you want to be able to lower the barrier of entry to be able to get in for independent values like
1:02:36
myself to be able to get into different chains right um at the same point there's church trade-offs to that as
1:02:41
well yeah well i mean like we were all for an example like we were all late to
1:02:47
the party right so um the atom or cosmos blockchain was well
1:02:53
established when we came along and for a lot of um us guys i haven't tried but
1:03:00
for a lot of the other guys that run in our circle um they would never have had the opportunity to try to get into
1:03:06
um adam if uh if you know the the set wasn't expanding
1:03:12
um so yeah i've got i've got mixed arguments
1:03:18
all over the thing um and it's hard to land on you know what's the right answer there is no
1:03:24
right answer there is you know what you come to a consensus as a group of people it's the right answer
1:03:30
and there's pros and cons to everything and i suppose you just have to weigh them up um
1:03:35
so at the moment i'm sort of in the in the um you know in on the side of the fence to
1:03:43
not be expanding sets at this time mainly because
1:03:49
they're not really economically viable down the bottom so unless maybe evmos might be but most of
1:03:56
the other chains uh if you're in the like below the bottom 50 or 75 like you're running at a loss
1:04:04
and it does nothing for centralization that i've seen i've seen these expansions many many times and the
1:04:11
the power still all sits at the top people don't redelegate it's not gonna fix decentralization at all but it does
1:04:19
give an entry point to two new validators right and that also comes
1:04:24
with its pros and it's cons so if you're a wasm chain like juno
1:04:30
um which i was trying to argue on twitter probably unsuccessfully the other day
1:04:35
has really low load at the moment it is basically you know minting a lot of empty blocks
1:04:41
in like 1 to 10 transaction blocks right which it's capable of much more but as load
1:04:48
increases on the network you know it starts to impact
1:04:53
the nodes that aren't fully equipped to handle that load and it's also worth saying as as well with that that when a
1:05:00
big contract comes along gets uploaded it gets instantiated whatever
1:05:06
you know this is why we're all running 64 gig and more
1:05:11
on on these on these nodes is because actually juno is and cosmism is very very
1:05:18
heavyweight when it has to do the serious operations that make the party happen
1:05:23
and [Music] like you were saying now if that usage goes up substantially
1:05:31
then that is going to pose a problem um depending on the size of the set and the
1:05:36
hardware used right but it's an unknown at the moment you don't know how many people are running good hardware you
1:05:41
don't know how many people are running [ __ ] hardware um you know it's fair to say that the
1:05:47
people uh well some people do some people don't it's and it's it you it won't be evident
1:05:52
until the load starts to increase on the network and then you know as people realize they need to
1:05:57
increase their resources right you're going to end up with a situation where it is just way too cost prohibitive for people
1:06:05
a certain way down the the line like you know and then you're going to end up with a set that doesn't have all the all
1:06:11
the slots filled people are just going to drop out and then people have delegated those people they'll be people
1:06:16
who there'll be permanent delegations that get stuck outside of the set because they've jailed themselves or
1:06:22
unbonded because they can't afford to run the hardware and then you've got all this delegation sitting on those because
1:06:27
a lot of people just delegate and forget and never come back or when they do come back they collect and go [ __ ] i've been
1:06:33
in jail for a month and not had any reward so you know we haven't really reached that point on any network yet where
1:06:40
um you know we're experiencing like high cost of hardware to run with um you know
1:06:47
smaller return because you know most workloads on most networks at the moment are pretty light
1:06:54
because of the the volume of transactions i don't think any there's not really any networks out there at the
1:06:59
moment that are fully utilized or even nearly fully utilized or 10 percent utilized and
1:07:06
you know a lot of the load obviously is just from people restaking it's just normal transactions which are
1:07:12
like virtually nothing so yeah yeah so the question is though in
1:07:17
that situation like if the the changes will be uh sdk level to
1:07:25
improve performance right and then at that point you're in more a position where
1:07:33
usage has gone up improvements have been made to speed and then it's kind of more natural to go
1:07:38
well we have this extra performance we have this extra need for hardware and additionally we can now expand the set
1:07:47
for but it's but it's also back to there is a little element of other than bringing the validators into the set what the
1:07:54
purpose of going up from arbitrary number to arbitrary number is yeah
1:07:59
again like the floor's always going to get expensive and there's always going to be like the push to expand it again
1:08:06
and where do you stop like do you want a 500 validator set just to
1:08:12
keep it cheap for the people coming in at the bottom like that's the advantage of being there at the start you get to be on the
1:08:18
network you get to build um you know community around your nodes and
1:08:24
yeah i just i don't see the point of ever expanding the set to be inclusive
1:08:30
um as good for you know the network performance and the and the network as a whole
1:08:36
like if there's a good reason for it other than that i would love to hear it um but uh at the moment it just seems like we
1:08:43
just keep expanding these sets just to keep it cheap for people to come on yeah it's kind of like dealing with fomo
1:08:49
isn't that it's like sometimes sometimes you miss a thing and that's fine go find the next thing because the next thing
1:08:54
can be just like like you pointed out now we were all late to cosmos in terms of the very
1:09:01
early change of the launch like cosmos regen you know akash
1:09:07
exactly and yeah we all joined stargazers and juno and these other projects that have come along and we've
1:09:12
found a way of making that work for our operations and those opportunities are
1:09:17
out there you know when when we first got involved in cosmos i think all of us got involved before the
1:09:23
last bull run so again it was looking at the position of going you know certainly i remember doing the sums of oh well if the price
1:09:31
on stargaze is particularly good we can probably afford an aws rig rather than running on hetzner which will help my
1:09:37
blood pressure you know um so it's it's also about expectations i think we
1:09:44
just come out of this big bull cycle and there's a lot of interest in doing this
1:09:49
from either people who are new to it or people who have got a background in it whatever it doesn't really matter
1:09:56
but there's there's a there is that kind of that it's like kind of fomo thing maybe that entitlement thing that kind
1:10:01
of rubs me up a little bit the wrong way because i think there's this we're in such a big tent
1:10:07
and it's like we're sat around this table we're drinking a cup of beers and it's like dude there's literally a table there
1:10:14
with free beer on it and nobody is touching it nobody's touching it because nobody has yet sat at that table
1:10:20
you don't need me to share a stool with you there is an entire table and i can even i will be glad to walk with you
1:10:27
over to that table and show you how to pick up a beer um
1:10:32
we were just talking about how many test nets there are right at this moment and these are all promising networks
1:10:38
that are coming out like some are incentivized yeah some are incentivized to get you like starting bag and that's i mean that
1:10:46
was the advantage of um you know it's the stargaze uh test nets right we were there
1:10:52
from well i wasn't there from the very beginning but you know some of us were there from the beginning and then that
1:10:57
gave us an opportunity through helping them develop the network to be able to get rewarded and build a bag for the for
1:11:03
the launch and have a good you know long-term perspective on that network so um same for for many networks you know
1:11:11
you get there you work with them while they're developing it and then they give they reward you for that and
1:11:17
then that's how you get your way into that network and secure your your place in the vowel set
1:11:22
um but just like you know i don't know people it seems like people just
1:11:29
don't want to think about other consequences and only just wanted to create an entry point at a cheaper
1:11:35
price it's not just like i'm not talking about it's not just new validators like it's
1:11:42
you know this sometimes comes as well from the people who are you know managing like the
1:11:48
the foundations or the people are managing it they want to be able to create that cheap entry point for people
1:11:53
as well so well they want to create opportunity right or at least the idea that there's opportunity there
1:11:59
this might this is the other issue with me which is this goes this is the right conversation it goes directly in the
1:12:05
face of like the chain security thing from from the cosmos hub which is a new
1:12:11
chains don't need to set up validator sets let's just use what's in atom and they can use that from a governance perspective
1:12:17
and this is the validator set which i think i don't know but i'm not in the atom space so maybe i'm biased within that
1:12:23
but my idea is that why are those the selected ones like there are new chains there are new validators and kevin just
1:12:29
pointed out kevin garrison said new values bring new development new viewpoints and new innovation it's exactly right so why am i why are we
1:12:34
tied to the same 125 or 50 validators that have been there since the beginning just because
1:12:39
is that the is that the limit of the of the innovation is that the limit of the new ideas um is that the governance viewpoint that
1:12:46
we want on all chains that choose to use that validator set and i think that's wrong like i think that
1:12:52
change will change will grow and change will die um including including the atom like
1:12:57
there's nothing that says that that's the going to be the center of the universe forever um and so as as that
1:13:03
naturally grows why why why why would new change be tied to that
1:13:09
same structure um and i think that this goes against the whole idea
1:13:14
of um of what we're trying to build so that's why i've been always kind of negative around that idea is because it just
1:13:21
seems like it's you know we're kind of limiting ourselves to say this is the structure that we want to be able to work with or
1:13:26
the viewpoints that have already been set i think layered security there is the kind of answer to that like there is
1:13:31
there is some there is a good idea in there somewhere and actually moreover than layered security i suppose
1:13:38
there's a future world in which it becomes even more arbitrary and you can actually bring in
1:13:44
additional validators from outside either of the sets that you're layering into it which
1:13:50
is an interesting theoretical idea for expanding a validator set but i guess we're always going to run to
1:13:56
the problem there is fundamentally there's a difference between sdk only
1:14:03
if you like networks and cosmosome networks or even you know evm networks
1:14:10
where there is an additional substrate on top that has additional performance implications that has additional
1:14:17
configuration implications that has all these these different variables that are going to
1:14:22
affect end users and affect the performance of the chain in a way that is outside of the normal functioning of
1:14:27
the sdk and that's what's kind of governed by ics consensus really
1:14:34
so yeah i there was another question in here let me see if i can find
1:14:41
it to pick up um on again on iconic validator stuff like
1:14:46
should um should somebody asked on twitter i'm sorry i forgot to write down the name should self state be a factor
1:14:54
in formulating the stake a given validate has in a chain which
1:15:01
th this is all back to kind of the ranking of validates isn't it but i thought it i was just reminded of it because obviously if you do an
1:15:06
incentivized test net as a validated team you're quite often given um a delegation to self-stake
1:15:13
um [Music] when the network starts right but you if a validator joining the set later
1:15:19
um might have a much much smaller self-stake that they've actually bought themselves um and to kevin's to kevin's
1:15:27
point in the chat that would to me indicate a much higher belief in the network potentially than somebody who
1:15:34
arrived early and said i also have a high belief in this network to give my time and energy to it
1:15:40
but you know a year down the line two years down like three years down like that looks like a
1:15:46
disproportional they probably have the same level of belief in the network but it's
1:15:53
it's all it's timing right you know um none of us are any smarter than anybody else just some people are
1:15:58
first right yeah like being being early has its advantages right
1:16:04
definitely no matter where you are being early is like a big advantage um
1:16:09
you know when when networks launch it is a race to try and get in that top area
1:16:16
because we know whether we like it or not we know that that's the easy delegation area once
1:16:22
you're there people just delegate to you without thought so that's as much as we hate to
1:16:28
think about that um or admit that that is like whenever a new network goes live
1:16:35
that's the scramble right and that's no lie
1:16:40
and that's every single validator that's not like you know people might say they've got good ideas
1:16:46
and all this type of crap and talk about decentralization and all that type of [ __ ] but when a network launches that is the
1:16:53
scramble from every single validator on that network is to try and get like a good position
1:16:58
to get the easier delegation as time goes by [Music]
1:17:04
jeremy irons the actor has if you guys have seen the movie margin call which is about 2008 u.s
1:17:10
housing crisis and and really the the fall of um of loan structures that was created
1:17:16
around kind of mortgage bonds is really what the issue was jeremy has a great speech in that movie the movie's awesome everybody should even if you're not in
1:17:22
us it's just a great movie to watch he says there's three ways to win this business is that he's either to be first be smarter or cheat and he says that we
1:17:29
don't cheat which there's a lot of values to cheat because you kind of see that we've talked about that here you could be smarter there's a lot of
1:17:34
validators that are really smart around how they do operations he's like it's a it's a hell of a lot easier to be first
1:17:40
and that i think that that that rule applies here as well right it's it's there's a lot of benefit to be early in
1:17:46
sets and and be able to get involved in those test nets and and those types of things and there's other aspects of that as well but i mean those those three
1:17:52
things are still pretty pretty damn appropriate for this ecosystem as well as that financial ecosystem
1:17:58
and you're providing a service in a test net too right you're not just getting oh sure like
1:18:04
it sometimes they take a very long time sometimes they take a lot of labor but they're providing a service for
1:18:11
a return and your return is um your initial stake in a lot of cases
1:18:17
and your initial position in in the chain so yeah absolutely being first has
1:18:23
its advantages and i think it's in a lot of cases like the biggest advantage except you know
1:18:30
well we say when and when um validators cheat but that's in our opinion they're cheating because of
1:18:35
our ideals of what validators should um the rules they should adhere to which is our
1:18:41
arbitrary set of rules there's no hard hard and fast rule just like what we think is socially acceptable and then
1:18:48
you know not everyone's going to [ __ ] care but then there's like you know some
1:18:54
people just have good business models or or well thought out business models to be able to attract that delegation and
1:18:59
work that into um you know how they how they run their
1:19:04
ecosystem as a whole which you know ask any of us we might think that's kickbacks but ask anyone else they might
1:19:10
think that's a valid thing and that might lead to concentration of power but everything's
1:19:16
got its nuances and everything's got its like you know pros and cons and and it's not you know
1:19:23
it's not necessarily cheating it's cheating in their opinions i i think jeremy was talking the same
1:19:28
way right like i think that that example wasn't necessarily illegal cheating it was doing things that they felt that
1:19:34
they were moral like they felt in their moral structure and i think you know that's why we talk about the stuff as
1:19:40
well like there's no spoilers but like from memory that point in the film is like very very specifically before they
1:19:46
do something that is gray but they are first on that's correct yeah they saw all these contracts and
1:19:52
like basically the rest of the market falls but they are able to survive right but again they're first
1:19:58
so yeah yep um so juno.love has uh has just posted in the
1:20:05
chat and just said uh question about self stake was about getting to the fact that there are a few values top ten less than a dozen
1:20:12
juniors or state um i think that's probably well so obviously there are validators that
1:20:17
actually don't have a very a very large amount of self-stake but that's very very uncommon
1:20:23
um purely because you'd fall out of the set or radically change position in the set if
1:20:29
your delegator structure changes um so i think probably what's going on there is just that people are delegated with
1:20:35
different wallets with alternative wallets i think it's probably the case that any almost any validator certainly all of us
1:20:42
on this call and most of our list we know probably
1:20:48
their self-staked percentage is not reflective of actually how much they are self-staked
1:20:54
i yeah i don't know i'm going to say the same thing we're gonna say the same thing but that that idea that there might be
1:21:00
multiple wallets that our own or all well i mean we do the same thing like we try to keep a decent amount on the
1:21:05
validator wallet um because they're all ledgers you know i got a stack you know look at this like you know we have stack
1:21:11
of ledgers that are sitting around that are associated to those validators there you go bag
1:21:16
jingly bag bag of ledges are in the back just like going [ __ ] man why are they
1:21:22
down they're gonna pull you know i i honestly once said they were like
1:21:27
you just keep it a sock bite that's no dj
1:21:33
i might have to sit on the shelf so that's that's one step up from mine i just get lost um but uh but whatever how are you gonna
1:21:39
control that but but the other thing is that we have like personal stuff that we stick to that as well i it would be
1:21:44
interesting to see the difference between that and there's a difference between that
1:21:50
and when those rewards and commissions come back and they're just ibc'd off that that's a little bit different like
1:21:56
like we do we do to our ten percent to this right now gives thing that we're starting as well as you know dude i gotta give it's
1:22:02
my news in my space and our rule is that we we ten percent off all the commissions and then
1:22:07
whatever's left we stake half back to the validator and then the other half we take as
1:22:13
maybe as cash or at least it comes into a core account that we can decide what we want to do with it because we need to run ops and so it's like 10 to
1:22:20
commission 50 whatever's left then 10 50 to the validator because we're giving back the chain and 50 back to us
1:22:28
um but that's different than i think what what everyone ever seen is that it's just ibc'd and once it goes in that
1:22:35
structure you're not really sure what the hell is going on right um and if that's actually i'd be seeing out to to
1:22:41
be staked back in which doesn't really make any sense or what's happening with that so there's a i think there's a rama's on
1:22:47
the chat as well because it is one of his areas is to be able to
1:22:52
um try to give more information around what validator operations are doing and and who's being kind of honest in that
1:22:58
structure um and so i think i think there's a it's all over
1:23:03
the board i think i don't even know i love this how do you know if validators uses a ledge or as raw
1:23:10
dogging nature [Laughter]
1:23:15
um so like i just looked at my validator for juno right so
1:23:20
we are like ourselves staked um
1:23:26
you know on the validator wallet is like two percent right but our total sales take is probably
1:23:33
closer to like 10 um so we definitely have
1:23:38
uh token elsewhere and that's some of it's for tax reasons some of it's for like
1:23:43
you know securities some of it's for there's various reasons to have different wallets but i personally think that
1:23:50
that you know like the corporate validators like you know the ones who tend to have very
1:23:56
low self stake with larger wallets elsewhere should make more of an effort to like have a higher self stake it's not like
1:24:02
they can't afford it it just doesn't it's i think it's more of like a security policy because of the amount of
1:24:07
hands on particular keys or something like that right but um you know
1:24:14
they're more of a corporate structure whereas we're more of like you know we've got our own ideals and
1:24:19
it's worth it's worth just it's worth just pointing out at this point that like you know in the independent in the
1:24:25
independent validator teams the largest ones are usually two to three people and some of those larger
1:24:32
um operators who are also running on eth or maybe some of the you know like near solana's and there's really big ticket
1:24:37
big money situations you know they've got like 50 or 100 staff so it you know they are the size
1:24:44
of technology company that they have a full-time security person on staff who
1:24:50
is like you know you can't ssh onto a box without jumping through
1:24:57
to bastion hosts and we have a key policy and you can't just spin up a server if you want to do some testing
1:25:03
and all that kind of corporate stuff that you would expect of a company that's gone hr department
1:25:10
so there is also that uh that's worth pointing out um so some of it is
1:25:15
probably less yeah like null says it's less about this and that it's more about just what you see
1:25:21
of of how the sausage is made because they're much bigger organizations so no what you're saying is that some
1:25:27
validators are more multi-sign and some are more loud bunch of ledger sitting in a sock
1:25:32
yeah okay [Laughter]
1:25:38
some people like have systems
1:25:44
i mean it's a it's not a zip it's not a sock it's like a little it's and you'll just tighten the top of it i
1:25:50
trick people by writing uh memory on there
1:25:56
so like oh that's not a bag full of ledges that's like is that that's literally your security
1:26:01
policy my mine is mine is literally like a you
1:26:07
know bang for all of the stuff mine's misdirection this is directly
1:26:12
paranoid memory as soon as the paranoia gets you though and really secure toenails
1:26:20
bag of coins uh too funny um how are we going on our
1:26:25
list guys i think uh i think we're pretty much uh squared away there was just uh
1:26:30
what are we excited about in the upcoming week and i believe somebody possibly usually put green candles well
1:26:36
it was until i put it in the spreadsheet and with the [ __ ] so it's the last time i write green candles on anything
1:26:42
it's gone bad well it today has been bad the last hour's been okay today has been dropping but the week has been pretty
1:26:49
good we're starting to see a little bit of turnaround and at least some not so
1:26:54
much for e-money and evmos but everything else that at least i'm tracking within the cosmos is is has
1:27:00
some good uplift uh so is this the pump we've been looking for is this like the turn is
1:27:06
this the bottom or is this just another little [ __ ] pump before it keeps going down
1:27:12
it's about a fifty percent at the moment right that's a very impressive when you look at earth it's about 50 eth yeah he said 1500
1:27:19
bucks us and bitcoin's finally above 20 grand it's at 23 000 something
1:27:25
um so i mean i'm guessing as it goes up it's going to do like its usual humps right and hopefully
1:27:31
those dips don't drop down to where it was but i think in all these situations i've been surprised on
1:27:37
just how far some of these things will drop i'm glad i don't try to chart i don't
1:27:42
want that kind of stress i have it above my head i i need to get rid of this like it's it's miserable just literally
1:27:48
looking at it all day it's a it's a bad like when i look you see me looking up on this podcast when i look up like this
1:27:54
i'm looking at prices that's not good that's not that's not it's not helping when you look up i can just see like
1:28:00
yeah like [ __ ] emoji exactly because i see the world in emojis now i
1:28:07
was talking to my girlfriend the other day on my phone and it was on a platform that didn't support
1:28:13
like the uh the the semicolon uh
1:28:18
you know shortcut oh yeah like a discord yeah yeah and
1:28:23
i just kept typing all these things in between semicolons she's like what the [ __ ] are you doing
1:28:29
and i'm like they're emojis it's like you're just putting words between semicolons yeah man
1:28:36
[ __ ] it's like why do you type emojis
1:28:42
or any messaging app that doesn't support marked a simplified markdown right i think to make it to make it
1:28:47
italic or something and then people are like
1:28:52
why did you do that rama's just called me out for having a girlfriend oh girlfriend big relationship heather
1:28:59
show up ever i mean name dropping skills it's pretty lame of
1:29:06
you to have a life null so you know there is that i try to avoid it religiously by living in my dungeons