And the Validator Commons meeting used to be right now anyway, right?
So it's not like he's being super indisposed.
Welcome to Game of Nodes, a weekly podcast from independent Validator teams.
Hello, welcome to Game of Nodes, a weekly podcast where we work out
whether we've been rugged by our guests.
This week we are waiting for Dan, who's been on the show
before in a previous, in a previous guy's in a previous life,
is currently heading up Validator ops for Alio,
place bets now, whether or not we get rugged.
Alio or Aleo?
Do we just say?
I say no.
We're not getting rugged.
Dan's always like, he's very, you know...
He's kind of an organised man.
He holds his reputation in high regard.
Yeah, he'll show up.
I thought Aleo...
I think...
I think Aleo.
Aleo?
Aleo, yeah.
Fucking every time.
Wait, how do you say that?
But Aleo is quite good.
Like, that's a funny phrase.
So maybe we should just rhyme with Aleo.
Aleo.
It works again.
I have to stop calling their networks obscure shit or like start,
you know, having the, you know, enunciation little brackets underneath it.
All the good things have been taken though.
Now people have...
We've got out of the phase where people could even go for famous science fiction,
famous classics, famous theological references.
You know, they've already worked their way through all the beaches in California or whatever.
I mean, there's...
What else is there, right?
There's...
We're onto like the fucking Zed tier.
We're like, we're not even at rubbish Robert Henline anymore.
We've got no names left that we can use.
We haven't even hit like Alpha, Beta, Gamma.
Like we're not even at the Greek characters yet.
Come on, get out of here.
There's so much left.
It's a good time.
Let's call your chain.
Just start naming chains after like hurricanes.
All right. So this is the Katrina chain.
You know what bullshit?
Actually, I'm calling bullshit on that.
There's almost certainly an alpha chain and I'm going to go and Google it.
There has to be.
Probably is an alpha chain.
There has to be.
So shit has been like kicking off in the last couple of weeks.
I know last hour that with the dump, the 10% dump.
Yeah.
I think brush that shit off.
They're like, we go up 10%.
Last out last hour is coming back though.
Things are clawing back since gaming notes.
What was the what was the catalyst for that?
Did they not approve the fucking ETF again?
Well, the only thing I saw was some some bullshit reporting outlets said that
there was 100% chance of it being rejected.
And then that same reporting outlet also had articles this last month about how
it was going to be 100% approved and and how Bitcoin was going to hit $50,000 or
something like that.
So it just seemed like a bunch of bullshit.
Honestly, what least what I saw was being reported and nobody knows.
The S&P 500 went down like 1% as well today.
So I would honestly, I'm simple.
If it dumps, it's probably just because it traced S&P.
Could be.
Yeah.
I mean, a lot of things are dumped.
I mean, I think stuff that we watch Aptos and then Dydx is down 10, 12%.
You know, Cosmos is down nine.
So I don't know.
It was it was pretty ugly there for a little bit, but things are starting to
claw back a little bit today.
That is the last couple of hours.
Say is up 91% in the last seven days.
They are just about to cross the two Billy market cap.
What do you think that is that?
What's driving that?
Do you know?
It's not that it's it.
I did just buy an FT on say like two days ago.
Something off say me.
I don't know where I bought it.
I should see if I can find it.
One really good say doubles the was I do double the usage.
They active users.
That's not nice.
Might be really not sure.
I think it's the narrative around the around the V2 and marketing and
market makers and the actual it seems like they are starting to get some
users in there too.
I don't know what they're what they're doing to get them.
It seems like that is not like in my Twitter.
It's more the validated shit.
So I don't really see much.
What they're doing.
It seems pretty stagnant until the inscriptions happened.
And I think that say doing really well on inscriptions kind of brought a
little bit of attention to it because say was one of the better networks that I
experienced doing it near was the best.
They basically let me be clear.
Whatever.
No, I was going to talk up near the up to I can and then say it great as well.
Except the price.
That always sucks.
Yeah.
I never really had that much problem with the price feeder, but definitely
like the network crushed it in terms of throughput.
I am having now a lot of problems with some of my nodes, but that's probably
a null thing.
No, today, today I had like I had huge issues in the EU.
I had a restake thinking like three out of four nodes this afternoon.
They're all they're all lagging back and then I started dumping things.
And also this upgrade actually made a huge difference.
And the upgrade was really weird.
I had like the traffic that I see at RPC is totally different post upgrade.
So I don't know what change.
Yeah.
It's like it's before we had a lot of like very, uh, like very consistent
jumps in traffic between like 500 to 1200 requests a second.
And for some reason now it totally flattened out.
So I don't know what the post upgrade, but it's really weird to see different
traffic pattern.
Oh, look, I, it's gotten a, I just got an alert for a, for a say Western EU issue
right now.
How exciting.
So I, my archive currently in the process of moving my archive, but it's
performance has not been great in the last couple of days.
It reset like four times overnight.
So I needed to do a bit of investigation on it in the morning.
Currently 7,500 blocks in the lag, which is a pain.
And so yeah, but not a hundred percent sure what that is.
I'm hoping that the upgrade sort of fixes it.
So this is, this isn't your, this isn't your archive.
That's, you know, physical tin in a data center.
No, this is the one which is very soon going to be moved to the physical tin.
I, uh, I nuked my tin.
I, uh, I'm new to, um, micro tick routers and, uh, I made a configuration
change on a page that I shouldn't have done it on.
So I have since found out that micro tick routers have this quick
microtech page.
Microtech.
Yeah.
Whatever.
Yeah.
I agree.
I need some micro tech.
You're like, you're like that chef that said, micro of a, instead of
microwave.
Wow.
I like to put emphasis on the wrong syllable.
Huh?
Anyway, what was I saying?
Oh yeah.
So there's this, the, the, the person who built my server, right?
They set the fucking IPMI or the management port as a static address.
And when we pre-configured the, um, micro tick router, uh, we had it all
to take DHCP, um, and then assign it to certain things on the public address.
Right.
And so it was expected that it would be DHCP because they didn't ask me for a,
you know, network schema configuration.
They just like set it to this random fucking.
Anyway, so I had to go in there and add some addresses, uh, to the router to
get it to talk to that local address.
And when I was adding them, um, there's a quick set page.
And I thought that when you add stuff in the quick set page, it just
adds that thing to the configuration, not completely nukes everything and adds that thing.
So I just completely nuked my access to that router.
So I'm flying back to Melbourne tomorrow to fix that.
You have remote hands or can you have somebody help with that?
Yeah, yeah, but I'd just rather do it myself because I anticipate that I'm
going to have other issues as well.
I don't really want to be like back and forthing with remote hands.
So wait, how much is it going to cost you to fly back to Melbourne?
A couple hundred bucks.
Some kangabugs.
Yeah, it's pretty cheap.
There's a flight that goes right from where I am to Melbourne direct.
So, um, yeah, it's, it's pretty cheap.
It costs more in the Uber to go from where I live down there to the
data center than what it cost to fly from here to fucking Melbourne.
Well, I mean, like it's, I feel like we've already tried to have an intervention,
but you just really want to be there.
You want to be there in your parents' basement, cutting.
Well, they don't live in, what are you talking about?
So my partner lives in Melbourne.
No, no, I'm not talking about living, I took a little problem with Melbourne.
That was a nice place.
I mean, I mean physical 10.
You kind of just self-harming.
We're just kind of having to watch it.
There becomes a point.
There's a, there's like, if you, you can graph the, the scale of like economy.
And there becomes a point in Australia where it's just better to get hosed.
With hosting and internet fees than what it is to rent metal with enormous hard
drives.
So that, like we've got, we've got a lot of space on this new machine.
And to rent that type of space, I'd just be getting hosed like every month.
I would never be able to recover the cost.
So.
I just, like, I don't know, there's got to be an easier way to, like, I know we
will run infrastructure companies, but like the, the, the goal of any scalability
question, right, is to find out the minimum amount of time of hours plus money to,
you know, like it feels like any situation, like, I know it's an interesting,
like it feels like, it feels like you're just playing on hard mode for just
because you're stubborn, like, like, I can't prove it because it's really hard
to prove, but, but it's just like an intuition that we're in the Jurassic Park
looking glass here where, where just, it's like, just because you, you, you could,
you know, just because you're signed to your infrastructure engineers could
didn't, didn't, didn't mean they should.
Look, man, we're blasting through the, the learning curve here.
Right.
I want to run metal for certain things and I want to run out of Australia.
It's just that I cannot fucking rent anything in Australia because there's so
uneconomical.
It's not funny.
Like the, the absolutely fucking ridiculous costs they ask you to rent
something is just outrageous.
It's worse than bloody soul.
Oh, I was surprised by how expensive it is to rent hardware in Asia Pacific,
just in general.
Like the fact that generally the internet though, the internet costs,
they're insane compared to the sense to me, right?
Because like my uncle lives in Japan and Hokkaido, like, in the US,
in the middle of nowhere, Japan, very North.
And he has had gigabit internet for like $3 a month, like the last decade.
So why are we paying so much?
Well, it's, it's not that it's, it's not that it's not available.
It's just that the congestion, like when we're talking about these data
centers, they're like in Tokyo or in Sapporo or whatever.
And the congestion in those areas, like it's really expensive property and
space and electricity to be able to put the shit.
And then like, you can't just go and build new, like trunk mains of fiber.
Like you have to use the existing.
It's just, there's no way to put it in the ground.
And so we're at a kind of more unstructured version of like an
economics explained YouTube video now.
Cause I mean, you watch economics explained another fellow Aussie.
Yeah.
It's reasonable.
Right.
Yeah.
It's pretty good if you just like, you know, making some pasta
bakes or something, you know, batch cooking or something like that.
And you're just like, I kind of want something that makes me feel like
I'm learning something, but I also want it to be semi Larry.
Oh, an Australian doing economics podcasts.
Let's go.
So our best YouTube expert, our experts economics explained the
Batuta news and the Batuta news.
And the, um, we should, why did we name this?
That's, that's, that's up in our name.
And the honest government ads and they've started doing like other
countries now, like Europe and shit.
Absolutely.
Absolutely not.
Vati video is Australia's number one export for YouTube.
No, no, no, no.
Drain addicts.
Vati video.
They do like, they do like dad souls videos.
Drain addict is a great voice.
Drain addict is Australia's number one YouTube export.
Back in the day, I would have said it was, um, uh, let's drown out maybe.
Cause that was like one of the OG let's plays.
And it was an Australian Gabe and then Yatsi from a zero punctuation,
just complaining about things while playing half life for like 55 hours.
It was absolutely classic.
If you just needed something on in the background while you were working.
And it was kind of, it was kind of like office noise cause it was just
two nerds complaining.
And so you're like, okay, I'm working from home.
This is kind of, this kind of checks out as like constant people complaining
about the different versions of half life and how they're stuck on a certain bit
or whatever.
That was pretty good.
Anyway, hi Daniel.
How's it going?
Good day, Dan.
I'm just listening to this.
It's interesting.
It's the, I don't know how we, I don't know how we sort of fell off.
Yeah.
Those commercials where they like saw people like dying on the train.
If they like, you step into it and it's like,
is that something completely different?
What?
There's like those commercials where it's like, if you don't obey the law,
then you're going to get hit by a train and die.
And it's like, it was like cute animation where they get sliced up like ham.
No.
I feel like that's Nord Australia.
Is that the Bosnian Ape Society YouTube where it's like, this is how you
defend your family, your family, your family sized car from an advanced
Russian style fighter.
I, I.
Maybe it's New Zealand.
It's all animated, but it's very surreal comedy.
I'll try to find it, but.
Okay.
We're into the, we're into the Googling, Googling for YouTube links section of the podcast.
I actually do have a brief, but I know we do actually have news, obviously,
and all those sorts of things.
But.
Before we get to the news, go on with your thing and then I'll do my thing.
Okay.
For those of you based in the UK, it is totally tacky.
Okay. It is totally tax time.
As the ticker suggests, remember you got till the 31st of Jan to put all of your
degeneracy into a spreadsheet and give it to old uncle tax.
Otherwise, you, you're going to the big house.
So that's, that's the end of that one.
We've got a fun thing related to tax time.
It's fucking tax time at the start of the year.
That is inconvenient.
Not only is it the filing deadline, but the, the filing and payment deadline are,
well, okay, it's different for companies and people.
It's a little bit, but, but the deadline is kind of the same, right?
31st of Jan, you've got to have your filing done and you've got to make
either your full payment or your first payment, depending on whether you're a
company, an individual or somebody who pays the, the one way you have to pay a
little bit in advance, but then you pay less later.
You see everybody in the comments just started saying drink.
Is Dan talked?
Yeah.
No, because we started talking tax within the first 20
minutes.
Um, anyway, so on governance news, I just saw this morning and she'll see and
serve, you probably also saw this.
Did you see the D Y DX proposal?
I saw that you posted it.
So this is a.
Unjail some validators during the D Y DX emergency that happened on Thanksgiving.
I think it was.
It is not on jail.
It is even better than that.
It's exactly.
It's purely a reputational rewrite for no fucking reason other than reputation.
They were slash though, weren't they?
It doesn't say anything about the slash.
It's set.
No, yeah.
They want to get rid of the record in the state because of marketing.
It says because it fucking competition.
Yeah, they accept the mistake and they accept the slash, but they want the
marketing info to be rewritten.
Ah, but there was no slash.
So if there was a slash, they probably pushed the point because then there's
actual consequences.
Now there's no consequences of the marketing.
So they want to get rid of that consequence because that's small.
Right.
So we just want to get this right.
The thing we do.
I mean, this kind of makes valid.
This is kind of what people already think validators are kind of like, but
these guys are like saying the quiet bit out loud.
So, um, Rama asked if I asked this from Kooji and I asked him,
if I asked this from Kooji and no, I didn't ask for it to be removed from the
state.
I asked for the fucking the, the website thing.
Wait, hang on a second.
Hang on a second.
Do we need to get out of the hypocrisy, Clarkson?
No, I firstly, I haven't told you my opinion yet on the dydx thing.
Secondly.
Yes.
Secondly, yes, I did ask them if they could time out the thing after a certain
amount of time.
So, you know, take your comeuppance for a year or six months or whatever,
and then have it disappear from the website, not from the blockchain state.
And the reason for that is, is because sure, you made a mistake, but, you know,
having that thing there, front and center on that website, guarantees you to
lose all of your delegations.
So the other option is, is just we unbond now and rip the bandaid off because
we've not been making money on that chain for now for a year and a half.
So.
What about air drops?
Have you claimed your airdrop?
Yeah, I'd say with air drops were probably just about break even.
If you don't count my time as anything.
I don't.
I don't because the, the, like the, just the, the price feeder alone on
Koogee is like expensive in terms of time.
Yeah, it is.
It's quite expensive.
Yeah.
But I mean, no one at all values validators time.
So.
I mean, not even validators, which is half the fucking problem, isn't it?
Like we're all just like doing the math of like, oh, well, does this make sense?
Does this make sense?
It's like, oh, this one's, you know, bringing your money in the servers are
costing less than what it's bringing in.
So that's all profit, right?
Yeah, it's, it's hilarious.
They're like, why am I so fucking tired all the time?
Why have I not slept at all week?
So, um, on the D.Y.D.X. thing though, I, there's a couple of aspects, right?
They didn't tell people at all, which is, I don't quite understand why they
didn't trust all of their validators enough to tell them.
Usually there's like a security group that you would give the upgrade to.
But in saying that,
just to clarify, you're talking about now the emergency upgrade for the
jailing, not for returning the proto rev funds, right?
So I'm about jailing.
So I'm still talking about the D.Y.D.X. jailing, right?
So the circumstances were for that upgrade is that they told a couple of people who
they believed to like be key for the chain to continue, right?
So obviously the big state guys and they told us, but I think it was because we
run infrastructure for them.
So they didn't want a break in the infrastructure, but it was not very
long ahead of time.
And then they didn't tell anybody.
The people that they told, do you know if they were just partners in like
institutional staking services companies or was it?
They were validators with a lot of staking.
It was out of stake, not whether or not they actually.
No, it was purely on will the chain keep running?
Right.
So it wasn't like, oh, this is Big Bulls venture backed staking.
No, it was more like, how can we tell the least amount of people possible and
share the new binary with the least amount of people possible?
So it wasn't about Jimmy Big Bulls and Jimmy Small Bulls.
No, it was about who could we tell, how could we tell the least amount of people
and distribute the binary to the least amount of people, but still have the
chain running is what I can gather.
So what you're saying is they decided to treat their trustless,
decentralized software like it was trustless, decentralized software running
in a low trust environment.
Yeah, I guess so.
It's I believe that the vulnerability was bad that they had.
So they had to like, it was security based, but in any case, I didn't tell.
Not exactly.
It was security based in the sense that I helped find the bug, right?
Which kind of is a little better in that they didn't help.
They didn't actually give us the binary, but we helped find whatever.
Anyway, wait, wait, wait, you helped find the bug and that they were like,
oh, you know what this is, but we're also not giving you the thing.
Yeah.
Yep.
Yeah.
So I was going to go big.
They didn't give you the FDI.
Yeah.
Say I literally ran across Manhattan.
No, no, it's because I was on a phone with them or yeah, we literally got into a
camera going out to dinner.
We're pulling into this place and this thing goes off and I was like, mother,
like I see his missing blocks and everything else.
I'm like, motherfucker.
And like, I could see it there.
And like, so I literally ran across like seven avenues to get back to the hotel
to run it.
Oh my God.
Okay.
But so it's like you smart as actual that I would take my laptop with me.
I brought it back with me.
I wasn't expecting that.
We're looking to be offered.
If I'm usually gone for a couple of hours, I usually don't do that, but just put a
toilet seat down to work with the toilet.
If I just.
SIRP was SIRP sent me this message.
It's like, what the fuck are all these alarms?
And I'm like, there's an upgrade and he's like, ah, what?
Just got in a taxi across time.
And I was like, you don't take your laptop with you everywhere.
No, I take my laptop.
That was only a couple of hours.
I was like, no.
That's what I want to do as well.
And for my mind went off really as we were serving Thanksgiving dinner.
So I was like, oh, shit, I'll be back in like, I don't know, half an hour guys.
And so people.
Yeah.
So they got through pretty quick.
It was bad, but.
Okay.
So they didn't tell anyone, right?
So that's like, that's fine.
They did it for whatever reason.
They didn't tell anyone.
They didn't tell anyone, but the jailings, like, I feel like they should have had
enough coverage and alerting to be able to fix it within three hours, regardless
of whether it's the middle of the night.
When alarm goes off, I wake up and I fucking go to work.
Like that's, you know, that's the validator way.
You don't sleep anymore.
So some of these people have teams, some of them don't.
But in any case, they should have adequate alerting that they're aware of
the issue and can get someone to fix it right away.
So, and particularly like, what is it?
What I think they're, they're a big, they're a pretty big outfit, right?
Yeah.
Who's, who's put the proposal up.
So if they got jailed, I would have expected they would have enough
coverage to be able to.
Yeah.
Take care of that when it happened.
I mean, I think the question there is more like about their own internal
process and whatever.
I mean, I know that we have a lot of our alerts.
And things are set up based on the slashing window or whatever network it is.
That's, that's the reality because I, we, we have very early days in the podcast,
we had Jack on here, didn't we?
We were talking about the slashing window.
And the idea is that in a crisis, if you need to break out and actually get
some fucking sleep, you can, right?
And that's kind of been our, our rule on that.
And there are some networks that are super forgiving of that.
There are some networks that not, but you should kind of plan your alerting
accordingly.
You know, there are, there are networks with no slashing window, but equally
you're losing money from the get go.
The page is going to go off straight away if we start missing blocks on
Aptos or something like that, where that is the case.
But it does beg the question of what the fuck auto stake we're doing with their
rotor.
I mean, I know it's a bear market.
Have they just canned off the, the expensive having people on call all the
time and every time soon is very expensive to do.
Yeah.
I mean, I don't mind last minute upgrades.
I kind of don't appreciate giving the VP heads up and then you have everybody
else underneath that, underneath that waterfall that's just fucked.
Like that to me doesn't, that doesn't really, doesn't seem like the best kind
of method for performing upgrades.
And there's ways to be able to communicate.
It's really in those types of things, but just to say, Hey, we have 67% and
whatever happens to the delegators on these other things.
I think it's just a, that's a, I think that's kind of poor.
Honestly.
Well, we have a, we have a guest here, right?
Who might have an opinion on this.
Like, is there a situation in which, you know, Dan, from your perspective, you're
like, well, no, the, the, the network becomes first as a priority, right?
So we're going to have to have a dig here.
So the network does come first at, at Dan land and they just block all the other validators.
That's true.
They're like, there's problems.
So bugger you people.
We're doing our own thing now.
Yeah.
Slap and climb both mode boys.
Cause you're not a validator anymore.
It's just a guy, this guy's.
Yeah.
I think maybe communication, if, if like, there's an agreement in place where the
type of communication is sort of understood by all parties.
It's that type of responsibility is something that people need to be ready for.
But yeah, I, I don't, I didn't take too deeply into what was going on in D Y DX, besides
like the sort of like the two big validators there.
But yeah, I, I, I was just kind of listening to how you guys were describing some of your
own sort of setups in terms of how you get alerted or if you have teammates in different
parts or different time zones.
But so all of us here would be in that very situation, everyone in this room would have
a setup to be alerted pretty soon.
Like we would all have stall warnings and we would all have missed warnings.
My miss warnings are tiered.
Like if we miss 5%, we'll get like an information in our, in our discord.
But once it goes up a little bit higher, you might get like an average warning.
And then that might be a ping with no noise on your phone.
And then if it goes to like, you know, another certain percent, it would be like a catastrophic
warning and you start getting phone calls and texts and all that type of crap to the on call.
So a crazy thing happened a while ago, which was that I got a new phone because I was this
one, I had my phone stolen or was this before that I got a new phone, set up page duty, set
it up to do the override thing.
You can see where this is going.
And then there was an incident and the phone didn't override, but it just happened to be on
vibrate.
And I used to be a super heavy sleeper.
You could have very stood on me and I would have just kept sleeping.
That has changed with small human.
And a bunch of people sent messages being like, yo, have people, have people noticed this has
happened or whatever.
And I think it was like signal or telegram, whatever, my phone buzzed so much on vibrate
downstairs, an entire floor away that it just like tripped my ear.
I just my ear was like, there's some kind of a buzzing.
Maybe it's a fire.
I don't know why evolutionary sense, but I just like kind of came to in the mid in like
three in the morning was like, I sense a small vibration that's worrying me.
So I get up.
I want to downstairs.
My phone's just like buzzing away with like notifications.
And also partly, I think it was one of our fallback alerts, which is a text message, which
is not as dramatic as pager duty, but it does just trigger my phone to buzz every 30 seconds.
And I was just like walking across the house towards this buzzing.
And I was like, I think it's the phone.
And I was like, oh, fuck, okay, we were 25 minutes late to this party.
But I was also like, holy shit, it only took 25 minutes for the buzzing level of a phone
to wake me up.
So this is a Rama sort of Rama sort of gets it here in that, you know,
it's a bit of a shock to hear in that he's making fun of alert PTSD, man.
I can tell you alert PTSD is fucking real.
So it's real.
And I feel physical pain when pager duty goes off.
That noise it makes will cut through the deepest sleep you could possibly be in and set you
into a state of alertness in less than half a second.
The funny thing is I heard it go off in the mall the other day.
I'm not sure if someone had it as a message tone or something.
And I almost threw up.
No, you know, we were talking about how you were burnt out.
And you say break last episode or the one before you were talking about how I was burnt out.
I was talking about how I can work 25 hours a day.
Yeah.
And you're talking about how horrible you feel.
You're like, I can't talk guys.
And when my phone goes off on it.
Yeah.
Incidentally, feeling great this week.
My bags bags are still there.
Still there.
Yeah.
Those are validator bags you're bringing with you.
I'm explaining the kids last week.
I have a question.
Like so.
So we had some something similar.
We we we even when I used to be at steak fish, we even had like physical sometimes physical
like pagers and all these types of things that you're kind of describing.
But what what do you think is the sort of sustainable solution to this type?
Right.
Because because like the the reason why maybe some people have this type of PTSD is because
if they don't respond like immediately, then right, you might get slash or you might your
uptime might go down and and some of these more competitive, at least at the moment,
the limited set of metrics that validators are compared across each other.
Right.
That's what you have to be competitive on or at least like be at the default.
But but yeah, how do you see that scaling as right from because there's many sides of validators,
right?
As you kind of mentioned to you guys, it's expensive to have other people in different time
zone, sometimes having that 24 seven through six, five coverage.
But as for you, you know, because it seems like you guys are all I think we're I think
you're going to get some very different answers for this, but I'll let I'll let the others go first.
So I think that part of the problem is tender duty itself.
Most of us use tender duty.
And when tender duty first came out, it didn't have different scaling for what level of crisis
a an alert was.
Right.
So it could have been anything from it being stalled for 30 seconds, say, and then they would
send up the most critical alarm it could.
Right.
And then now I'm like the different alerts probably six months ago, but I don't know anyone
that has actually updated theirs with the new system.
Right.
So now it can be you can you can set it relatively fine tune what is actually going to inform you
or not.
And I think that to to your arms to Rom, your arm,
Rama's point, setting up good alerting is that's very advanced, but it's so necessary.
Right.
There are so many things that you can be alerted for that only kind of sort of a little bit
matter.
Something you need to know about, but you don't need to address immediately.
It be aware of.
Yeah.
Be aware of.
Yes.
Send it to discord, whatever, but you don't need to be called it for in the morning for it.
And making that leap of the call versus a small alert.
That's a that's a I would say it's a very big leap that most validators haven't started making yet.
Well, it's a it's a quality of life thing.
Like you go from, you know, if you manage your alerts properly and even the way like alerts
are raised, like we've changed a lot of things to be time based.
So, you know, sometimes stuff just teeters over and then goes back or you might get a
temporary stall while there's like a lot of transactions in a block or something.
And well, that might be perceived as a stall, but it's just, you know, part of the process
of the block.
So we've like tuned our actual alerts to be more representative of what's going on.
So, you know, a stall won't immediately give you a warning if it's stalled for like
five seconds or longer than a normal block time.
It'll wait like a minute because it doesn't matter if you're stalled for a minute or it
doesn't matter if your particular node is stalled for a minute at all.
You might miss a couple of blocks, but, you know, but if you're persistently
stalling, then you're going to start missing a lot of blocks.
And then that's where, you know, you tear up your your miscounters.
So, you know, if you're at 5%, I think that's a good good place to start telling you.
But that's not a that's not a page of duty alert.
If you're at 10%, like maybe you want to get a text about that, but that's still not wake
you up in the middle of the night territory.
If you're at 50%, you really need to address that issue.
So, I mean, I think especially if you if you share a house with somebody else and I mean,
doubly so, when the small human problem occurs, I think until number don't go up anymore,
it probably isn't a phone call.
Like I went back like properly full time, like nine days after that we want appeared.
And I think within three days, I'd prioritise changing all the page of duty because
if if this is the only 90 minutes of sleep, clear sleep, you're going to get that night.
And page of duty goes off 45 minutes in, you are dead.
Not because it's a problem for you, maybe you can function.
Your partner is going to cut your skin off and wear it as a coat.
And that's incredibly potent incentive to fix page of duty and also to add like,
so one of the I think the only two ones now that will wake me up in the middle of the night
are number really don't go up.
Number has not been going out for some time.
And I don't know where the fuck the web server has gone, which the second one actually is the
only alert in our entire suite where because it's never happened that alert has never triggered.
And as a result, I don't trust it.
And I keep coming up with more and more elaborate ways to try and test that eventuality.
Because we do use some cloud stuff and blah, blah, blah, and the odd bit of on demand.
But I think eventually I'm going to add too much alerting, right?
And it is just going to go off because I've added too many alerts and one of them just
fails because of some reason.
And I fucking guarantee it would be at the worst possible time in the worst possible situation.
It probably won't even wake me up, but it will wake the rest of the house up.
And you know, I don't even need to explain why, you know,
it's going to be a bad morning for me after that.
I'm going to be in the doghouse.
So Dan, you and a bunch of steakfish people have started a new validator year.
How do you guys manage this?
Can I get up real quick?
Yeah.
We haven't introduced Dan.
Dan, we all know you, of course.
Do you want to introduce yourself a little bit?
I mean, we've lost some kind of media.
Well, that's a good idea.
We said he is Dan.
That's a good idea.
Yeah, exactly.
Yeah, I guess in this capacity, I'm representing Alio.
It's a ZKL1 that's like this hybrid.
It's basically straight forward.
Consensus and block building are done with POS,
but there is like a POW element, which is this mining component for grinding ZK snarks,
which you get kind of off chain computation verified through ZK.
But yeah, I've been in the Valor space for a while.
I wouldn't necessarily call it a Valor.
It's like an incubator.
One of my goals is to, I want like a, as corny as it sounds,
I want a Valorator to exist in every country.
And so like, because yeah, that's going to be a long spiel, but that's kind of what I want to help
create.
I want to have more different parts of the world have more representation.
It's interesting hearing you guys talk about Australia,
because that's like a pretty interesting kind of like, right?
It's literally geographically separated, but then being able to have that type of connectivity
to represent like that part of the APAC, right?
Even though Australia is not Asian, right?
It's still considered APAC.
I think it is extremely important just for most of like the infrastructure resilience of networks
that we had seen obviously with like what happened on Solana in 2021 with Hetzner.
And then also the ideological component of that for, right, if these blockchains are supposed to serve
the entire world, it should be served, you know, pun intended with servers and stuff from all around the world.
Right?
I don't think there should be like a specific contingent.
A specific contingent, sorry, I don't think that it's great to have a specific like region in the world that gets to
dictate the type of infrastructure services and how those infrastructure services can
eventually be responsible for guiding from off-chain to on-chain governance,
how the rest of the world use it, right?
The rest of the world has to also be at the table.
And then if there's this table that this blockchain governance off-chain or on-chain is happening,
the seats around that table should be representative of many different types of regions.
And then I want to make sure that the people that are sitting in those chairs are capable of contributing to these global conversations at an adequate level,
where if the conversations are calculus and you only know how to speak algebra, then you need to get them to a level where they are capable of speaking that language.
So, funny enough, it's like, you know, if we can have like a UN of validators or, I don't know, we can maybe call it like the G20 or something,
or BRICS versus NATO, I don't know.
But I think that's the way that, right, that both services the ideological goal for this type of decentralization,
which then services or makes it a lot easier for this type of infrastructure resilience and decentralized infrastructure hotspots,
which maybe we can call it like black holes, right?
Because we definitely had a black hole in Western Europe for a certain amount of time.
But then we need to plant other black holes in different parts, so the gravity well doesn't get bigger and bigger the longer we wait.
So, back to your question a minute ago, Dan, which was about like uptime resilience, that kind of stuff from a validator to a validator basis.
Like, you know, from the fact that you've just been talking about, and I suppose what you're saying with, you know, the status quo right now,
there are validators at the table and there are validators not.
Is there attention there, right?
Because, you know, the way to have uptime is to have professional operations that are making enough money.
So, first of all, there's an economic thing that's currently not happening in a bear market,
which is that you need to probably from every single chain be able to wash your own face and possibly pay for multiple employees,
which is obviously just not the status quo.
And arguably with the number of blockchains that are at extant now, it is not possible.
There's not big enough market for enough money to pay for enough people to get that security,
other than a small number of very big validators, which is not enough that number, whatever that is,
of really big validator companies is not nearly enough for decentralization, right?
And then you look at the, and that's in the context again, so this is where the tension is even more so,
the context where even if you take all validators, including all the small ones,
there still aren't enough validators to meaningfully talk about decentralization in a lot of cases in terms of true fault tolerance,
of the kind envisaged ideologically, I think at a really high level for these kinds of systems, right?
So I guess I'm kind of interested in like, what do you think of the, do you think that the,
I guess there's two questions here, right?
Number one is, do you think that the current big incumbents are actually like an anti-pattern thing that we need to be worried about?
Because if so, there's a tension there between the fact that they're the only ones that can currently run professionally,
or at least whether or not they do, they seem to do get slashed a lot.
So the reality is different, right?
But on paper, they're the only ones that can afford to do the professional infrastructure things that we'd expect.
And the number two is that like, well, does the entire system, if it's going to expand to be truly decentralized,
like you said, have a huge number of participants, should we be just trying to take governance and all that stuff out of the picture,
and require no education other than the technical education to run the thing?
I suppose back to the original Bitcoin, run it on your laptop, mine's in Bitcoins, right?
It didn't require much brain power.
You just ran the demon, and it obviously wasn't decentralized, but it was.
You ran the laptop, it did the thing, it did one thing.
You didn't have to worry about voting.
Yeah, all three of those questions are fascinating questions that I've been thinking a lot about.
So one was actually, I had hosted this infrastructure conference in Vienna, Austria in July,
that I think, and we wrote a report from that, and it was basically a survey of validators that had attended.
And I believe I, yeah, it was kind of, that was very like last minute.
I think I invited most of you guys as well.
But we basically had the whole stack, right?
We had the foundations like EF, Solana Foundation, Web3 Foundation, ICF,
and then Alio.
And then we had like a lot of the validators from each of those networks,
and the data centers, cloud providers, etc., like Tencent, Alibaba,
Hetsner, OVH, Equinix, Latitude.
And it was really fascinating because there's sort of like this, right?
Short-term and long-term kind of thinking, which addresses, I think,
your second question about the professional versus non-professional valid.
I wouldn't call it non-professional, but professional, like large-scale validators versus smaller
and medium-size.
And I do have an opinion on that.
But most validators, I think, want, prefer that type of, to address just your last point,
it's like the commoditization of it.
I'm not saying like I am deeply rooted in the type of role that validators or as infrastructure
service providers should take in that difference between a POW, specifically commoditized to a POS,
which is a bit more specialized.
Because in my opinion, I feel like the moment we switched to POS, we did not become commoditized
because there was this very clear element of specialization that required publicized
differentiation between the different infrastructure providers to get different stake, right?
Because everyone has to fight for stake, right?
Have stake come to them.
And that's where that specialization differs from that commoditization because it's not like a
infrastructure down to the different types of...
Is it...
It is specialization, I suppose.
I would agree with that.
But it's more of a million monkeys, million type writers specialization than it is.
It's people brute-forcing a solution to get more stake rather than...
And this is what we would expect, I think, from...
I'm not hip to the maths, but I would expect from a kind of evolutionary behavior perspective,
or like a complex systems perspective, that people would try the dumbest things first before
they get to the point of going, oh, I'm going to read a bunch of governance and think carefully
about stuff and write essays and publish blog posts that are well thought out, like some of
Shulze's or whatever, to try and get more stake.
First of all, I'm going to try...
I'm going to copy-paste somebody's code that has a staking UI that will only state to our thing.
The second thing I'll do is I'll go on Twitter and slag off a bunch of people really loudly,
whip up a mob of people and get a bunch of stake.
Those are the kinds of things that...
A lot of... Various degrees in various places, but a lot of the tactics so far in the space
have been very lowest common denominator.
Yeah, 100%.
That's maybe my feeling, but I don't think it's uncommoner feeling.
Yeah, it isn't.
And I do think it's also because of the lack of, let's say, actionable analytics.
So we have a very small set of quantitative data that people are able to see,
but that is also something that people don't necessarily care about and see too much.
I would say lazy or sticky stake.
I think a really good taste of that we saw, and even in Ethereum, which is some of the more commoditized
POS infrastructure, is there was a very large validator that did eventually get acquired by a large
centralized exchange that when the beacon chain first launched, they got slashed 75 times in a row.
So much so bad that a lot of validators and we, myself, had reached out to them asking,
like, hey, do you need any help or what's going on?
But the reality was that you should not have even gotten slashed and been kicked out of the set
even once because at least with the beacon chain, you're out for a whole year.
You're not getting a stake and you can't withdraw it.
But so the quantitative metric that people would be able to base their decision on
who to stake was very bad for that particular validator service provider.
But the reality was that people still staked in mass, incredibly large amounts,
which spoke to this sort of like people in care or people didn't know enough to care
or they didn't know where to look to care.
And it was just branding that made it a such a sort of like this.
Well, it's kind of, I suppose, back to the mistaking technological system for a social system.
Yeah, yeah.
And because they're the wherever, however it is that social discovery of like,
because it's the thing of like, you know, why do people not steal?
They're concerned as much about like in a small community, you're concerned more about
or a better example would be like maybe credit or debt, right?
You could go to if you live in a small village, you can go to the pub, drink a few pints and say,
oh, fuck, I haven't got my wallet with me.
Can I come back and pay tomorrow?
You're going to come back and pay tomorrow because otherwise that person's going to tell everybody,
you fucking pay me is owes me five pints.
Like, and whatever that kind of social discovery means is, you know, some of those
means could be very, very complex, but they don't exist yet or they're poor.
Poorly available, maybe you're poorly understood.
I think it is.
That's also still trying to that.
That's also still a little bit like my engineering how on there.
I'm like, oh, well, maybe we can build a solution.
But yeah, I don't think I don't know.
It's a it's a tricky one, isn't it?
It's it's the reason why rated exists, right?
Like, Elias Seamos from Bison Trails, he used to be a protocol specialist there.
He left and he made like a very comprehensive Ethereum validator analytics review.
Review tool and then they raise a lot of money and then they're servicing that same with JK,
the former strategy head of state fresh he left and created a Pibara, which also does
that type of it's kind of like a mixture of like stake pooling with validator analytics
on different chains.
And that I think is a evolution of a rightful evolution of the type of analytics that can
be actionable that will change us from what we saw with that previous validator service
right that got slash like 75 times a row when the Ethereum beacon changes launched to how
people can kind of be judged on their merits.
And that type of sort of like frameworks or standardization, I think is going to help
solve, right?
I really think that's the level of us getting the validator ecosystem coming from like
this adolescent phase into like this maturity phase.
I think three, four years ago, I think validator ecosystem was definitely still in its infancy.
But right, I think there are, it's like inefficient market resulted in very inefficient sort of
let's say stake being distributed.
But hopefully we're going to be able to see some of that more efficient sort of stake
being flown away and perhaps just removing the choice from users that like state pools
do.
Right.
Okay.
In state pools is a really interesting one, isn't it?
Because you're effectively, it's like hedging your risk, isn't it?
Across multiple validates, quite smart.
Yeah.
The smart thing to present to is an example of a technological solution to the social
problem, right?
Just go, well, don't ever let people stake to one provider and try and spread it out
in some way.
But I suppose there's another question here, right?
We're obviously talking primarily from experience, background, cosmos, Ethereum, and look, okay,
three to five years time, obviously Ethereum staking is still going to be a thing.
It's like Bitcoin, it has intrinsic value just on brand at this point and will continue
to do so.
But all of the other networks that we're kind of a part of now that might be worth something
now, are we at the point where all of those sort of validator run networks are going to
survive almost certainly not like, almost certainly like the 90th percentile, right?
If we're looking at startups, we'd say nine out of 10 would fail.
I think for blockchains, it's likely to be higher than that.
But what does this mean in terms of validators and proof of stake, right?
Because this, so we've been talking about it as a social problem and a technical problem.
What happens if somebody's technical solution that ends up getting traction is we just abandon
proof of stake and validators do move back to more of an infrastructure provider role.
But maybe they become, let's say, even more fungible or maybe they become less fungible,
right?
I don't know that I haven't thought quite enough about it to know what the pros and cons
would be of designing a system where validators were more fungible, sorry, less fungible.
But I guess I do know that the reality is that if you wanted to be super decentralized,
a good way of doing that would be to make them way more fungible, right?
Yes.
As much as I love POW and Bitcoin, I give this presentation a lot when I was doing these
validator workshops in Southeast Asia and stuff.
But something that I think is given sort of like, and I love the whole network state
idea and concept, by the way, and I'm not ragging on it too.
I don't want to say I'll be ragging on it too much, but everyone, we all live somewhere,
right?
We all live in some jurisdiction and we're obviously doxxed and stuff.
And I don't want to live in a cave with some node strap to my back and try to beam stuff
into space to stay with the ideology.
Because believe me, I was probably the most maximalistic Ethereum classic supporter back
in the day.
But the reality is that we all have to abide by certain jurisdictions.
There's jurisdictions that require compliance to particular views that they have.
And whether that be OFAC compliance or the way that you tax your validator or staking
rewards, et cetera, that are beholden to sort of, and that becomes like a limiting factor
to where you can host your infrastructure, which is an existential threat to the type
of infrastructure resilience that a network requires to be sort of like claim its resilience
and because of that decentralization.
And so I think there needs to be this complementary step that validator operators, let's say infrastructure
service providers have where they are formed in a certain way where that commoditization
argument of sorry, that anonymous commoditization path becomes less and less certain as let's
say the next five to 10 years regulatory regimes or countries or different like groups of countries
get to decide or bring the gavel down on this.
So a technical version of a vision like that might be to say, okay, is a validator a single
validator or is a validator a distributed group of three validators that all get equal
reward?
I mean, let's not get into the fact that legally that might be considered a joint venture
or multilateral joint venture or something.
I know it is, I'm not going to bring out the totally tax time banner again.
We're not going to, like anything else, we'll get the legal advice we need when we get
there.
But is that the kind of thing you're talking about where you'd have multiple entities combined
or is it some other kind of situation?
No, I don't necessarily think, yeah, maybe that'd be actually quite interesting if there's
like a group of entities that do come back.
I pulled that out of my back pocket purely because so something that's happened since
I think you're on the podcast last time Daniel, as I am a computer science researcher, when
I'm not running the validator company, I'm doing a part-time PhD, I'm a computer science
researcher into digital cash, like future of money stuff.
So there's all CBDC type stuff.
One of the interesting problems you have with that is that you have a bunch of small, a
small number of operators that are banks, right?
If they fuck up, the world melts down, right?
A government has to go into parliament and be like, why has this validator stop signing
blocks, right?
Those are not the guys you need to worry about.
But they also, as part of a lot of these infrastructure designs for anything that might ever potentially
interface into any kind of permissionless network, there's different tiers, right?
And you get to the point where you're like, okay, well, it depends on your solution for
digital cash, right?
But in a bunch of situations, you have parts of your architectural piece that are essential,
but potentially untrusted compared to or of a lower trust level or lower quality than
a bank, right?
So one of the things that I've had a lot of thinking time on in the last couple of months
has been how do you sort of, what's your level of trust for say an entity that is like a
company but may be fallible, but nobody is going to get pulled in front of like a political,
like what's it called an inquiry board if something goes wrong, because people's money
has disappeared, right?
Because we're talking about actual dollars here, actual dollars issued by the central
government, actual pounds, actual euros, whatever.
And one of the interesting things that I've been thinking about was multi-jurisdictional
entities because a lot of times, all our validators are as a public key, right?
And you can constitute a public key in a whole bunch of ways.
So I don't know, there's something quite interesting in that design space, which is
something that's very front of my mind.
So that's a long-winded way of saying, I know I said something that was interesting there,
it wasn't something I'm not clever enough to have just pulled out of my back pocket.
Yeah, I think maybe the closest like that DVT style way of running it, but also I think
what I would like to see it from a lens of is that, because you're describing like multi-jurisdictional,
like currency, and I think you have to look at it.
I would say that I would like to look at it from that perspective, where it's that you're
running the assumption is that the technological solution or the financial technology platform
that is this blockchain that you're servicing and need to keep up is going to be the de facto
platform that the world uses.
And from that perspective, that comes with a certain level of responsibility that some
of these infrastructure providers or node runners, et cetera, have to operate with.
And then that requires, and just working backward, that requires all these different things that
we need to fix, not fix, but address, like the type of, as we were talking about initially,
what type of operators are necessary to ensure that there is high uptime, so that there is
no relatively lower interruptions to this network that the entire world's financial
ecosystem and whatever is built on top of it is going to continue to operate with minimal
catastrophic damage if it does happen and the ability to recover from that, which is obviously
decentralization, addresses that with its infrastructure resilience.
But also from the operator level, and as you mentioned, the future of validators on different
POS networks that may survive or not, I kind of see it, and sorry for my bad analogies,
but it's like, I kind of see networks as different cities, and maybe validators are in some ways
a bit mercenary because they are operating different networks.
They wouldn't necessarily claim to be loyalists to a particular network in this analogy, like
a city, but I do see, think that eventually that consolidation as proof of stake networks,
they don't succeed, validators can easily leave, or relatively with a bit of friction
leave depending on what they're doing.
There's a more alarming adenture to that analogy, which is in the city-state's period of European
history, primarily mercenaries were paid in movable goods and gold, and when they weren't
paid, they sacked the city-state that they were formally contracted to.
I'm not saying this is my opinion or statement, but sometimes extreme circumstances do help
build better systems.
I'm not advocating for that.
I think a really great example is when the wildfires were going on in Canada and they
blew out to New York and everyone was complaining, there were those studies that were coming out
and saying 30% of these wildfires are started by firemen or ex-firemen because their communities
were threatening to take away their budget or something.
Sometimes you have to hold someone's hand to the fire, but that's not the solution,
but I wouldn't necessarily say it's in the wrong path, but because of that-
I'm waiting for when there is a blockchain that whether it's due to a foundation mismanaging
delegations or some other mechanism, the validator set colludes to essentially hold the blockchain's
feet to the fire.
We'll be fast-dating to see why.
There was-
It's only a matter of time, is my opinion on that one.
There was a really interesting one that I was really excited to see, not necessarily because
there was disruptions, but there was a foundation that had made a particular decision that angered
the validator community.
Some of the validators purposely, obviously at their own expense, they're shooting themselves
in the foot, but to make a statement, they were not producing blocks and then that disrupted
the network and slowed it down a bit.
It's kind of like the whole Buddhist monk-
What network was that?
I'm guessing it wasn't in Cosmos as well.
It was a smaller chain.
I have to look it up again, but it's like self-immolation protests.
It brought attention and it disrupted things.
Protesters going on to the highways and blocking traffic, even though it had nothing to do
with it, but it got attention.
I don't know if the foundation, that blockchain's foundation, did anything.
I wouldn't say it's collusion.
I would say it's coordination.
I think that's more democratic coordination to make things happen where actually infrastructure
service providers like validators basically hold the keys to the castle.
If there is enough coordination, validators can do whatever they want.
No, it was a separate one, not from Jacob.
Just to finish my analogy for the consolidation of chains where validators will have to be
like choose, a mercenary eventually has to have a headquarters somewhere.
I think the winning chain or the winning town or city or however large that becomes, that
will become where it will plant the bricks instead of tent poles and stuff.
I think that is the process through where we don't necessarily have too many of that
choice and loyalty situation.
Again, this is just off the top of my head.
What would that even look like in terms of validators planting deeper roots in one community
or one blockchain?
There isn't a lot of other than maybe investing more in the community.
It would seem that the better, more optimal play for validators always to plant deeper
roots within the validator community to shore up a position of, well, I use the word collusion,
which is probably not the right word, but you're saying like, I guess I mutual support,
because if you look at something like mesh security on the one hand, Roach is all about,
I mean, I guess it's in some sense, it's a little bit like a shared state pool, right?
But the thing for a validator to do, if you're, hey, we can afford to do whatever we want,
twiddle our thumbs as long as the validators stay up for three years and we've got, let's
imagine some amount, we've got $3 million, we can hire some engineers, what should we
do with our time?
It would be interesting to know whether spending $3 million investing in a community and trying
to get more stake on a network versus going, should we implement a staking solution where
we could gang together with several validators, increase our overall size and all of us take
a slice of a pie?
I would guess that the supply side one going with the validators would be a better business
decision and bring more benefit from an economic perspective.
Yeah, it's like a union, right?
A union could, you like the union.
Well, you're trying to create monopsony power.
So yes, it's exactly like a union.
It's like, is it caught out?
Yeah, yeah.
And or a guild, if we're being charitable.
Yeah, yeah.
And I think the very existence of guilds that have the ability to take power will also invite
the creation of other guilds as well, which I think from this more like capitalistic or
game, this capitalistic approach of incentives drive everything on these chains for operators
that should be a thing.
Because for example, if you guys collude and you have enough stake power and you change
a protocol and then you increase your, you say like, if you're part of this guild, you
have to be get more stake rewards, staking rewards or something, right?
Then other people are going to protest whoever.
That's an extreme example.
But I think the, maybe we should start a game of nodes guild where you have to acquire some
stake and then you give, you give the people on this podcast 5% of the stake that you've
acquired in order to be a part of the guild.
And then you get to keep 95%, no big deal.
But then, you know, we have, you know, is it more like NATO?
Is it more like the mafia you decide?
Or is it just like a two party system, right?
But I guess like getting back to the, I guess like the future of what validators should be
in terms of like powerful, larger service provider, validator service providers that
are just professional versus like other types.
I am of the opinion that if POS versus POW is commoditized to specialized, then there
needs to be more leaning into specialized and that validators themselves should not
only have the responsibility of just running infrastructure, because if you just run infrastructure,
that type of specialization will just aggregate power to larger providers that can sort of
just run infrastructure, right?
That's all they need.
And their DevOps team will be like five people and their marketing team will be like 30 people,
right?
Like, because that's that type of game that exists in reality with the larger staking
service providers.
But in order to avoid that, I think the leaning into specialization is going to be, I think,
a more diverse and healthy ecosystem where to respond to your question of like what can
validators do or contribute into their ecosystems, you have many things that are not just infrastructure
providers.
And I think a very small example or like one part of an example is the hackathon company
that runs validators and then they run validators for public goods.
I forgot what that was called.
Dora hacks, they do hackathons, but then they also run validators.
And then they fulfill like a pretty important part of an early ecosystem and establishing
these this element of this ecosystem.
And if we're using this sort of town or city analogy, there's a lot of different parts
of the of the of this town or city or ecosystem that that that is critical to keep it alive.
And so right, who are the firemen or fire policemen or like water companies or like,
etc.
Public library of these ecosystems that are like this this fundamental layer.
And I do think that maybe it's like the other way, maybe it's also that there are different
companies or organizations that can fulfill parts of the ecosystem while and then also
run run run a validator.
I'm not saying that's like the only solution, but that's an interesting way of kind of seeding
in addressing the this was a there was a variation on this very, very common in in Cosmos, which
is dev should run validators.
But then as dev you spend a huge amount of your time running the validator and less time
deving.
And actually, in certain cases, the validator becomes more profitable than development.
And then the development stops.
Yeah.
And I say that because like my my path into validation was a dev who ran a validator and
then the validator business, you know, we bring in people we we bring in contractors
and other people to help run our validator business because it is that much more significant
than anything I could do as an individual dev.
And that's why the business went in that direction, right?
So so that's I mean, and there's another interesting corollary to what you said, which is obviously
that there's these entities that have agency in the system, right?
And some of them are documented at the moment, some of them aren't two of the only ones we
know about for certain a devs validators.
And we obviously the foundation is super obvious.
But you know, there are likely a lot of other personas that we don't know about at the moment.
But the question is, like, should all those personas run validators?
Probably not.
Is the system ever going to account for those people formally?
Maybe not.
But then you're like, well, it's back to the question of like, you know, with the validator
comments and everything, like who's lobbying validators?
And these are interesting questions, right?
But at the end of the day, it only matters on the chains where there's money to be made
or where there's economic value or.
Yeah, exactly, right?
If a city is doing well, has a lot of money, then everyone goes to that city, right?
Like that's that like no one wants to live in a small country town anymore.
They all move to the city.
And it's that small country towns prerogative to become a big city.
Right?
It's a problem also, I think, with increasing inflation, right?
Because you can increase inflation as much as you want.
You get more of these tokens.
But if they're not worth anything, it's a fight that you have to have.
But it's kind of breaking the analogy of like you try to compare it to like how cities
became greater or nations became greater.
They like like fought wars and took off each other.
But I don't think like networks have like armies that they can they can.
They took over they took over supplies of movable goods and used it to pay for more
soldiers.
And then eventually ran out of goods to pay the soldiers and usually got sacked themselves.
So I don't know what that solution is.
But I think a lot of things can become a bit clear in terms of motives.
If you do look at it from that lens of like these blockchains, some of these blockchains
are literally trying to replace the monetary supply.
Right?
And like, what does that look like?
And then things definitely become inherently geopolitical and you have to be complimentary
to particular regulatory regions and stuff like that.
But but yeah, the TLDR is is I think that the specialization is at least from the L1
perspective is nice because you can have this diversity, diversity of organizations and
contributors to your community that make it inherently, right?
Diversity is resilient.
And then they're also all taking part in what you would say it's like staking or sorry,
node operating and staking is both contributing to your own network security and then also
providing right this sort of redundancy layer and resilience where you're just having this
these these nodes that are operating also right?
It's like a bakery running a validator in this running something or what's called staking
like moving water around.
Everyone needs water, right?
Fire department also pouring water like police department.
I think it's that becomes that maybe that part can be come a bit commoditized.
I know that argument breaks because there's that whole white label problem that you have.
But yeah, it's we will we will hopefully we'll figure that out.
But I think my personal opinion is specialization is better and can address a lot of or can potentially
address some of the issues that we've been having.
And with a lot of these sort of parallel developments in in the validator and L1 ecosystem and ecosystem
evolution, hopefully we're going to be able to see a more mature network after a lot of
the problems that we've seen in the past, right?
And I mentioned like a few like like better analytics to judge people, judge validators
right like like rated or abibara and then increase like operational expertise and experience
right from validators have been running for a couple years now.
And then with L1 teams that are open to kind of like developing these ecosystems with these
validators like I do help with alio for the validator side.
And then hopefully getting this type of clarity on on how the blockchain world exists in our
world because right, I don't think anyone is comfortable with living their life in caves.
We all want to be able to have some clarity without having like the FBI knocking on your
door and telling you you're going to jail if unless you shut down your notes or something
like that.
So I think I'm pretty bullish on a lot of these these sort of parallels having evolved
over time and turning into something that we can weave together to make a something that
looks straightforward.
And then right making sure that there is a sustainability for how everyone exists.
But the reality is that eventually there's going to be a winner or a handful of winners
and the vast majority of these chains will not be around.
I'd like to circle back around something you said.
You said the problem of white labels hinders this.
Do you consider white labeling as a problem?
It's really something we've discussed too much on this podcast mostly because I imagine
it doesn't come up too often.
And so there's not a lot of reason to discuss it.
I think that's the fact that you've raised it.
Yeah.
I think it's a temporary solution.
The most glaring problem was the Ronin hack because the majority of those validators I
think six out of seven or I forget were white labeled.
And then there was one that was Dow controlled and all the North Korean hackers had to do
is get enough tokens to control that Dow to control that last validator.
And they were able to pull off the largest heist because right, right, like decentralized
and name only and then they're all just being controlled by a single entity or like, you
know, you can also say decentralized and name only and all running on AWS or right.
So I think that's the existential risk of white labeling, but white labeling can help.
I think it white labeling is also used for decentralization.
For example, a lot of centralized exchanges white label their services, right?
They don't run their own validator.
They do.
But then they also allow other validators extra third parties come.
They do diligence them, they vet them, and then they will run parts of their stake or
parts of their networks, right?
If there's they need to run polka dot validators and staking, they'll have some polka dot validators,
right?
Like come in and help help service those.
But I do think and I know Cosmos has always had like these witch hunts with white labeling,
which is, you know, I don't think that's necessarily bad or good, but it's, it's, it's, yeah, it,
I think it's a temporary, they are temporary solutions because in my ideal world, I think
centralized exchanges should run their own validators and take more active, active roles
in the ecosystem that way.
And yeah, so.
So in the case of like centralized exchanges running validators, so actually, I mean, really
in the case of any entity running validators, I suppose, you're back back to the decentralization
tension, right?
There are a bunch of ways that we've talked about the podcast over the years now about
how you can measure decentralization in a set.
That's a whole other conversation.
But you know, do you think that there's some kind of thing where there is like other than
the obvious, which is the minimum threshold of voting power, be that 50%, be that 33.4%.
But do you think that there is a pragmatic case for always limiting like the upper bound
of what a validator should have relative to some arbitrary number, right?
I mean, that might just be simply profitability, right?
The upper bound should be just above where profitability is and the validator set should
be just as wide as is profitable.
I don't know.
Yeah, that's, there's a sort of consensus level answer to that, right?
Because for example, on Alio, if you have more than 25% of the network, you don't get
any rewards beyond that.
And then that's specifically to make sure that there are at least four node operators running
a chain at all times, the Alio chain at all times.
Yeah, go ahead.
So the 25% threshold, does that mean that they can't be delegated to above 25%?
They can delegate to them, but then you're just not going to get any extra rewards beyond
that 25% staked.
So does that mean like, say you've got 10 different people with 100, you know, staked
and that, and that equals 50% of the total available to stake, right?
Does that mean that like those 10 people get rewards on 50 of their tokens and not 100?
Yeah, yeah, exactly.
You don't get anything beyond that.
So the reduction is distributed across?
Yeah, it's proportional, proportionate.
Yeah.
It's still pretty high.
I couldn't imagine that happening, to be honest.
But so yeah, that's like, at least with Alio, that's like the very specific technical requirement
for that type of decentralization, right?
Because like you're not incentivized from a game theoretic grade to stake with a validator
just because it's at the top and it has more than 25%.
So I guess communicating that to the stake is also important because you can have that
limitation there, but if no one knows about it, they'll just be receiving less rewards
and not know why.
I just think it's normal.
Yeah, and it's just like how on a lot of other networks, people will stake because
a validator is initially sitting their commission at like 5% and then a week later they set
it to like 50%, no one checks and they're like, what happened to all my, what happened
to all my rewards?
So that's also a UX problem and I believe like leap wallet attempt to help the people
who use that wallet with those types of things.
Like I know, for example, that you will get an alert if you get slashed, particularly on
Kajira and everyone will leave.
Which is nice.
I think like the interface that wallets have, like mobile wallets, for example, are really
great.
For example, Phantom wallet, like the popular Solana wallet, they run their own validator,
and you can choose from that validator and then it's like you stake with Phantom wallet.
Kepler also does.
Kepler has Kepler-
Oh, there's a bunch of people who do it.
And I think that is a good example of one critical piece of the ecosystem that provides
a service that isn't staking that also has that.
And that's the same as the centralized exchanges as well.
Like they take all of the tokens and stake them with their own validator.
Yeah.
And they just don't.
I think there's a very popular, there's a infamous chain, I think that's mostly centralized
exchanges who are running those validators at the top of the list.
But then they were also punished, right?
Like a lot of people, a lot of the ex independent validators were able to deal with that problem
through protocol level changes.
But to get back to the-
Wait, just to be clear, we're showing, we're talking about them?
Which chain?
Oh, I didn't say the chain.
I don't want to shit on any chain, but there's a, I will say-
You punched down here, man.
Shit on it.
There was, you'll probably know what I'm talking about.
It was so bad that there was a security upgrade that needed to pass a governance proposal,
but we didn't hit quorum because none of the centralized exchange validators voted.
So we couldn't pass the security upgrade.
And then-
Was it an SDK chain or is it some other?
Oh yeah.
And then, and then, this is a long time ago, but, and then-
It sounds like Terra.
No, no.
I think most of the validators on Terra were quite active.
But, and then we had to vote, do another vote, rally the validators, lower the quorum, and
then pass the security upgrade.
It's just do things by the book.
But the lower quorum means weaker security, but-
I think I remember something like that happening.
It's two syllables.
But, yeah, it's, it's, I, but it's also, and I did hear a lot of these-
The points that some of the centralized exchange validators-
Sorry, centralized exchanges did make revalidators because they didn't want to have that type
of compliance risk.
But I think eventually we will have compliance clarity, let's say in the next five-ish years,
or however many, however long it takes, on the assumption that blockchains do get completely
mass adopted, and then this is the financial technology platform that we will use.
And it's just going to be straight forward from there.
But, but that's like, that's kind of like the lens that I like to kind of look at it
from because this is where we assume it will be, and then we, like how do we get there
is, is that, and I do like value that type of unique specialization that can contribute,
right?
Centralized exchanges are a very critical part of our ecosystem.
They're not all bad.
And they need to play an active part and role.
So none of you guys can guess it.
You can DM it in private chat and I'll confirm it tonight.
Fuck, there's so many SDK networks.
It sounds like Carver, maybe?
I don't know.
Carver, they had a lot of fucking-
Ah, fuck, I don't know.
They had a lot of fucking-
Where the shit is all that trading out?