Welcome to Game of Nodes, a weekly podcast on the cosmos from independent Validator teams.
Hello and welcome to Game of Nodes everybody, the podcast about the cosmos from independent
Validator teams.
On this week's episode we have a few of our members sat out there with probably cosmos
induced fatigue.
So we're expecting some drop-ins a little bit later on.
But we've reached out to fill our panel and we've got Demi here today to join us thanks
for coming Demi on incredibly short notice.
I think we only just realized that a lot of people weren't available this morning for
various reasons.
I think Alec LaFray has been working quite hard on the upgrades and the testing on Scotty.
So I think he's a little bit worn out at the moment and Jaby is deep in data and I don't
think he can pull himself away.
And then we've got Jack has showed up.
So we're just talking about our lack of people today but you know, four is good too.
So I think we can expect potentially block pain to drop in a little bit later.
And I'm not sure who else said they were coming but I'm not sure if I mentioned usurper is
a pretty crook at the moment so he won't be coming.
And yeah, so welcome everybody.
And I think we usually start with a question right, Schultz?
Yeah though, to be honest with you, I can't think of an example of the dumb thing that
happened this week.
Oh, we've got a ton.
I mean, you got a selection, you know, you've got what prompt 19 to work with, you've got
other bits and bobs but I don't know that feels kind of unsatisfying and played out
at this point.
Yeah, I think I don't really like to talk too much anymore about the old prop 16 and
all the subsequent props that stuff's gonna play it play itself out.
The trouble we should just start calling it the troubles.
I like that better.
It is like jam our head in the sand and pretend it doesn't exist until it comes time to vote.
You've got ostriches down there, right?
We have emus.
Emus.
Yeah, they're tasty little fuckers.
I do like emu jerky.
Really, I've never partaken in emu jerky.
I've only seen that they chase people and it doesn't look like fun, so I try to avoid
that.
I'm loosely aware of Australia going to war with the emus, but that's about as far as
my knowledge goes.
Yeah, so that was way back in the day we had like the emu war and I can't remember what
exactly that was about, but there was a lot of emus and they came to town and I think
they either had to all get murdered or hunted out of town.
But we've got like camels, armies out in the outback in Australia.
There's just like just camels man everywhere.
There's just so many of them because they survive really well in the Australian desert.
And in droughts, like when there's no rain for a long time, they all decide to go to
town and that can get quite bad as well.
So Jack, what is the dumbest thing or something notable of that you've seen in the cosmos
this week?
You got anything there?
Oh, I mean, if you see me looking down, I'm trying to submit a proposal to Crescent so
that I can claim the rest of my airdrop.
So if you're asking about real dumb things, that's probably one of them.
Let's see, what else has gone on in the cosmos?
Having to submit the prop was dumb that there was none available to claim your airdrop.
Yeah, I think that's dumb, but also having a governance proposal whose purpose of which
is only to vote to claim airdrop is like, yeah, okay.
It's not the first network to do it.
Stargaze had them numbered.
They had like four or five of them.
Yeah, I think it was like, what was it like?
One of Schiltz, do you remember who was submitting those?
Block Payne?
It was like someone was just like one after the other submitting those bloody things every
time they ran out.
And so now my voting history is terrible on Stargaze because I kept missing them.
Yeah, I don't know about Stargaze, but I know on Lume, on Lum, Whispering of submitted
something like 30 proposals on there to keep the airdrops going.
So it looks absolutely ridiculous on that blockchain right now.
Wow.
That's really funny.
Todd, you seen anything dumb this week, man?
Yeah, I botched a proposal this week.
Cheers.
We've all been there.
Yeah, yeah.
So what were the particulars of this proposal?
We just wanted to raise the minimum proposal fee on Serbis and the description didn't match
the actual parameters because we decided to change it at the last minute.
That was the 100 million versus the 150 million?
Yeah, something, 10 versus 15.
I saw someone else who botched a proposal where I think they meant to say the word million
a few times, but said the word billion instead.
And do you know what I'm talking about, Chelsea?
I can't remember.
Maybe it was Serbis as well or...?
I think that is Serbis.
I think it was for the Osmosis matching external incentives where Serbis meant to put like
1.75 billion total and instead added up to something like, I don't know, like 800 billion
or something.
500, 750 and 1.
Yeah, yeah.
A million.
Yeah, that wasn't me, thankfully.
Todd, you're in good company.
If you go look at the Cosmos Hub, we have a few, I remember Zuckie and I put up a few
failed proposals.
I was just looking at the vote history on MintScan and there's proposal number two,
Adam Transfer Enablement, which failed because we did it wrong.
And then proposal number three passed and then we have a Cosmos Hub three upgrade proposal
on there.
And my absolute favorite one is there's upgrade proposal A, upgrade proposal B, upgrade proposal
D and upgrade proposal E. And if you will notice, only one of those actually worked.
So the first time we tried upgrade proposals in broad, we had tried five times.
I remember that.
I felt like such an idiot.
It was not a fun couple of days of work.
That is a little bit out of control.
Yeah.
Sometimes like, yeah.
So Dimmy, do you have anything dumb that you've seen this week?
No, nothing particular came to my mind.
No?
All right.
I'm usually the same way.
I'm like, oh man, it's like on the spot.
I can't remember anything.
This week.
He's just like living that cool hacker house life.
He's just like, yeah, man.
Deep in it, fish hacker house.
Yeah.
I'm closing it closed here of building.
So I can see the reflection.
Yeah, it's nice.
Every time you post pictures, I'm like, oh, fuck, that looks great.
Yeah, look at that.
Where is it that?
Don't talk to yourself.
Palm Springs.
Okay.
You found a Palm Springs.
You don't want DJs knocking on your door, man.
Can I see the pool?
I've come to clean the pool.
I, what have I done this week?
So I botched an upgrade this week on dig.
Oh, fun.
Yeah.
Well, so I tend to not do the copy paste.
When I do upgrades, I tend to like type it in.
I don't know.
It's good practice to type things in.
And what happened on dig?
I like I think things are until I did a typo, I think, because I use, I use Cosmovisor on
dig.
And this is one of the reasons I don't like Cosmovisor.
One of the many reasons to not use Cosmovisor.
Yeah.
You get like sniped all the time with Cosmovisor.
But anyway, so I had planned to be out that day and no one else was available.
So I did a, you know, I just had preset up the upgrade, but I had a typo in the bloody
directory.
And so when it came time to do the upgrade, obviously couldn't find the binary and freaked
out.
But I was able to get to a terminal and repair it before we had all the votes to start up.
So I didn't miss a block, but I still botched it.
So that was pretty dumb.
Perfect.
Yeah.
I guess, but it's hard to be there for every upgrade.
It's hard to have someone available for every upgrade.
And lately there's been a lot of upgrades.
It is, this is like one of the reasons why we're trying to hire it strange love is to
make sure that we don't have to have some kind of on call rotation.
Yeah, that's a tough one.
Wait, hold on.
What do you mean?
So you don't have an on call rotation?
How would you get around that?
I mean, we do have an on call rotation right now.
There's like two people on the validator, but generally, because like all of the, you
know, all of the time, like the, the slashing times are like 18 hours.
I don't wake myself up if our validators down.
I'll like wake up and I'll be like, Oh, it's been down for eight hours.
Okay.
Time to recover the validator.
So this is why we don't do it on call rotation, Chelsea.
Okay.
I'm picking up what you're putting down.
Believe it or not, when we were doing the parameters for the cosmos hub, everyone was
like, so like, what should we do this miss blocks thing?
Like should this be like a thousand?
Should this be like 10,000?
Should this be like a hundred thousand?
I was like, well, like what's a short amount of time that allows somebody to like sleep
for eight hours and then wake up and have a reasonable amount of time to recover the
validator really somewhere around 18 hours.
We're like, Oh, great.
Cool.
Good.
So that in terms of decentralization of the validator set, that's one of those like
underappreciated really important variables.
Because if it was only an hour, like you would really have to have it on call rotation and
it would be a lot harder.
So yeah, that's that's how we do it.
I shall say.
Yeah.
So it's 28% vote power, right?
Yeah.
What was that?
So your mark phones a little bit quiet today.
I don't know if there's any way to like trying to get out of the brutal light here.
So I'm sitting back from my mic.
You look like you look like one of those dudes like, you know, when they're doing the non
hacker video and they've got like the light on their face.
Yeah.
I mean, it's all fine and good unless you've got a validator with, you know, 28, 30% vote
power and they go down for 10 hours.
That is true.
I wish we had that problem with strange love.
One of these days.
Or yeah, you're doing upgrade and you're sitting there waiting for someone with two people
with 20% to like show up when they wake up or whatever.
Upgrades are the only time where we need to really have like strong coverage.
You know, how do you guys like, do you guys use that we've got all of our discord alerts
filtering into like a common discord channel for that?
But I find they're still like, you kind of have to surf the channels and kind of keep
up with things because you know, it's governance and how are you guys doing that?
Calpa tech is Calpa tech.
It's great.
Calpa tech.
He does a great job.
Shout out, Adriana.
Shout out that I just started using not long ago there.
Discord, not discord, the telegram bot to keep a track of Gov proposals.
So basically every time a Gov proposal comes through, their bot picks it up and spits it
out into your own personal channel.
I mean, telegram pretty handy.
I check them like once every couple of days on all the networks we validate anyway.
That is my personal methodology, but it's good to get a prompt.
You know, if I'm doing some other stuff and I forget about it and then it comes up on
the Calpa tech bot, incredibly handy.
Yeah.
So anyone else got any like tricks?
It's taking to the hardcast on.
Sorry.
I use Zapier.
It's a modular cloud type thing.
And I have that watch the RSS feeds for GitHub releases.
Oh, yeah.
Yeah.
So you actually watch the actual, so you actually watch the GitHub's for their actual tags.
Yeah.
So as soon as there's a new tag, I know.
And I'm like, all right, I've got a Gov proposal going to drop any minute now on whatever chain,
you know.
So you should parlay that into a bot.
Telegram.
I was doing that using panic, which is, which is for telegram.
And I found that to be kind of unreliable.
Sometimes they just post tags, whether they mean for it to be the next release or not.
So it became one of those things where like I started paying attention to it, but there's
so much basically false noise that I almost immediately started knowing you're probably
within the first month of setting it up.
If you're anything like Juno, you would have like a thousand tags before we actually do
a release.
Yeah.
So you're about to say on how do you do it at Statefish?
Oh, yeah.
So we also have the tag tracker, but yeah, also for us, it's not that useful because
sometimes something gets tagged, but it's not time of the upgrade yet.
So we have, of course, a governance tracking system that it's custom integrated.
So we have our business slack and we also use Discord aggregating all the different
announcement channel to single one and keep track of that also on slack.
Yeah.
We use many tools and in general we have basically one dedicated person for like nine networks
or something.
So it's a lot of manual work.
One person per nine networks.
Is that what you said?
No, it's yes.
It's like this only because the protocol team is not that big, but we aim to expand.
How many networks are Statefish validating on?
Yeah, around 30.
30?
Yeah.
Yeah, they do everything too.
They're not going to easy mode all Cosmos Chans.
Yeah.
Yeah, it's a big number.
I know, I think it was Schiltzy.
You guys used to or still have your Discord channel, which also aggregates the announcement
channels of a lot of chains.
Quite handy as well.
Honestly, I think that's a good idea to start, but with how many channels there are in there
now, again, it just has basically completely turned into white noise.
And some of the networks I don't even validate on.
And so again, it's just one of those notifications pop up so frequently that aren't necessarily
worth following that again, it's just something that comes in and then like, mark red, move
on.
Man, there is like so many bloody everyone tags.
It's just so annoying that I think you should just not do that.
If people are subscribed to the channel or haven't muted the channel, then they obviously
want the notifications.
But why haven't it, it's a notification channel.
Don't like add everyone every time you do a bloody notification because then you just
wake up in the morning to a thousand notifications and you're like, oh, God, don't have four
hours to read all these.
Yeah, exactly right.
And the funny thing about that is that Zach from EasyStake and I actually got into it
once over me using Add Everyone because there was a Genesis that was happening and like
half the people got their Genesis wrong because I was just reviewing them and I never know
it was okay.
Everyone like got this wrong.
You basically all need to go in here and fix this.
And now I've like come full circle where now I've told people like, hey, you're at everything
way too much because he came in and like shamed me for it.
I'm like, well, I guess what's the quote?
Live long enough to see yourself become the villain, something that definitely it's very
true, I'll say.
It's a guaranteed.
You have to be.
Yeah, I think you have to be everyone.
Everyone just take this note.
Be sparing with the goddamn everyone or the here.
Well, the here's not so bad because you're probably subscribed to that channel if you
get to hear it.
But just think twice before doing the everyone because it's being annoying.
And I got chipped once for doing the everyone back in the day and I don't think I've had
everyone except for maybe one or two emergencies since.
But yeah, man, it's like it's just too many.
So can kind of parlay into what we were going to talk about today.
And I think we sort of came up with the idea that we might talk about security and people's
various methods of validator security.
So I know that, for example, Shilty and myself or maybe maybe no, sorry, that's Pocotu and
myself sort of subscribe at the moment to about a similar model.
Shilty, I'm not sure you're using it.
And I can almost just assume that Todd is using centuries and and dimmy.
Well, steakfish would have probably centuries for every chain and Jack is a Horcrux man
as far as I can tell.
So we're all good.
We are all Horcrux all day here at Strange Lock.
We are all in on.
Do you guys go?
Oh, no, I've lost you.
There you are.
Yeah, we are all in on threshold validation.
Extremely important.
I can sit here and talk about threshold validation for hours if you want.
So I think starting with me because I'm happy to talk about it, I'm going to put down where
we're at and where we're headed.
And then maybe if we just do a quick round robin of the room and if people are comfortable
to talk about their security, you can give us a quick heads up of your method.
So at King Nerds, when we first started out, we evaluated people's security methodologies.
We talked to a lot of different validators, asked them what they use, found a lot of people
were using centuries.
I found a lot of people probably said they were using centuries and aren't.
And others didn't use centuries.
So we were deciding whether or not to use centuries and trying to think of why we might,
instead of just automatically using them, thinking of why we might use them.
Because as a junior validator, we didn't have a big budget to be spending on extra infrastructure
at the time.
And so we came up with the methodology that we would start out without centuries while
we were like a low risk operation and not really a target of anyone.
And then as we grew, we would look at implementing centuries.
So further to that, I kept poking lots and lots of people and asking the pros and cons.
And in the end, we decided not to run centuries because we couldn't see other than people
trying to protect themselves from the host attacks, we couldn't see a huge benefit for
them if we're already being able to hit all of the blocks and not miss a sign.
So I mean, a lot of people use centuries as a way of getting global coverage so they don't
miss any blocks when they're signing and others are scared of DDoS and others are scared of
people being able to gain access to their box to get keys.
So there's various ways to keep keys off your box.
And there's also a very...
I think that's all of those are great reasons for centuries.
This is why that's the first...
If you're going to run a validator, everyone says don't do it on one node.
And there's a lot of reasons why.
And I think that the recent North Korean attacks on a couple of bridges are great.
It's sort of a harbinger of what's to come in this space.
The internet is one giant PVP game and we're going to see more and more of that.
So don't run with one node.
The easiest and first thing you can do is run multiple centuries.
That provides reliability and extra layer of security.
If you're running a full node for your validator on a node that's completely disconnected from
the internet and only connects to PVP through centuries, that's one barrier.
Somebody can still hone your OS, get into your century and then jump from your century
to the back node once they're in your protected environment.
But that's one...
It's at least a really important one.
And then the next step that you can take after that is either Horcrux or TMKMS.
TMKMS is a great other step.
That gives you a lot more operational flexibility.
Once you divorce the state verification function of validation from the signing function of
validation, you have a lot of flexibility in your deployments that you didn't have before.
TMKMS with this soft sign is definitely the easiest way to do that.
So those are the next steps.
And then you can do all kinds of fancy stuff in terms of making sure that your OSes are
automatically updating on all of your nodes to protect against the common set of attacks
that come through these internet-connected endpoints.
What DME just led for the last Juneau thing, security upgrades for the individual chains
are extremely important.
One thing that we as validators don't talk about a lot is those upgrades bring in new
versions of Go continuously.
They bring in new versions of the underlying HTTP libraries that often fix critical security
vulnerabilities that would allow an attacker to get into your Go process, break out of
that, and then into the host beyond that.
And many of these things, if you go read security reports of serious hacks, it's a chain of
vulnerabilities that are used to dive deeper and deeper into these systems.
So the more and more defense in depth you can put up as a validator operator, the better
chance you have of resisting some of these attacks.
And OrcRox is another one of those defense in depth.
Right now, if you're somebody who's going to steal some OrcRox keys, you're going to
have to get a couple layers into the validator.
You're going to end up with these keys, and then you're going to have to write the cryptography
to reconstitute the whole key.
Yeah.
So I'm a huge fan of where OrcRox is going, and that's ultimately where we want to be
is running OrcRox.
And I think it's probably the closest thing you're going to be able to get to being able
to run like a 5-9 as well because of the redundancy and the ability to sign on any node.
So if you have one go down for any reason, like if the supplier goes down from the internet
connection, one of the big problems is if your validator node is behind in terms of
uptime, one of the biggest problems is if your validator node is on one supplier and
you lose access to it because their internet's gone down, you stop missing blocks, but then
you can't pick up on another spare node or use one of your centuries because if you do
and the other one comes back up and you're not quick enough to catch it, then you can
potentially double sign and brick your node, or brick your validator, permajail your validator.
So pretty tough situation when that happens and you don't have access.
So something, as you mentioned, Jack, TKMS is a way to manage that as well as OrcRox.
So particularly around uptime, OrcRox is awesome and security, OrcRox is awesome, and we can't
wait to get onto it.
We just have to do more testing.
I think making the leap on the first network is scary.
And setting it up for the first time is scary.
The first time I did it, it was the Polychain code.
So it was like, there were zero guardrails on that thing, and it wasn't even really tested.
There were no tests.
The Polychain team handed to me, they were like, this is what we run.
If you want to try it, go for it.
I was like, yeah, I'm cool.
I can run this code.
And I played around with it a little bit.
I spun up a couple local networks with it, and I was like, pretty sure this isn't going
to double sign.
I was like, okay, I'm ready to do this hub upgrade.
And it was the first not for zero height hub upgrade.
So I've got this like, privvalidator state file that my single-dode signer had had prior
to that.
And I'm looking at it, and I'm like, okay.
So what height do I put in these state files on Horcrux?
And there's like two state files on each node on all of these Horcrux clusters.
And I'm sitting here at SSH, did to all the machines, manually setting the height in all
of these state files on Horcrux on this upgrade.
And I turned the thing on, and I'm like, and now I have to wait for 30 minutes for my hub
nodes to come up because it was like a horrible startup.
And so I'm just sitting there, and I'm like, maybe I've already fucked myself.
Like, I don't know.
And I'm not going to know for another 30 minutes.
It was so nail-blinding.
But yeah, it's not nearly as bad as that now.
And like, it's much better understood.
So it's still scary though.
And I think the instructions tell you to go out and take a walk or have a drink after
you finish it.
I do.
Yeah.
I remember, I read those, the initial set of instructions like months ago, certainly
probably had updates since then.
But it seemed like it was a nail-biting experience just reading the instructions.
Do we know who all is using Horcrux?
Or is it just French Club that uses it entirely?
No.
Oh, entirely?
I know there's a few people who have publicly said they've run it on a couple of networks.
Coffee Validator runs that on all their networks.
Larry from Delphi Digital runs it on at least two networks.
There's a couple of Luna validators that I know that run it as well.
Most of those Luna validators are just validating Luna, so for whatever that's worth.
And then I know there's a lot of folks maybe running it on Chihuahua or Cerberus or some
of these smaller chains to test it.
But I'd say there's probably five or six other validators who run it for all their networks.
Yeah, the telegram that you have for that, Jack, there's not a lot.
I think it's probably the same reason why I'm not there yet is the fear.
I think I'm going to jump soon.
I've got the playbooks together.
I've tested it on test nets.
Can you hear me okay now?
You're so little choppy, but I can definitely hear you.
Come join us, Todd.
You come.
It's good.
Yeah, the dark side.
I'm on my way.
Yeah.
So, Todd, jam that microphone up to your face and give us the quick rundown on your setup
if you're comfortable with it.
I assume you're using Centres and go.
So, my standard setup is three Centres in a validator.
Validator in one century in the same data center.
And then the other two on different providers.
So, are you running like a, say, for the private network?
Do you run like a wire guard mesh or do you just use firewalling?
I can actually use a wire guard mesh, but I allow the nodes to talk directly to each
other.
I open up those ports between them.
And, like, what?
At least the IPs or something like that?
Yeah, totally.
Because running it over a private network requires you to turn on that feature.
And I think that might actually be, you know, you don't want to do that.
I don't think you want to publish your private nodes in your address book.
Because then people can see that, right?
Am I wrong there?
I think if you don't have peer exchange on, they don't gossip the private addresses.
I could be wrong.
I mean, I guess you kind of have to be careful with it.
But I had less reliability using wire guard.
So, and, you know, I honestly see Centres a lot more, like you said, how it ensures stability.
So I've got a connection to another data center always.
You know, that's the big thing for me.
And you were mentioning earlier about how you see downtime at a data center.
They fuck up their BGP table and it's offline, right?
One way to help with that is I always configure my validator node to connect out on IPv6.
And Centres connected on IPv4 and that saved my ass on a number of data center outages,
you know, because usually the network goes down for IPv4 or IPv6, but not usually both,
unless it's a hardware thing.
So that helps a lot.
Yeah, other security things.
I think one thing a lot of people screw up is using Docker and UFW, which is the simplified
firewall.
And they don't realize that Docker, anything you publish on Docker gets put in the chain
before UFW rules.
So you've got all these people running nodes out there and containers that have their RPC
exposed and stuff like that.
So I always use IP tables, first of all.
And if the provider has a firewall, I use that as well.
But see what else.
Todd, is your, you've got like some like an analytics type thing as one of your websites
that from memory crawls open 2557 ports, right?
Probably all these people fucking up with Docker.
So what's really strange is when you do a peer exchange, it gives you the RPC endpoint
for the node.
So and then you've got the IP address already.
So if it's all zeros, just I just try it and say, hey, give me your peer list.
And that's how I build those maps.
So yeah.
Probably also got the most robust list of RPCs in all of Cosmos.
Well, yeah, it's all that that has done horribly.
Actually, I don't actually have a list.
It's all stored in memory and then it's dumped out as a gob.
Just a big map.
Let me see what's, I was most of this, I think is the biggest right now with 226 open RPC
nodes.
Wow.
So that's pretty massive.
So I feel like you should share that list with Tom so we can add it to RESTAKE.
Just extra nodes to use in case you have load balance against it.
Yeah.
Imagine that just getting taking everybody's RPC and just load balancing it with a smart
balancer.
And then just, you know, that's the network's big RPC super, super distributed.
It's honestly really not that bad of an idea.
Like if you're doing it for not bad things like RESTAKE, then do it.
Let's just do it.
Testing purposes.
Yeah.
Dimmy, what are they doing over at Stakefish, man?
You guys just century the world?
So I'm not sure how much I can share about the Stakefish infra, but yeah, we follow all
the best practices.
We have remote signers and yeah, of course, the size of the infra, the pencils on the
network, we are going to validate and how much is delegated to the validator itself.
But yeah, we always have a base layer of security that follows all the best practices
and maybe for some more like bigger networks, we also have some extra.
One thing that I'm not sure if anybody else is using is the relays.
I'm not speaking about relays of IBC, but relays between Centres and Validator.
I'm not sure if someone is using that or...
I know a couple people.
What was that question, Dimmy?
So this is the backstop bullshit right here.
It's basically another layer.
So you have the Centres that are connected to the world, the relays that are connected
only between the Centres and the Validator and then the Validator itself.
And it was quite a common thing in the early stage of Cosmosup.
Right now I'm not sure how many people are still doing that and I'm not even sure if
it's useful or not.
I mean, it's another layer of defense.
Yeah, I know Polychain was.
I think the conclusion still does.
Anyway.
And what about Dimmy Validator?
My Validator, yeah.
I also follow the best practices I have.
Centres almost everywhere I validate.
Yeah, except one network.
I'm not going to tell you which, but it's not profitable.
So one node is more than enough.
And so do you try to keep those light with state syncing and stuff like that?
So I always use default pruning.
Like I don't like to have pruning everything or to state sync and not have default history.
I'm not going...
Even on your Centres?
Yeah, yeah, yeah.
I do have default almost everywhere.
And I also have some totally separated nodes that I use maybe as archive or even more
history.
Like, I don't like to run my Validator on some pruned state.
I prefer to have all the history.
So yeah, this is my setup.
All my Centres have default pruning and also the Validator.
Interesting.
Cool.
Yeah, I actually started bringing up a bunch of archive nodes just for taxes.
So...
Yeah, Todd, you and me both.
I maintain full archives separate from my Validator on every network I run on.
And then I'm only keeping the last state of the network on the Centres.
So like real thin.
So with Jack, with your...
So hang on.
Centres, whole crux.
If Centres end whole crux?
Yeah.
So we do...
Our setup, I'll just tell you the whole thing because we've open sourced it and it's publicly
available for anyone who wants to give it a try.
We use Google Compute for our cloud and we use their GKE product, the Google Kubernetes
Engine.
But we have two separate Kubernetes clusters.
So one Kubernetes cluster is three machines underneath it and we do three Centres on top
of that.
So one Centre on each machine.
And those Centres are state synced on networks where it's available.
We use snapshots for networks where it's not.
And those Centres are only keeping the latest state and then throwing out anything new.
There's no indexes, we keep them literally as thin as possible.
And on that cluster, the only port that's available is 26656.
The Kubernetes control plan is private.
Everything else is private.
The only thing that's open to the Internet is 26656.
And then we have, how dare you not to use a Cache?
I know Adam.
I know.
Very not on brand for me.
It's sad for me too.
And then we have the Horcrux cluster and the Horcrux cluster actually reaches out to the
TenderBit instances to connect.
So the Horcrux cluster does not have any open ports.
It's an entirely separate cluster and it's even more locked down than the other cluster.
And in fact, there's a Junk box that's also incredibly locked down.
It is the only box that has access to the Kubernetes API for that cluster.
So a defense in depth type setup.
So you've got your Horcrux swarms.
Let me just see if I've got this understood.
So you've got your Horcrux swarm under a couple of layers.
And then you've got, so when you say Centres, they are all of your nodes that can potentially
sign.
So here's how it works.
All three of those nodes are connected to the P2P network.
They all are validated.
They're all like flagged as validators.
We turn that ProVal thing on.
And what that does is it opens a port on the TenderBit instance.
That port is only available to the Horcrux cluster.
So when TenderBit starts up, it waits for five seconds for something to contact it and
say, hey, I've got a pub key.
And once that happens, then it's connected to the Horcrux cluster in the back.
So the nodes start up, the Horcrux cluster sends the pub key to them, and then they know
what validator they are.
Yeah, so it's just the terminology between them.
So your validators are your Centres, essentially.
Yeah, so it's just, because the terminology just, when you traditionally think about what
a Century and what a validator is, and then you sort of mix together in that sense.
So you're not having your cluster of nodes that can sign with the pub key from your
Horcrux swarm.
Then behind another layer of Centres, it's the same thing where all of them can sign.
Yeah.
So you could do another layer like Demi was talking about with the relay nodes.
There's a number of things you can do there.
What my thought on the scaling of this is to do, to auto scale the cluster.
Also there's a change coming to the TenderBit prevale, where instead of the TenderBit instance
opening up a server that then waits for the pub key request, the prevale server is going
to have to open up a server and TenderBit is going to reach out to that via the GRPC
thing.
So we're going to be doing that GRPC conversion and re-jiggering our architecture a little
bit.
But yeah, that's how that works.
Where was I going with that?
I had a point there.
Well, I have two questions.
One, is that pretty cost effective on Google Cloud to do it that way considering how thin
those nodes are on the Kubernetes cluster?
Is it cost effective?
We are still profitable on all the networks we run on.
I'll say that.
Make five cents.
Yeah.
Well, is it really expensive in comparison to a traditional setup?
No, I don't think it's crazy expensive.
I ran, I used to run just on raw VMs and the price on the additional cost for the Kubernetes
cluster is minimal for Google.
That's pretty minimal.
But the additional cost is on Osmosis, we have to run the price ECPUs.
If you want to run the expensive desks, then that's a 10x increase in the monthly cost.
I think we're running, I'm going to quote you a number and I feel like it's off by $100
or so, but I think it's about $650 a month for the three node setup and standard config
for us.
That's in my mind what the number is for what a validator costs a month for us to run one.
The second question is, do you feel like a DGEN not having history on your validators?
No.
No?
You're good?
Yeah.
Obviously, you keep your own separate archive nodes.
Is that on junk hardware or is that?
Yeah, they're generally run on cheap hardware.
Again, some of these, you do have to run them on fairly expensive hardware just to keep
up with chain tips.
Those archives are a fucking beast.
We actually run three different classes of nodes.
We have the super thin centuries and then we have relayer nodes.
Those relayer nodes keep an unbonding period of data.
Then we have the full archives for, as Todd mentioned, paying the federal government.
That's kind of our setup.
Shilty, man, we haven't got to you.
While I appreciate the conversation we're having here, well, I guess let me go through
first what we do and then I'll do my caveat.
Yeah, go ahead.
No, no, no.
I guess I'll just jump right into it.
Basically, we're talking about centuries.
We're talking about validators specific best practices.
One thing we haven't really touched on is in most docs you see on anything, you see
that you're supposed to use the root user for your validator.
The reality is I don't think many of us that are experienced use root, but a lot of the
docs assume you use root and all docs are written for it.
In order to note, go ahead.
No docs I've ever written have.
All right.
That's fair.
Most that you encounter do because if you then add in the complexity of saying, okay,
you should have this user created, then it just adds one more layer of complexity to
creating the docs.
I think it's tough because a lot of the validators that spin up are still using root and they
don't know to actually turn on even their UFW settings as the very base level of protection.
There's just two things that I can guarantee you fewer than 50% of validators use.
One, users, or you could even go so far as doing it so that users, the specific user
doesn't have pseudo access because you could have like, you could have it such that a user
doesn't have pseudo access, it does all the files and then you go into root or some other
user that has pseudo access to that starts up the machine itself.
A lot of people don't do anything like that.
They're just using root.
They have no firewall settings.
Now they probably have the benefit of not having the RPC open because you have to manually
make that change for most networks to actually open it, but otherwise, they're open to the
world full stop.
We're having this conversation about what best practices are we doing, but I don't think
that's the most effective conversation to have to make it the best conversation to have.
We're talking like the advanced stuff.
We should just be talking very low level stuff.
Whenever you start up your machine, one, update everything.
Just update your security things.
We're starting if you need to.
Great.
After that, then you create a new user.
Don't use root and then log into that and do everything from there.
Then add a password.
Turn off password logging in.
Only use SSH keys, blah, blah, blah, blah.
Use UB key in order to lock your SSH key, things like that.
There's this entire conversation to be had that honestly, should be written as a medium
article or something.
People love really invalidator medium articles.
How about like, Shilty, I'd be happy to collaborate with you on that because there should be a
best practice for lower spec.
When I say lower spec, I mean general validators like you're trying to get to because there
is the extreme level like Jack, which is on the tip of the security spear.
There's security professionals like Todd who just couldn't possibly see himself not doing
things a certain way.
But then there's a lot of entry level validators.
There's a lot of mid tier validators that either don't have the resources to provide
that and or don't have experience to know those best practices in the lower space.
I think you're hitting it right on the nail, right in the head with just so many basic
things you can do to protect yourself from the outside world.
For the most part, even if you've just locked it down to your P2P port and even if you do
have SSH open, which is not a good idea.
But when was the last time there was a zero day on SSH if you've configured it properly
with no root and certificates?
There are a few things you can do to get yourself set up more protected than just you're lowering
in with root and copy pasting yourself to glory.
Exactly right.
And to be perfectly honest, so I come from a software engineering degree and so I'm happy
to go write code, but I didn't know too much about Linux.
I use Linux daily because that's where a lot of code is written, but that doesn't mean
I knew security practices.
So for my first network, I was one of those people who just copy pasted in and then I
was like, well, hold on.
This doesn't seem like it's quite right.
Locking straight into root doesn't seem quite right.
Or if you're using Contabo, I think you're defaulted to Ubuntu.
And then I stumbled upon Jack, I think I have you to blame for the century node architecture
right?
It's your article from 2018, right?
Funny enough, I posted that article.
Greg from fucking Saplapod wrote it.
But yeah, I mean, like I have been the century node advocate.
You can blame me.
Yeah, that's fine.
I didn't realize you weren't the one that wrote this.
Yeah, no, it does.
This is probably the first time I've, well, I think I've said it on podcasts before, but
no one ever remembers.
So I was like, oh, it's Jack.
And I'm like, yeah, they kind of, yeah, I posted it.
Greg didn't want to create a forum account because he didn't want more accounts.
So he was like, Jack, can you post this for me?
But yeah.
Greg, Greg from Akash?
No, Greg from, Greg that nobody knows, the Greg behind Saplapod.
Oh, Saplapod, yeah.
Yeah.
Yeah, I actually, I reached out to Greg to get him on the show here in the next couple
of weeks.
So hopefully we've got that coming up.
Right from Akash that is.
From Akash, yeah.
Yeah.
Quick plug.
Don't know what we're going to talk about, but I hit him up anyway.
I guess Akash.
So anyway, sorry, Schilti.
I don't believe you really finished your story before I butted in and started agreeing with
you.
Yeah, I mean, I guess that's the end of it really is that, you know, there's so many
small things that can be done before we talk about Centuries, before we talk about Quark
Crux, before we talk about TMKMS that would be far more beneficial for, let's say, the
every man validated rather than us hardcore extreme.
I'm on 25 networks.
Extreme.
Exactly right.
They should provide some documentation, like tutorial how to put some base layer security
to your nodes.
Like, yeah, I think I think that is definitely something that we need to work on Schilti.
Yeah, because it is very important.
And I think it probably should be prominent in like, you know, like network documentation,
even if it's just a link to something that validators have collaborated on, just as,
you know, if you're going to, if you're going to YOLO and Dubino validator here is some
baseline security that you should probably observe.
And you know, from a documentation perspective, for a network, for networks, I guess it's
kind of assumed that, well, I've always just assumed, I don't actually know if I'm specifically
right to make a non-privileged user, but I guess I've just always assumed that people
would be using a non-privileged user and that.
And that's why I always write sudo, just assuming that they are.
But I guess that works the same as if you just have root and write sudo, you just don't
have to type a password if you've got passwords.
Yeah, basically right.
You just bypass it.
Yeah.
So kind of important and kind of not addressed anywhere really.
Okay.
So I've signed myself up to begin a medium article.
Got it.
All right.
But like I would happily work through that with you, man.
You know, there might be things that both of us do, there might be things that either
of us do or what have you.
So as baseline security.
I'm sorry.
Go ahead.
No, no, go.
That was done.
I've been talking forever.
I find that automation really helps with this stuff because it's so hard to be consistent
every time.
You know, if you're running somewhere, I think it's somewhere around 20 nodes, it starts
to get to the point where you can't always remember what you've done where and having
automation tools to build that stuff.
Be it, you know, if you're doing in the cloud, something like Terraform or I do a lot of hardware.
So I use Ansible.
And but having the security built into your playbooks is a great way to ensure that it's
always done right.
Because I can't tell you before I started automating stuff how many times I went and
I was like, oh, shit, my RPC is exposed on my validator.
I'm, you know, stuff like that, you know, and it's like, so having a little bit of DevOps
experience helps a lot with the security aspects.
And and every time you make a mistake, you just put it in the playbook and then you won't
make the mistake twice, you know, so infrastructure is code.
Shout out Todd.
Mateo says one of the better validator articles is from Todd.
So take some notes, guys.
Oh, thank you.
Oh, yeah, Todd has single-handedly made our lives of under five much better on multiple
occasions with tender duty, even just today, he was helping me debug something that was
going sideways, but there were many times where that's that saved us.
Tender duty, I'm assuming has saved my life.
It's hilarious because it's it's it's like 200 lines of code.
It's like so simple.
Yeah, man, but like huge shout out to that piece of something I resisted it at first.
I resisted it at first.
And then once I used it, I was like, why did I not get on this train earlier?
It's just the most simple but most effective piece of like.
And it's like the first line of security, not security, but the first line of knowing
whether or not you're assigned.
It's like the base layer fundamental thing.
Did you sign the block?
No, here's a warning.
So it's like, you know, there might be things going wrong with your box beforehand, which
hopefully you'd pick up in other with other, you know, monitoring and alerting.
But if it comes down to it and then that's, you know, everything else fails.
And then that's the last line of defense.
Like, did you sign the thing?
No, here's a warning as a validator, you have one job.
Sign blocks.
Sign block.
Got it.
Got to keep a key online and secure.
That's it.
Yeah, exactly.
Anyway, so what if you guys want to talk amongst yourselves for a second, I have to figure out
where we get amongst yourselves.
Well, I mean, the phrase not here, direct traffic.
So usually when we stall, he keeps the things going, but I can't remember what the things
are.
So I need to go back and look at some fun sound effects.
Can you just like, can you make sound effects happen?
Well last time I made last time I made any sound on here, we got like, what do they call
it on YouTube when you get like, band does.
Yeah, we had to like just DMCA.
Yeah, we had to like just make a big blank bit in there when I pulled up a meme.
At least we didn't get shadow band.
That's a Twitter thing, right?
Yeah, it is.
It's a rumor.
Is it real?
Who knows?
Jay, I was claiming to be shadow band.
A question we've had like from, I'm not sure whether this came through YouTube or Twitter,
and I think we might have almost asked it last week.
So can you run a validator with no coding experience?
Does anyone want to take a stab at that?
Ooh, I could definitely take a stab at that.
Now I do have coding experience, but I've helped numerous validators get started that
I've had no coding experience.
I think really the only thing that's required is some level of curiosity.
I think that's the thing that really sets you up for success is like setting up the
validator.
Like at this point, I can set up a new node in let's say five minutes, but even that
feels like a long time for what it can take.
Your first time, it might take 12 hours.
You're running like eight lines that you copy and paste it.
Somehow it takes you 12 hours and you don't know how.
But as long as you're curious and you keep going, it, yeah, you can totally do it without
coding experience.
I am confident that if you jump onto the Uni2 test network on Juno, you can be, you can
not even have bloody experience in a CLI and have a node up within half an hour.
Do you have a link to the Uni2 stuff?
I have one of my lawyer friends that's interested.
Yeah, man.
I'll pull it out while we're talking.
To that end, I want to actually talk about Uni2 a little bit.
Yeah.
Okay.
Yeah.
Sorry for nothing.
Uni2, so that's Juno's testnet.
One of the things that Juno does especially well is the orchestration of their testnet.
And I think that's really what sets Juno up for success.
There was the transition, but I think basically transition between Uni1 and Uni2 was, I think
what created a lot of us is good validators because it was something of a gauntlet of
issues that we found in security holes.
And so we were breaking it daily.
There was six back-to-back genesis transactions.
There were issues that somehow we just kept working through.
It was almost like a boot camp.
It's not like that anymore.
You're not expected to be there 12 hours a day every day anymore.
But during that one transition period, all led by the fray, that made me certainly a
far better validator than it was before that.
That is the best way to get better at validation is to deal with crisis.
Yeah.
Because that is the hard part of validation.
The hard part of validation is not sitting there calmly trying to get your note up and
then you're like, how do I connect to this peer-to-peer network?
Or how long does this thing take to sync, which is always something that newcomers are
shocked by and like, why doesn't my computer just work immediately?
Yeah.
Yeah, I know.
But yeah, it's like when Dimi is desperately trying to find you on Telegram.
And it's like, did you upgrade your binary to version 2.5.6?
Because if you didn't, you're fucked.
That's that real validator shit right there.
Tell me, the days before governance upgrades must have been enjoyable.
Yeah.
It was really funny.
I liked it.
I thought it was fun.
I really enjoyed it.
I think Dimi did too.
I imagine that's why Dimi is so good with the spreadsheets for readiness, the readiness
spreadsheet.
You can bust out one of those in like 30 seconds.
Yeah.
Like a cartel.
Yeah, that was a throwback.
I tell you what is good about test networks for anyone who is looking to become a validator
that watches this or looking for information like how to become a better validator.
Besides the information that other validators put out to help new people is that if you
can help it, don't go straight to a mainnet.
A lot of people yolo into some low value mainnets or even high value mainnets if they can get
a gathering.
But recommend just jump in and spend a little while at least in a test net.
Because you can do things in a test net that you can't do in a mainnet.
You can attack yourself.
You can beat yourself up.
You can ask other people to beat you up.
You can spam transactions.
You can try to break things without annoying people.
There's just so much more stuff you can do to test your own skills and also to benefit
the network by testing the network.
If I had any advice for anyone who wants to become a validator, play around in test nets,
get familiar.
Just yolo straight into a mainnet.
Because it's not yourself.
Even if you are proficient in Linux or whatever you need to run that network, it's not just
your money.
If you get jailed or you double sign yourself accidentally, then that's other people's funds
that you're playing around with as well.
Yeah, I mean, some of the stuff we went through on uni was fantastic.
Using the network by messing up state on a bunch of validators, trying to recover from
a bad block.
To be fair, we did that on mainnet as well.
That's a little different.
Yeah, but I mean, when the ethmost thing was happening, people started seeing the same
things and I was like, dude, you're screwed.
Just turn off your node and wait for somebody else to start signing blocks.
If you don't do that, you're going to double sign.
Then what happened?
Everybody double signed.
I remember you saying that.
I'm pretty sure you said that in the chat.
Is there like anyone who's like...
I'm not a validator there, but I was...
Some of you guys somewhere else and yeah, somebody mentioned that they were seeing this
error and it's like, oh, yeah, that's what happened on Juno.
Just stop your node now.
Yeah.
Just stop your node.
There's a new block.
When is ethmost going to come back?
Can we wait?
Oh, Tuesday.
Tuesday, 26.
26.
Let's go.
Come on.
Yeah.
Want to see it.
So in there, it was like, yeah, they literally were telling people in the validator chat to
like re-sync your node after you put down a signature and then explosions happened.
But yeah, man, I believe that I don't think...
I don't remember seeing like any public announcement yet that it's coming back on Tuesday, but
I believe it came from the last community call that it's coming back on the 26th.
Is what I am gathering.
I made a tweet about it, but to be fair, I copy it.
You didn't see any French...
What's that, Dimitri?
I didn't see any official announcement as well.
No.
So only a few validators have tweeted about it that I've seen and I think it was from the
last community call.
I tweeted about it, but I copied it from friends.
So if it's wrong, blame friends, not me.
Just retweeting the news, people.
Retweeting the news.
That's how quality journalism...
Yeah.
That's how it happens now, right?
Someone makes a tweet and then CNN picks it up.
So we usually round out with...
And I think we're at about an hour.
This is like where we try to start to round out these episodes.
We usually try to round out with a question again.
And it is, what are you most excited about this week upcoming?
So you start with you, Dimitri.
If you've got something you're excited about, man?
Yeah, about the raw calculator.
I hope to be today or tomorrow.
I think you're not the only person who's excited for you to finish the raw calculator.
I think a lot of people want to get on their regular address.
And yeah, I know there is also a lot of people that are unhappy for some decisions that have
been made.
I can tell them that there will be a dull.
So we can do something later as well.
Potentially do something later.
Of course.
They're right checks.
You can't catch me.
So like when raw, like Dimitri, do you want to talk snapshots?
I think it's like...
I know.
Please bring it up.
So Dimitri, is it going to be like...
I'm just taking snapshots right now.
I got it on this other screen over here.
I'm just snapshot.
Like months of snapshots.
Guys, just snap shots.
So is it going to be like just putting your one address and done job?
Like you can put in any address from all over the cosmos and we'll just reduce it back to
like the hex and then go from there?
It will be similar to onata.
So there will be a page you connect your wallet, you claim.
If you had multiple wallets, of course, multiple playing to do.
Awesome.
Chelsea, man, you got excitement?
Yeah.
So I'm actually really excited for the secret network upgrade next week.
I've been here since secret two and we're a secret four right now and this will be the
first upgrade that doesn't require a hard fork.
And that sounds pretty minor, but I'm pretty stoked about it.
Chelsea, after we did, I'll just say 20 on the order of 20 for the first upgrade where
it didn't require a hard fork.
I was just like, this isn't even an upgrade.
What is this?
So easy.
Yeah.
It feels good.
It really does.
Asaf sent me the instructions to review before he posted them and it was like, it was the
normal six steps.
Like fetch GitHub, make install, restart node.
Okay, you're done.
I was like, oh man, this is great.
This is fantastic.
I could do that all day.
Also shout out Asaf.
He was the one who finally root cause the contract issue on June.
Yeah.
Asaf is absolutely fantastic.
I interacted with him fairly frequently and every single time he's just fantastic.
He's such a great person to work with.
He's awesome.
We should get him on here one day.
Shilty.
Oh, he might be into that.
That's a good idea.
Yeah.
I'd love to have a soft one.
That'd be fantastic.
All right.
Jack, you looking forward to something, man?
You're not done yet.
I'm not done yet.
Fuck.
It's the work ever done.
Am I looking forward to something?
I don't know.
We're making chicken casselade tonight.
So we got some chicken.
We got some chicken broth.
It's beans.
We got some mirepoix.
You know, just doing the thing.
So that's what we're working tonight.
Struggle to remember to eat.
That sounds delicious.
Yeah.
I have not eaten a lot today.
So I did get some eggs around noon, which is better than nothing, right?
Like a boiled egg?
Oh, no.
I made some sausage and eggs scrambled.
I prefer mine scrambled personally.
Cool.
If I'm feeling really spicy, I'll fry one.
That's sad.
Good.
I'm just glad I'm not sitting around and trying to balance ledgers to figure out how
much I earned for my greedy uncle.
You're looking forward to not have to do any tax calculations.
Well, I've got a lot of work to do yet.
Still slogging away at it.
Yeah, but it's nice that I can take a break for a few days.
I got my estimate in.
It's probably right.
I don't think I'm going to jail.
It's all good.
Our tax isn't due until the end of like, well, our tax year finishes at the end of June,
and then it's not due until like May the next year.
But I just, so May's coming up.
So I'm still doing tax from a year ago.
And I put on a crypto bookkeeper to figure it out.
It's just a completely lost track.
It was actually a DGEND a bit in Define, and it's just like so hard to figure out that
I just had to get a bookkeeper, man.
I just don't have the time.
But yeah, anyway, so I guess what am I excited about?
I was talking to Juice Labs yesterday, and I can tell you that I am excited about Juno
Tools being released whenever they get released.
Oh, it's ready.
Yeah, it's ready.
So when it goes mainnet, I'm excited.
So I can drop the Bloody King token and people can stop asking me about it because it's got
the Merkle drop built in and waiting and waiting and waiting and waiting and copying
so much.
King shed.
Just being getting hosed by everyone when King, it's coming in.
I'm personally really excited about Juno Tools.
Shout out to Strange Love Engineering team who built the front end there.
It's been exciting.
Did they?
Yeah, sorry, I just gave credit to Juice Labs.
But can you tell us who's involved other than those guys?
Juice Labs and then the Strange Love Engineering.
Juice Labs built the first version of the front end, and then we came in, did some design
on it, helped them with a logo, built out a lot of the functionality on the front end,
and they focused on the smart contracts.
So yeah, Strange Love, if you need front end engineering, holler at your boy.
So I just walked through that yesterday with Finn, the front end, and he walked me through
it, man.
It's pretty slick.
Grico crushed it.
He did a great job with it.
And then my buddy Max did the logo and the visual design.
It is very nice.
Actually, the logo is super crisp.
It's just, I don't know how it is so crisp, but it's crisper than what anything else displays
on my monitor.
And then the logo is just like, it almost stands out of the page a little bit.
I don't know how you did it.
Max is a great visual designer.
I love his stuff.
I've known Max for, I'm about to say the number 15 years, and we went to college together.
So anyway, yeah, it's been a while.
Well then, he's knocked it out of the park.
So I guess that probably brings us to a close for today, unless anyone's got any final words.
It's some strange coincidence that Pupmos is here, but Javi isn't.
Yeah, what a coincidence.
The alter ego coming out.
Thanks everybody.