Hello and welcome to Game and Nodes,
a weekly podcast from
independent validator teams.
This week we have Nalon,
who's going to be talking
all about his time racing
boats in Australia.
He is by his own account,
and I quote from the
pre-show conversation, a salty sea dog.
So no, joking aside,
we have fellow validator
PFC with us on the podo.
Um, increasing our Australian quota by.
One hundred percent for those counting.
Hey,
most people don't think I'm Australian
when I'm in Australia,
they actually think I'm American,
which is kind of weird.
Hey, why do they think that?
Yeah, I don't buy it for a second.
Don't buy it for a second.
I was, you know, in downtown Queensland,
uh, you know,
going to places and they
actually thought I was a Yank.
know it but when I'm in
america they think I'm
english so go go figure do
you know when I've been in
america like I have been
called australian a whole
bunch of times what is with
yeah really yeah yeah like
I distinctly remember like
uh like amongst a bunch
it's happened several times
like it happened on train
once but like really
distinctly I remember being
a giants game in san
francisco and the guy in
the row behind us like
tapped me on the shoulder
and was just like hey you
guys from australia and I
was like no we're okay I'm
gonna piece together why
this probably is bluey is
an extremely popular kids
tv show and it's australian
and you guys say a lot of
stuff similar to us in
terms of the lingo and I
think they're probably
picking up on your lingo
and linking you to being australian
so it's true americans
aren't very educated right
but seventeen years ago
that that was I think
that's pre-bluey an island
and it was this was just
decidedly australian this
was a middle-aged black
gentleman drinking beer at
a ball game I mean I don't
think he was like prime bluey
demo I mean he probably had
kids but like I think he
was maybe I don't know I
think he's representing the
bluey demo I think uh
that's what it's targeted to
Or is it like My Little Pony
where middle-aged men also
watch it or whatever?
Mate, the fairy thing is real.
But the storyline.
Just watch the storyline, man.
It's no big deal.
Yeah, I mean,
I guess there's quite a bit
of slang in common between
Brits and Aussies.
In PFC, to be honest,
you sound a little bit like,
I can kind of see maybe that in Oz,
people would be like,
have you been away somewhere?
Because you have a little bit of that.
There's a lot of Australians in London.
And you have a little bit of
that East Coast Australian
living in London for quite
a while accent about you.
Yeah.
Logs, sorry.
Americans don't understand
anywhere not American.
I think they just flip a
coin about where you're from.
Was he going to say something?
No.
it's like um they really
don't maybe I don't know uh
if your microphone's up
dude we can just find oh to
see even australians can't
hear me that's how bad my
accent is my speakers just
explode I don't know this
is really sensational
welcome welcome again to
professional podcasts
As frequent viewers and
listeners may recall,
we took a break to professionalize,
to think about how we were
going to move this forward.
And this is the result.
But hey, look,
we're less than four minutes in.
We've already talked to our guests.
We've acknowledged our guests.
This is the new
professionalism you can
come to expect from us.
PFC, we know you from working in Cosmos.
Yeah.
Is that your home ecosystem?
Is that the first area that
you... How did you get into validating?
What's the deal with that?
There was a guy called Pete.
I know Pete.
Yeah.
And he just kept on telling
me about this chain called Luna.
So if we, if the, the first thing,
the first way I got
introduced to Luna was via
a D and D game.
It was all about the moon.
So Pete created this D and D game.
They had the moon and all
this kind of stuff.
And that's how we, you know, this,
you know,
I was working with him at the time and,
uh, he introduced me to Luna.
I said, nah, that's forget about it.
Three months later, I was, you know,
tracking the price of it off.
I'm a.
Oh, fuck.
And basically I should have
grabbed it in then.
And I went in and made a
validator because there was
like fifteen validators or
twenty validators at the time.
So I was twenty one.
Got in there, learnt it,
ran on a little knock to start with.
And that was how I got into
the Cosmos world.
Validated number two was Injective,
maybe six months afterwards.
So
know it was all good uh and
slowly but surely uh you
know just learnt um
different things and just
expanded and you know got
that and yeah and then luna
went all to as I'm not
going to revisit that one
and just as I was getting
to into the lido validus
validator set too which kind of irked me
because they were the ones
who basically had all the anchor lending.
All the deposits were
sitting in Lido based validators.
That's where they tell you.
And I was just about to get
into just about to look at, you know,
retirement and yachts and whatever.
And it all went down.
So, yeah, but luckily for me,
injective went from four bucks, I think,
to twenty at the time and save my bacon.
I think there's quite a bit
from validators that have
ridden out the bear market.
Like quite a common story is like, oh,
no matter how many networks you run,
at any given time,
there's one that's doing well,
one that's absorbing the
majority of your effort
while not doing well,
and then a miscellaneous
school of cats that you
heard that are mainly
shitting on your carpet.
Yeah,
the good old one dollar a week
validators.
They generate one dollar a week,
and you look at them and go,
Why?
I'm not going to mention
Chihuahua by name or Star Wars.
Some people did make money off Chihuahua.
Yeah, they did.
I mean,
I'm pretty sure that King knows Big
Bull's blockchain.
He did.
We've had discussions about it.
He made his money and
There are other ones like
Coogee where you make a
dollar and then they do a
token drop or an airdrop
based on you being a
validator and just making
like twenty to thirty K in one day.
So, you know, yeah,
you didn't get that one.
That was the nutsack one.
I've got an airdrop.
Yeah.
Make the money.
Yeah.
So I made money said no.
So what?
I didn't get the email.
No, I just got the token drop.
But yeah, so it's interesting.
It's getting way more competitive.
And every time there's a new test net,
there's like a million
people joining or jumping in.
And yeah, it's an interesting game,
as I'm sure you guys are aware.
Yeah.
What's your feeling about
how things are changing at the moment?
What are the challenges that
you're seeing as an independent operator?
Uh,
it's more just reach or business
development.
You have to be able to know
when the chains come up and
you have to form like
little communities to
actually be aware of when things come in.
Um, and it's, uh, networking.
So on a non-cosmos chain,
I've got somebody reached
out to me and said, Hey,
I've heard you're good.
Would you mind validating, you know,
six months later on their
test net and they went main and it's, uh,
a nice little winner that was, uh,
you know,
getting six to a thousand bucks a month,
sorry, a week on that and still doing it.
And that's non-cosmos.
And that was just an
introduction that somebody
made based on friendship or
whatever in that, um,
friendship or whatever, being again,
a fantastically Aussie way of looking.
Yeah.
And, um, yeah, so it's that, I mean,
the actual technical component
isn't as challenging anymore.
I mean, running the first one or two,
after you've run them for a
couple of months, for me, it was more,
you start getting experience.
You learn new things as it goes along,
but it's more about, for me,
it's more about just optimizing.
How can you actually squeeze
yet another validator on
the same node and they still perform?
And how do you make sure
you're current with, I don't know,
Kubernetes ingress hacks
before they get hacked?
Thank you, by the way, Rhino,
for that one.
No worries.
that was that was a fun day
and uh then it's the joy of
you know waking up at three
a.m for uh oh you know for
a good old ibc
vulnerability and just how
do you go how do you do
upgrades across you know
twenty different chains or
fifteen different chains
all at the same time while
while retaining the sanity
what possessed you to start
using kubernetes that seems
like so much overkill to me
I was running individually
on single machines and just managing it.
And I had previous
experience with Kubernetes.
So I used to work at a
company with EOS on there.
And they were a very strong
Kubernetes culture there.
So I learned that and I knew it.
And I just installed it on
three boxes and just kept on moving.
The challenge with it is
basically the dis-scheduling for me.
Everything else just works.
I mean, um,
I have my configurations in Git.
I basically, if I do an upgrade,
I can basically,
I build up the Docker container.
I upgrade the Git version to
point to it and that's about it.
Uh, so it doesn't use Cosmovisor.
It uses something slightly different.
What is the actual challenge
with disk scheduling?
Is it like an IOPS thing,
just making sure that
you're not slamming the
same shit too much?
Two things.
One, you can't schedule based on IOPS.
Well, I haven't found out how to.
So you can have machines
which are basically
overwhelmed on disk IO and
you can't see it in the dashboard.
The second thing is if you
want to get your nodes fast,
you have to have local disk.
So no NFS,
no sharing and all that kind of stuff,
no replicating.
So one of my first
challenges was I was using
a replicated file system
and it was great for reliability,
but the machines just sucked.
So you could run two nodes if that.
I went, they said, screwed it,
let's go and get some disk
mirror and go to local.
So I'm using LVS on the
boxes and they run fine.
I'm running the say testnet
via Kubernetes now.
The injective testnet when I
have it up is also running
on via Kubernetes.
and the great thing about it
is if a machine dies I just
destroy the disk allocation
kubernetes um find a new
snapshot somewhere and it
just finds another another
machine with the right
amount of disk and it just
picks it up so are you
running like your own
hardware for your
kubernetes or do you run
like same data center from
a bare metal provider or
how do you figure that out
so I run ovh and leaseweb so
I rent machines from them I
used to do it by a headsner
but uh they kind of got a
bit snarky at me when I was
running I think it was
avalanche and they I
misconfigured something I
was using port eight five
four five and uh that's one
of their network monitors
they've got a couple of
other ones and they said
they sent me a letter
saying please explain
Uh, which is nice of them.
I was running on their,
on their hardware for three
years and I said, okay,
that was my little, you know,
no notice to remove myself.
And, um, yeah,
so I'm running in four
different data centers and OVH,
and I think one in or two
in a lease web at the moment.
So I think the good thing
about OVH is they have like
a pretty decent private
network bandwidth.
Yeah.
Which would be ideal for Kubernetes,
right?
I don't use that.
I'm just using a public one.
Again, it's simpler.
I'm trying to make it so I
don't have to think because
thinking is hard.
They do have a private network,
but most of the nodes are
public facing and they've
got IP tables and security at that level.
And
Is everything meshed
together as one big Kubernetes cluster?
Yeah.
There's two different options.
The way I could have done it
is I could have had three data centers,
each one of them being
their own unique data center,
and then basically have the
cross-connect at the whole crux level,
so to speak,
or I could basically just
have one big data center.
What I have in the actual
pod specification is affinities.
So what I have in most of
them is what they call an anti-infinity,
which is basically do not do this.
And I say do not run a node
if there already is a node
of that type running in that zone.
I call it a zone, but it's a data center.
So then, for example,
what I do is for the Terra one,
I basically say don't run
another Terra node in here.
And the rules don't have to
be super complex,
but I can do it at the zone level.
I could have done it at the provider level,
but because I've only got
two providers now,
I just do it at the zones.
Yeah, fair enough.
Yeah.
It's not perfect, but it kind of works.
I don't know.
That seems pretty sweet.
It just spins up another one
if it happens to be it goes down.
Yeah.
That's kind of nice.
I mean, that's the dream, isn't it?
Do you find with that setup
any problems with the fact
that obviously a lot of
these services are...
quite well especially smart
contract chains quite often
are very heavy on say ram
requirements or whatever
and do you find that you're
running fewer larger boxes
to get around that in terms
of making resources
available to the cluster or
are you still erring on the
side of redundancy in terms
of box provisions and just
relying on the scheduler to
to figure out how to cut it
um basically figure letting
the scheduler do its job
and um it figures that kind
of stuff out when you
specify when you start a
node you say I would like
to have this much memory
and this much cpu so
typically I start with
sixteen gig so you know I'm
looking for a spare sixteen
gig so it will only do I
think machines running a
hundred twenty eight gigs
so one hundred twenty eight
divided by sixteen is a theoretical limit
And that's how many you can do.
And it just finds them.
At the moment,
I'm probably under capacity.
Sorry, over capacity.
Because every time I add one,
it doesn't complain.
So, you know,
I added another Thor node on
there and just said, okay.
And it was like, huh?
Obviously, I've got too many machines.
So it's fine.
But I'd rather have a little
bit too many because when
one or two goes down,
it's much easier just to deal with it.
But, uh, yeah, I mean,
looking at my machine now,
this one's a bad one.
It's got a load average of seven team.
Um, again, just IO.
Um, but it seems to be running.
Oh, let me go to one,
which isn't widely going.
Oh, look, my servers are, no, no.
Um, we've got load average of four.
Uh,
let me see how many different it's
running.
One, two, three, four.
Five, six validators on it at the moment.
That's a node with a hundred
and twenty eight gig.
right nice I guess like one
of the things that it's
like um first time set type
that kind of a setup with
that kind of automation
something I've looked into
before in the past for what
we do but like we basically
found that the requirements
of the different networks
we run are so you're like
okay even if the capacity
they actually need to run is
maybe a lot less than the advertiser.
If you have like,
let's say foundation
delegation that requires X
amount of capacity and provisioning,
and there's some way of kind of... Well,
I guess they're relying on
your honesty really, aren't they?
But...
you're relying on this really,
really lopsided provision of capacity.
I can never quite work out
how to cut that.
It always just ended up
being easy to just go one box,
one client when it's that big.
And then after that, you're like, well,
there's no point running
Kubernetes just for the Cosmos networks,
right?
We should all have similar
requirements or at least
that was the way I kind of ended up.
Yeah.
I mean,
I just looked at that box that I
was talking about.
It's also got an avalanche
and a Thor node on there too, just FYI.
So again, the way, the way I,
to get it right in my head
is if Injective had a massive spike,
Injective's not running on
its own hardware,
but if Terra had a massive spike,
there's enough spare memory
on those machines combined
to allow it to have its spike.
A ride chain, for example,
notoriously gets up to, you know,
gets up to sixty-four,
ninety-six gigs sometimes.
It's just the way it works.
And it lets it have its
spike and it goes down.
Now,
if every single Cosmos network was
going to have spikes at the same time,
that might not work as well.
You know?
Yeah, fair.
Now I'm going to get all the
foundation delegations to
me just going whoosh.
I disagree.
I mean,
I know you're just joking about the
foundation delegations,
but there are very few that
run Kubernetes.
I can think of like three offhand.
There's like the Nibiru team.
Noel,
I think you're pursuing Kubernetes
for a little bit, right?
Or at least you flirted with it.
Yeah,
I had to look at Kubernetes for a
particular network and it's like,
we didn't use it.
Just because at the moment,
it's a learning curve for me.
It's like I need time to
figure all of that out.
It's not something that I
can just do right away.
But also,
you need to dedicate a certain
amount of resource to it.
The juice has to be worth
the squeeze at the time.
Yeah.
It is a pretty good fit for
Cosmos nodes though.
Like for, for, for Cosmos,
I think it starts to, I mean,
we run a bunch of avalanches as well.
And I could see that like
when you have a node that takes, you know,
it's seven terabytes of
data and it's sucking down, uh,
half the cores and half the RAM.
Like you kind of lose the need, um,
versus just regular automation.
You know what I mean?
But, but Cosmos knows,
especially if you're running valves,
cause then you're state
syncing them and you're moving around.
I don't care.
And Horcrux is dealing signing.
Like that's a,
That containerization and
orchestration makes a lot of sense.
I think global RPC and some
of these other types of things,
it seems like it just, to me,
it seems like it would get in the way.
But for like running VALs,
that's actually pretty nice.
Yeah.
I mean,
I have had challenges running Polygon.
Yeah.
And I need to move off
I mean,
I'm currently running it on Kubernetes,
but I need to move that off.
Injective, say,
I wouldn't even dream of
using Kubernetes for that.
They're magnets and stuff like that.
They're just too big.
Yeah.
Especially, and say, I mean,
say it might not be bad now
with CDB and everything,
because it's not that hungry,
and it states things fast
and things like that,
where you could keep the
container somewhat,
keep the data somewhat small,
but it does need the resources,
that's for sure.
Yeah, I mean, every day in the morning,
I check which containers
are running out of disk
space and remedy them.
And that's basically my daily chore.
Operational run book is just
basically see what's about to run out.
And because everything's
running whole crux, if machine dies,
it's not the end of the world.
And then since it's in one
Kubernetes cluster,
that means all the
containers can DNS themselves, right?
You have a single namespace
to get across all machines.
Is that right?
You can.
What you can also do is
limit it by namespace.
So for example,
you can put security controls in.
So in this pod,
you have this security pod.
context, so to speak,
and give it permissions
like you cannot look
outside your namespace.
So because it's a fairly private and again,
if I do this again,
I might do something like that.
I changed the layout a
little bit of how I design
namespaces and do that stuff.
At the moment, it's fine.
For me,
I'm not too concerned about them
being able to access each other because
know the code that they have
is fairly self-contained
unless you get into a place
where you have a rogue
network and they all of a
sudden start doing port
sniffing and end mapping of
around starting attacking
other nodes inside I mean
possibly I mean I stopped
running those kind of
networks a while ago but um
I'm not not as concerned my
security posture on this
might be a bit more
optimistic than others you know uh
Mainly, if my validator goes down,
you'll screw up the people
who delegate to it, which sucks.
And I don't want that,
but it's not going to screw
up the network because I
don't have fifty percent or
one hundred percent running.
There is one network I run
by myself as in it's like a
four node validator set for
somebody that presents its
own issues because
You have for seeding and for peering,
you have to actually go
outside the network to get
the real IP addresses so
that they can do discovery.
So that's probably one of
the challenges of
Kubernetes if you run the entire network.
You need at least one or two
machines out there.
But, yeah, I mean,
it's been working quite well.
I probably need to look at
the machine hardware now
because my delegations have
decreased and the market's gone down.
I need to basically look at
my costs a bit more, as one does.
Overall, it's served me well.
That's cool.
And so, so if nodes, if,
if you have a Horcrux
structure and one of those,
you need to move those or,
or one of those Horcrux
nodes is on a machine that's dying,
or if it's talking to an end node,
that's on a machine that's dying.
Um,
You can replace those.
How do you deal with like
horcrux situations if you
have failures on on those
types of things?
I delete the the PVC on it
and just hit control delete on the pod.
OK, you just you just regenerate it,
regenerate it.
It works.
The challenge for double
signing is still because
there's two nodes and it's done by DNS.
So my configurations are DNS based.
It will look to the DNS and
I haven't had a chance, you know,
Again, touch wood.
Most of the challenges with
Horcrux is when you upgrade
versions for me.
And that's where it's like that.
And all of them,
you just do it the old fashioned way.
You delete all the Horcruxes
and you bring them back up again.
Bring them all up again.
Yeah.
The challenge sometimes can
be where you have, you know, three,
three centuries running in
three in the network and
they cross and they cross
over the whole cruxes.
So you have going from DC
one and the whole cruxes and DC two,
and then DC twos,
whole cruxes and DC one
that can confuse it,
especially in a box when a DC goes down,
which has happened.
Um,
but overall it was annoying more than
anything else.
Yeah.
Yep, that's cool.
Makes sense.
Yeah, I think it's, I think it's, I mean,
the Horcrux should be the one thing that,
that I would consider moving across.
That and I run, we run a,
I run a ton of containers
for another chain that we
probably has fifty containers on it.
And that,
that would be great to move into
that situation because all those are just
basically external adapters,
and they just take RPC requests.
So the ability to have that,
that should be in Kubernetes.
So that would be great,
because then that literally
totally could survive
anything and be able to
move those things around
and namespace aspect of
things versus just having a, you know,
those are all just ansible eyes, but they,
they're still running on a single machine,
which would be great to be
able to distribute out and
do some better and smarter things with.
What you also should be
looking at is probably how
you can do multiple user levels.
So then one of the things
which is getting me is how
can I basically get a DevOps,
hire another DevOps and not
let them have access to the whole cruxes?
Sure.
Yeah.
And then it's like, well,
what can they do?
They can screw up a note.
Oh, well,
but they haven't got access to the keys.
And then it's just like, yeah,
I shot myself in the foot,
but they can't steal my business.
Yeah, we do that.
We do that through VLAN segmentation.
And like,
there's some IAM stuff stuff
associated to that.
So yeah, we Yeah, that's, that's,
that's definitely a it's
nice to be able to say, I mean, when we,
when we can,
but we do have different
layers for for who can get
into what and how.
And so that's a great piece in there too.
And I know Kate has the ability to do that,
right?
You can control that.
I don't remember what that's
actually called,
but I know you can kind of
build some different layers
of access within there.
But that's the beauty of the
Cosmos network.
You can do that.
I mean,
go to Ethereum and everything's the
same little bit.
All the private keys are all
embedded into it.
They haven't got
multi-signing type facility.
It's sucky.
I'll say it's antiquated, but yeah,
it is antiquated.
Yeah.
I mean,
you have Web three signer and
things like that where you
can at least you can have a
third party node that that carries keys,
which we use here and there.
But but but yeah,
there's no there's no NPC type structure,
right?
Like, yeah,
there's nothing you can do
other than be a part of a
Lido SSV network that
doesn't make any money.
But, you know, it's all tradeoffs.
Hey, man, it makes some money.
That's what I'm talking about.
Well, we haven't we're not live yet,
so I can say that.
like just don't don't do the
math don't do the math it's
a good five of seven yeah
we can take we can take two
notes down it's fine
everything's fine yeah um
yeah it's fine it's fine so
then what last question for
me is on the what do you
run on the actual on the hardware itself
I was initially going to rerunning Talos.
That was, it looked like a dream thing.
It basically, you know,
it's a small footprint
designed to run Kubernetes under,
you know, fully.
It's got huge security things.
but due to the way that they put their,
they need a separate system
disc and stuff like that.
So you need a smaller system
disc and then the bigger, larger ones.
And Hetzner at the time,
you just basically got one
disc or you got one size disc.
So you couldn't actually configure that.
Thank you, Hetzner.
So I'm just running Ubuntu.
Oh, okay.
So it's just, so just to base on Ubuntu.
So, I mean, those other ones,
do they give you like the
firewall controls and like, you know,
tune d and things like that
that you can control the
base hardware at that level
because some of those
things you still want to do
some hardware tweaks and
make sure I have assist
control uh tweaks that I do
I think that they're
actually in github yeah
and I have something on the
network adapter level just
to open up the size of the packets.
Those are probably the only two.
There was a guy on Secret
who basically gave a
recommendation on how to
performance tune the box at that level,
and they use that.
Otherwise, it's just LVMs.
I think that guy was Todd, by the way.
Yeah, Todd.
Thank you.
It was Blockpain.
Yeah, it was him.
Definitely use that kind of stuff.
Haven't really heard much
from him recently.
I don't know what he's up to.
He hopped out last year.
Yeah, he decided he was done.
Yeah, that's cool.
So you do that.
You do on the Ubuntu side.
I just never know those
other Kubernetes like base
images like Telos and things like that.
I've heard some of those
that are that are I just
don't know if they're if
they're really architected
to be in a public IP space.
You know what I mean?
Are they really meant to be
in a in a private data
center that has hardware firewalls and,
you know,
you have less shit hanging out
in the wind?
so inside of kubernetes you
have psyllium as well and
that has some fine-grained
um yeah so most of the
stuff is all all the
communications all natted
oh that's cool between the
between the machines and
the clusters yeah so you
don't you know they
communicate freak a lot but
most of the stuff is natted
every once in a while I hiccuped
And I'd see like these ten,
ten IP addresses with eight port, eight,
four, five, four, five being registered.
And that's basically how
what has Hetzner logged.
So it could basically see
the internal communication on the ten,
ten.
But yeah.
Again,
I'm not running billions and
billions of dollars.
If you're going to start
spoofing that and maybe it
becomes an issue then,
but I trust my data centers
to do network segmentation
to the best of their ability.
And judging from what I can see, they do.
Yep.
That's pretty cool.
Yeah.
It's pretty interesting.
Yeah.
I should, I had to mess with it.
Cause my,
my experience has always been like in,
in a public cloud, you know what I mean?
Like easy to fire up an AWS
or back in the days to use
Linode before they got bought by Alchemy.
Um, and those are nice.
Those are nice things you
can point a help chart to,
and it just worked.
You know what I mean?
Like you, Hey,
I want three nodes or I want this.
I want, here's my disc and this and that.
But as the fray knows that gets pretty,
pretty pricey, pretty effing quick.
And so to be able to,
to be able to actually use
some of that in maybe a mix
of cloud and bare metal,
which is actually pretty,
that's actually what I was thinking was,
would it be nice to split
environments if possible in
some of those senses,
because it'd be nice to
have some things on,
on some cloud managed type
of infrastructure,
and then also have some nodes that are,
Um, on bare metal,
just so not it's not all cloud, um,
and have the ability to
move things around or,
or have the things to be able to,
if there was an outage to be able to,
you know, recreate certain containers in,
in different environments.
But that's as far as I got.
And then I was like, eh, we'll see.
Yeah.
It would have been really
cool if I could have had like one of the,
you know, for Google's, uh, AWS,
Google's Kubernetes have
one of those controllers
built in and then just say, well,
and here's another,
here's another data center out of Google.
Here's one in Amazon.
Right.
let them talk to each other.
And that way, if Google goes down or,
you know,
Google decides to be anti-crypto,
then they're not because
they're fully supporting
crypto and they're, you know,
launching validators now.
Yeah.
Do that.
But yeah.
How often do you have to
manually interfere with a
node that goes down?
Rarely, maybe once a month if.
I mean, for security releases,
the machines get rebooted
and that happens.
On a security release,
some of the pods get stuck on there.
So I have to recognize that
and I get alerted and I
have to restart the pod.
That's a level what happens
when a machine reboots sometimes.
Sometimes the machine itself hangs up.
God knows why.
And then I have to either go into this IPM,
what's it called?
The serial port.
AVM?
Yeah, KPMI or something like that.
I can't remember.
IPMI, yeah.
Yeah,
IPMI and just go in there and figure
out what's going on.
But for the most part, it looks okay.
Sometimes it's, you know,
one of the machines,
Ubuntu doesn't update
properly because something about grub.
There was something in Ubuntu twenty-two
where it had some grub
dependency or something and
I've still got some of the
boxes running uber so
that's you know that that's
one of the annoying things
but typically the
kubernetes boxes most of
them just auto restart so
they'll just reboot
themselves when there's a
security update for certain
things I don't want that so
I've made it so that they
don't auto reboot they
would apply the security
patches but they don't auto
reboot yeah and for that
I'm usually just waiting for um
In Thornode,
you have a thing called churning out.
So the way the validator set
works is different.
But if you're the lowest
active in the active set,
you're the worst performing
or the oldest.
And I think there's another
reason you get churned out.
You might get churned out.
I do that every three days.
And then you get churned back in again.
So when you get churned out,
that's when you can restart the machines,
apply patches,
and start configuring things.
So I'm waiting for two of
the nodes to churn out now
because of things I need to do.
And you also can't unbond
when they're churned in either.
It's a different network
setup and pros and cons and all that.
But yeah,
so it's just a matter of waiting
for those when they get there.
And I can get them to leave, but
That means losing money.
That is not critical.
Right.
You know, I mean, one of them,
I misconfigured it and the
actual four nodes nodes
running on it on a machine
I was going to get rid of.
So the only thing running on
now is that little four node.
And I really don't want to
have the downtime.
So it's just going to sit
there until it turns out.
And then I just delete the PVC, sorry,
the disk volume and restart
it and just reboots itself.
So it's cool.
So no, it's not only,
I find myself way more
interested in using Kubernetes.
Yeah.
I mean the, again, I'm, you know,
the other key thing, you know, is,
you know, the plane trips, you know,
flying hours when there's
no wifi can get nervous,
but most of the time it's fine.
And then, you know,
that's probably the only gotcha I've,
I've got really on this setup and.
whatever reason it's one
direction flying back from
australia has no wi-fi the
way you're in always it has
wi-fi so it's good this is
why I don't go to the
states anymore yeah it's
got nothing to do with like
the leadership or tips no
no we love our fearless
leader uh we can't complain
Because if we do as a non-citizen,
we might not be let back in.
Might end up in fucking Guantanamo.
Yeah.
Or El Salvador.
Yeah.
But no, for all your avid listeners,
I'll be in Paris Blockchain
Week and also in ConsenSys in Toronto.
So I'll be going to those places.
So hopefully we'll catch
some Cosmos people there.
Go to ConsenSys Toronto.
I nearly went to PBW and
then didn't because there
was other... Yeah,
I didn't know anybody going particularly,
so I was just like, okay,
nobody seems that psyched about it,
so I won't bother even
though it's just on the train.
Well, this isn't me.
I think KBW is the one for me this year.
KBW?
Korea Blockchain Week.
Yeah?
Yeah, I reckon it's going to be
Pretty good over this side of the world.
What about the token twenty
forty nine in Singapore?
Yeah,
I think that's the other one I want
to go to as well.
And maybe DevConnect over in
I think it's in Singapore as well.
Yeah.
So KBW is in September.
Consensus Toronto,
if you're going up there,
I might go up there.
I didn't know that was even
I think I looked at this last year,
but that might be something to go to.
I mean, I'm pretty close.
are there multiple consensus
yeah or they just move it
around there's one in hong
kong as well I think
there's usually multiple
but is the one in austin
still happening or do they
just move it I think it was
moved from austin this year
really oh there you go to where toronto
Toronto?
Canada?
Under the assumption that
Canada will be a part of
the United States by then.
The Fifty-Third State?
Isn't it already?
Isn't Canada a state?
I mean,
you can't joke about that because
as members of the Commonwealth,
we're all going to be in
the trenches defending
Canada against America.
It's going to be like...
is turning to be an
interesting world order
that is for sure um yeah
you know definitely
definitely becoming life is
becoming interesting you'll
have to border hop pfc
you'll have to jump over
there to winnipeg yeah I
wonder if mexi works in
canada because all my vpns
don't work with mexi
anymore when I was in
australia it was great I
didn't have to use a vpn
things just worked you know
You could almost just get a
Canadian SIM from where you
are and like jump into like Ontario's, uh,
network, right?
Yeah.
By the border with your cell tower.
Yeah.
I got looped in, um, what was his name?
Um, the Osmos guy who's got the tattoo, uh,
into an encrypt encrypted SIM.
Jacob.
Yes.
He let me in onto an
encrypted SIM discussion.
which is really interesting, but it's like,
uh, my phone is blowing up, you know,
and signal, you know, when it comes up,
I just get fifty messages
on fifty different things
because the one good red
thing about signal is you
need to make sure it's on
multiple devices just so,
so that everybody can see it.
Yep.
Um, but yeah,
he was talking about encrypted SIMs.
There's like free SIMs and all that,
but Canadian SIM might, might be it.
Uh,
just I'll shove my VPN in there and
just see if I can get wifi
signal from where I am.
Wi-Fi works that far.
It makes the most sense.
Yeah.
Or just like stick a VPN
from yourself on one of
your little Kubernetes
containers and then happy days.
That would work.
Yeah.
That's probably the best thing.
And then I can use MexC.
Own VPN, baby.
Yeah.
No, I mean, the Paris and Toronto ones,
because they're tourist destinations.
Yeah, fair enough.
Well, I mean,
the other thing was instead of Paris,
Paris Blockchain Week,
because it's fucking expensive in Paris.
Although actually what I'm
about to say next doesn't
make a lick of sense,
if that's your reason.
But ETHCC is in Cannes.
on the French Riviera in the
middle of fucking summer.
So quite literally a holiday.
Yeah, I mean,
these people go to these
tourist destinations in the name of,
you know, just fact-finding.
I mean,
I would never just fly to a tourist
hotspot in Australia and
just sit there for a week or two.
That's the one cultural
difference that I
appreciate in Australia.
Sitting on the beach in...
God, where was it?
Noosa.
Just watching the birds go by, so to speak,
versus, you know, it was great.
I mean, I could do this.
You know, I had my laptop out from,
you know,
overlooking the beach and just
watching that.
Got you a Maito.
Exactly.
Yeah.
know all of them are
tourists all of them are
foreign and you have you
know a great you know great
cultural appeal you know
tons of different things to
see no sir is a pretty good
spot for bird watching too
exactly big national forest
there right on the headline
is that I didn't realize oh
yeah maybe um but yes uh
this is great and um then
you go down to broad beach
and you see it you know and
the cultural norms you know
change as you go further south
Yeah.
Actually, Noosa is a fair way up.
It is a fair way up there.
It's at least halfway.
Yeah.
Jesus, man.
I just, God, as you go, as you go North,
I'm just looking at the map and I just,
I get,
I'm sweating as I look further
North to like Townsville and Cairns.
It's fucking hot.
Yeah.
I was, we were playing.
So my wife is a native from here, which,
so I live in the Midwest
and it's very cold typically,
although it's like, it's like, it's like,
it's like, it's like, it's like,
it's like, it's like
But it gets down to negatives.
So she's not used to the heat.
So we took her up to Noosa
and that was her limit.
Even that was above her limit.
It was just too humid.
And we were planning to go
up to Townsville and Cairns,
which are even worse.
And luckily they got, well, not luckily,
but they got flooded a week
before we left.
So we just did a major
U-turn and drove down to Sydney instead.
The humidity in Cairns is what kills you.
It is fucking so humid up there.
Humidity.
The humidity.
Also, like, yeah,
quite close to the equator.
Yeah.
I mean, I'm from Melbourne,
so the heat didn't bother me.
It was the humidity.
Just nice dry heat.
I miss that.
Well, you know, sometimes.
Later that day, it's a rainstorm.
That is true.
And a bit later on that day,
you'll probably have some snow.
Yeah, yeah, yeah.
Yeah, I mean,
I've only been up that far
north like twice in my life.
Once to Darwin or Northern
Territory and once to
Cairns slash Port Douglas.
I've been to fucking Port Douglas.
Have you?
Yeah.
I like Port Douglas and Palm Coast.
Go see the Daintree.
Go on that train,
the train that goes into the forest.
You know me, I like a train.
Yeah.
You know, the one that goes up the old,
was it the mining railway?
Was it the, it was a working train,
wasn't it?
For doing something in the rainforest.
I can't remember what though.
It must have been mining.
Not going to lie.
I never caught the train,
but Daintree is a massive rainforest.
Yeah.
I'm sure they've got mining
there somewhere and you
know if australia's can't
dig it out then it can't be
dug out it's like the
karanda tourist railway or
something now like it's it
used to have stations on it
but basically just takes
uses one town is this the
one like on top of the
mountain yeah yeah yeah
yeah that's the one it's
good it's nice it's like
thing to do there's some
good four wheel drive
tracks up right up north
there like the old
telegraph track and stuff
right up there I saw the
arse of one of those what's
the big the fucking
dinosaur killer bird oh
cassowary saw the arse of a
cassowary disappearing into
the bush at one point my
wife was just like that's a
fucking cassowary I was
like huh and by the time I
turned around it was like just this bum
Yeah,
and that's the great thing about Far
North Queensland.
They've got big signs, do not swim here,
you know,
right next to where you can see
some towels and all that
kind of stuff of people swimming.
Yeah.
yeah when I well there's
also when I was there it
was like box jellyfish
season so it was like
really like the signs were
like do not fucking go in
the water or you will die
immediately I was like
fucking australia man what
the fuck like can't do most
of like most of the north
you can't swim in like year
round on the northeast there's fucking
crocodiles and and uh box
generally near a kanji
everywhere in like like the
wharf bits around where
there were like people
getting on and off boats
and shit it was just like
don't go in the fucking
water there's crocs here
and then don't pull up like
don't pull up and you're
like all right yeah winding
up towards if you see the
crocs and the boats are
like that's a croc
literally right there that
log that's a fucking croc and you're like
oh,
it does look a little bit like a
crocodile, actually.
And they're like, yeah,
because it's a fucking crocodile.
You're like, man,
Australia is so full of dangerous shit.
What the fuck?
Yeah, but it's okay,
because if you see a croc
on one of those things,
just run into a school, and they're safe.
What?
As opposed to other
countries where you run out of school.
Anyway, fell flat.
Sorry.
That one's over my head as well.
Just Google, are schools safe?
USA will find out what I mean.
Or Texas more specifically.
Yeah, Texas is a great place.
But yeah,
we went to Australia Zoo while we
were there.
And yes,
they had the croc shows and all
that kind of stuff.
But there was two of the
guys actually trying to
clean out one of the croc
pits while there was two crocs in there.
and you could just see you
know they had their
routines but as soon as one
of them went in the water
they both you know took two
steps back trying to get it
back out of the water so
they could see where it was
before they kept on doing
it so yeah the three is
real crazy stuff so like uh
back on off of australian
danger and what are you um
aside from obviously seeing
the sites which is
obviously not something you
would do on a business trip
Um, but like, what's your main, like,
what's the thing you're
most excited about,
about the conference season?
I mean, it's something we talk about,
like perhaps too much on the podcast.
Like what, what,
what are you interested in?
But is it like,
is it mainly like sort of
new networks and getting a
feel for things?
Or is there something
specific where you're like, I mean,
other than the obvious shit, right.
Which is that we, we all run businesses.
So, you know, networking.
It's networking.
I mean, you know, I missed the Denver,
East Denver.
So catching up with people
if they're there,
seeing if there's new
chains out there and just
seeing what's interesting.
You know,
there's always something going on
in the space and you have
to go to these places just
to hear what they are.
What are the use cases that
like have been interesting
you the most over the last
like year or so?
It's like I've been asleep for six months.
But some of the, you know,
I've always got to ear out
for encryption and privacy
type chains because I think
they're super interesting.
know stuff around finance
you know are there any more
interesting finance
applications I'm still
looking for stuff to help
displace banking so lending
pools and stuff like that
and seeing how people doing
that stuff how are people
approaching credit so those
are the things I'm looking
for to see how people have
done those kind of things
the zk stuff is interesting
but it's like you know it
doesn't solve a lot of the
problems you know everyone
says oh zk will solve this
and they go well no
it doesn't you know I'm
looking at how can a bank
actually run run on a
blockchain you know or do
something on a blockchain
that they do today and zk
doesn't help for a lot of
this stuff um that's
probably the more
interesting thing but then
it's just more just
networking and try to get some new leads.
The number of test nets I'm
running on is drying up and
I need some more and just
need to get into different places.
I'm finding out too late
about new networks and that
kind of sucks because by
the time I find out that, sorry,
we've already got a
thousand applications.
How will you be able to burn
your cash if you don't have
a steady flow of test nets
coming through the door?
There's a thing called thought chain.
um you put it you put your
money in there when it's
four or seven bucks and I
did it with luna beforehand
it was great and I now I'm
doing it with thor chain so
anyway I shouldn't you know
they'll probably piss off
but now piss me off but at
the moment they actually
got a really great story
coming through I'm looking
forward to the uh upgrade
so hopefully that will make
my money back can you
explain rugira to me okay um
Basically,
ThorChain is a DeFi swap space
where you basically swap
coin A on network A to coin
B on network B. What
they're trying to do is add
some finance applications
inside the actual network itself.
So that while you have
access to a Bitcoin or a AVAX or whatever,
you can actually do
something with your coin in the middle,
as opposed to just swapping them.
So they're adding some functionality.
So Rudra was an order book.
It has,
there's some perps going on with
that and various other
finance applications that
they took off the Kudra network.
And they're basically
bringing it across from a
technical perspective.
They should be running with
Thor chain security model.
So they'll basically have,
that level of network activity on there.
I don't think there's another Thor chat,
a Rudra node that you run.
It's basically the Thornet
node operators will just
run the Rudra node if they
need it and so on.
So it's a DeFi chain.
So it's basically all the
stuff that was on Kujara,
they've upgraded and
And it will now work on a thought chain,
which is again,
it's a thought chain under
the covers is a SDK, uh, network.
Hmm.
You know, they've got other bids,
but that's the majority of it.
Because Kujira is
effectively abandoned now, right?
Correct.
Yeah.
I've not seen any dev
activity there in like two years.
So is it L two?
It's not L two, right?
It's not like a CS type structure.
No, it's, I think it's actually,
I don't know exactly how
it's supposed to operate.
I haven't paid that much attention.
uh to the technical things
about how it will work but
it they do something okay
it could be it could be
running this it could be
running it could be an
l-two that might be the
easiest way or there might
be some there's some sort
of finality on thor chain I
guess this is what has to
happen right but yeah but I
believe so but the rugira
but there is a separate
blockchain like so they are
still keeping along some of the kujira
chain specifics um and
running I guess the new
site on that on that chain
is that right yeah it's you
know definitely because
they've got smart contracts
and they'll be very
permission smart contracts
so could you had some
permission ones but I think
it will work like like that
they'll have like their own
their vm in there and they'll run that
don't think there's any
custom modules uh you know
they're really close I
think three point four is
supposed to be the one that
they release on because it
was like one or two little
things strange love is
actually building it
they'll be the right people
to talk to yeah yeah I
think yeah it's not just
it's not just thor chain
integrating a smart
contract vm and then
launching kajira's
contracts and permission
mode then I don't know it
could be but my
understanding was it
slightly different it's more than that
Yeah,
I think Jack and team have been
working on Thor for about a
year or something like that.
It feels like it's been a while.
Yeah, that's pretty cool.
I don't think I even knew.
I mean, until the Kujira thing happened,
I didn't even know Thirteen
was a Cosmos Wasm chain.
But yeah,
it's just a standard Tendermint chain.
There's no separate execution layer.
It's all Tendermint.
they do so that it's a
customized tenderman
heavily customized they've
changed some of the modules
for example the staking
modules completely changed
the governance has
completely changed yeah uh
so they started off with
tendermint and they went
their own path yeah um so
they've got that the key
thing which they do which
is interesting is they have
security vaults so so
security vault is eighteen
notes so they have up to
one hundred and twenty
Um, nodes running on the network.
I think there's like,
running at the moment.
Yeah.
And they are divided up and like,
of the nodes, uh, create a vault,
a multi-sig vault.
Mm-hmm and that's where all
the funds are locked.
So when you bond,
what you're doing is you're
actually contributing to that vault.
And when you churn in and out,
they change the keys of the
multi-sigs and add the new
pla add the new people in and out and.
and all that stuff.
So that's the key thing.
And they run something called Bifrost,
which does that.
And that's, I think Bifrost, again,
I'm not,
I haven't gone too far into the
weeds technically because I
just had a hundred other things going on.
But Bifrost is the one which
actually signs all the
packages and talks to all
the other networks.
So that's the agent.
So for example,
when I'm doing a Bitcoin trade,
it's the Bifrost,
which is actually
communicating to Bitcoin
and talking to it.
I think that's how it works.
That's pretty interesting in
terms of having the,
the kind of in and out of that, of that.
It's kind of like after the
helium set went live, that was,
I forget what they call that proof.
It's proof of it's not proof of stake.
It's proof of,
I don't even know what the hell it is,
but the same type of thing,
like you got elected in and it was there.
It was random.
Like you had, I forget what it was,
forty one nodes or
something like that that
got elected in for a
specific time period.
Say it's eight.
That was short.
It's like eight hours or
twelve hours or something.
And then you produce blocks
for that eight hours and then and then
It would basically take ten
or twenty five percent,
kick them out and would
randomly pick another
twenty five percent.
So you could stay in there
one time and get kicked out
or you could stay in there
for a month if you were
lucky enough to get not
kicked out of that twenty five percent.
And then that that consent,
that quorum just basically continued.
Right.
And you got elected in.
If you're a poor performer, you got booted,
all that kind of stuff,
which is actually it's a
pretty cool model, actually.
Yeah, I mean, it was very fair.
That was a totally fair
model in terms of if you if you earned
uh commissions or if you
know if you were in that in
that consortium or not that
was actually pretty cool
yeah I mean for me uh it's
similar what I'm getting
though is there's nothing
stopping you running
multiple nodes like any
other network and just for
cost savings you have to
sometimes but the churn in
and out can be bumpy
Yeah.
So because they have to all
sign that key or sign the
new vault and they've got timeouts,
if one of the existing
people who wasn't going to
get churned out misses a
signature or has something bad,
then you have to wait
another hour to try again.
So, for example, the last one,
it's gotten way better,
but the last churn,
I think it started at eleven p.m.
and I woke up at seven a.m.
and it was just finishing.
Yeah, it's crazy.
Yeah.
But it came in and you still
This has got nothing to do
with the actual
functionality that the chain provides.
It's still basically doing
trades and all that stuff.
It's more just the nodes
themselves have just taken
a long time to agree.
That's irritating.
And also, if you're trying to join in,
there's a slashing.
So if you miss a signature,
you get slashed,
which if you're in the active set,
doesn't cost you anything.
But if you're not and you're
trying to join,
you actually get some of
your bond is taken away.
So if you screw up your configuration,
trying to get in, you,
it will cost you money.
Yeah.
That's cool.
I like, I like, I mean,
I'm not saying that's perfect,
but I do like the idea of
like having some different
consensus engines that, that provide, uh,
you know,
kind of bespoke functionality that,
that meets the need of that chain.
Um, cuz proof of stake, you know, we're,
we're huge fans,
huge proof of stake in this group,
especially, I mean,
distributed proof of stake
is like really the end all be all.
And they have very different
opinions to how governance works as well.
So you haven't got these
components where you have like,
fifteen percent of the
staking power who controls the stuff.
Every node is one node, one vote.
But they have limits to how
much bond you can actually
have on a node as well.
Sure.
Yeah.
It's interesting.
Like for the governance, is it one node,
one vote?
Yes.
Okay.
That's good.
It's been interesting seeing
evolving of governance and
different models for staking recently.
And that one node, one vote, I think is a
I think that's a good thing
for a chain rather than the vote weighted,
staking voted weight is
what I'm trying to say.
Yeah.
I mean,
if you really want your group to
have a say,
there's always these
outweighed nodes which sort
of control most of everything.
Yeah.
And it's also what the node actually,
what type of things you can vote on.
So most of the things you
vote on as a node are
operational in nature.
It's not about grants or you don't,
there's no treasury that
you have the semblance of control over.
Yeah.
um well man it's been
interesting having you on
uh and thanks for coming
and joining us um we uh
have never done an outro
before so I'm not even sure
how to do it okay I'll just
leave then all right