Welcome to Game of Nodes, a weekly podcast from
Independent Validator Teams. Hello and welcome
to Game of Nodes, a bi -weekly and occasionally
slightly more than bi -weekly podcast from Independent
Validator Nodes. This week we have Brendan from
Whisper Nodes with us, old friend of the treehouse
of the bunker. of whatever it is, whichever nefarious
location that we are doing all of our plotting
at any given time. And recently joined Flipside,
right? That's the news in the in the Brendan
bunker. Yeah, happy to be over there kind of
getting away from the validator world and into
the data world. The getting away from the validator
world into the data world sounds a little bit
like out of the frying pan into the fire. Is
it, I'm guessing in your day, in your day to
day right now, just like everybody in tech, you're
hearing the words, words, AI about a thousand
times a day, right? Jesus Christ. Yeah. Yeah.
And I'm using, and I'm using this shit too. I'm
trying to anyway. Yeah. What kind of, what kind
of stuff, what kind of stuff for? Just like automating.
All day to day actions that I can. So with flip
side, I'm doing sales and account management.
So, you know, I have a CRM and I kind of have
a lot of follow ups to do conversations to track.
So I built an MCP server and then hooked that
into Claude. So basically I can do everything
with work by clicking a button, speaking to Claude
and then Claude will go use the MCP server to
make a bunch of API calls. So it's pretty cool.
That's pretty sick. Yeah. I guess that's what
happens when you hire somebody that used to be
like running essentially mission critical stuff
for years in a stressful situation to do something
that is not strictly technical. It's like, well,
this is like a lot less stressful than everything
else I've been doing. I guess I'll just come
out with a great way of automating this that
takes some of the pain away from my bollocks.
Yeah, I feel like we like to over -engineer everything,
hence whatever the hell Schulte has going on
in his background that he was talking about pre
-show. My mini data center, yes. So if you've
got a Lavender 5 testnet participation coming
up, it's going to be, the action's going to be
happening from behind Schulte, right? Most, well,
it depends on, probably, probably, probably.
You know, I'm not good at business like the rest
of you, but I remember GCSE business studies.
You get those fixed costs and that's all upside
baby, right? That's how business works. That's
what I've heard. Yes. Um, how much is, how much
has your, if you don't mind saying like how much
has your, uh, your mini data center set you back
so far? Uh, Well, it depends on what you build
in there, right? So like just in hardware, about
$12 ,000. Um, yeah. And most of that is in hard
drives. So the second you say hard drives, you
know that null is going to appear on the cool,
right? That's right. Well, like this one is 700,
$800. And there's basically that this machine
right here is full of them. That's for the archive
nodes. Um, so yeah. So quite a bit. So just for
hardware, 12 ,000 servers themselves. The pricing
for the service themselves are fairly cheap,
but RAM and hard drives is just ridiculously
expensive. And then on top of that, I have new
networking gear because previously I couldn't
create VLANs in my network. Now I can, so now
if someone decides to jump into the servers,
then they can't access everything else on the
network. And then I'm also working on building
out basically a whole house. Universal power
supply so that if power goes out the nodes would
still be running so I should have in theory upwards
of a week of runtime if all power goes out and
then there's like redundant network providers
and it's it's been a whole project and a lot
of fun this this is definitely the sort of project
that somebody does when they're finally getting
some sleep uh so I can yeah I can only assume
that what that is what has happened and so it's
just unleashed this tidal wave of what seems
to be to be an exhausting amount of energy and
planning for for what is to be fair a cool project
but it yeah i mean it's been fine um a lot of
it is like so like with the devaluation of the
dollar we are paying about three thousand dollars
more per month than we were like six months ago
with no change in servers right so like twelve
thousand dollars sounds like a lot until suddenly
that's actually four months of server costs And
so basically, as I'm shifting things over from
servers under these and other servers, it's going
to eat up all that cost really quickly. Yeah,
it kind of makes sense when I used to work for
like agency type setups, we quite often had like,
you know, customer type, you know, like a staging
server for a customer or like pre production
or whatever. be like, hey, what's this thing
under the desk that's beeping? It's like, oh,
that's like Jeff's old server from home. We just
stuck eight hard drives on it and juiced the
RAM. And that's staging. When you log into staging,
it's the one on the desk. Don't fucking unplug
it. So I feel like this is a well trodden path.
I mean, we'd actually looked into it in our shared
office. But unfortunately, in terms of our lease,
we can't just rack up. stuff and run a data center
from our shared office because Well, so so there
is a thing in the police basically saying that
we can't run a data center from that But it is
quite specifically a data center. So I wonder
Yeah, it's like there's like I think it's because
I think it's because the the whole building's
networking is like obviously done By it's managed
by the people in the building Um, and then you
have like the networking and everything for your
unit and blah, blah, blah, blah, blah. I think
it must be something to do with like the network
switch or damage their hardware or their use
somewhere. Like there's some, there must be some
reason that they're a bit iffy. I mean, it might,
it might just straight up be that they don't
want to fire. And they, well, and power's really
expensive, right? I mean, what do I pay for that?
No, but we, we, we pay for it as part of our
realm. We have a power agreement. Oh, actually,
I say that very confidently. Maybe we do. Maybe
we did. Maybe there is something in that. Maybe
that is it. Are you burning a lot of power then?
Not actually all that much. These servers are
running about like 30 to 40 Watts each, but that
still means there's a constant 250 watt draw.
So six kilowatt hours a day. So a dollar a day.
Oh, wow. I wish you I wish we could run more
stuff on them. Do you remember when the people
were like home hacking those Raspberry Pi blades
where you did like a cluster of Raspberry Pi
is in a big stack. That was the coolest shit,
man. That was, yeah, that is actually, this rack
is actually intended exactly for that. This is
by desk pie, the desk pie team. So it's intended
for creating racks of raspberry pies. And I actually
have a bunch of modules that come with it that
I was just like, man, throw that in the corner.
Did you build any weird stuff with it? Like my
buddy actually just started playing with those
raspberry pies and he took an old cell phone
and he made it into like a nanny cam. So like
I wonder what's going on on Schultz's shelf over
there. He's definitely got some some spyware
or something. Well, no, mine's pretty run -of
-the -mill. It's all it's literally just mini
PCs. They're effectively laptops without screens
really. They're even running laptop chips. So
which are surprisingly powerful these days. So
like the headster 7950 axis, right? These are
basically as powerful as them. And that and those
little cubes, they're technically a little bit
less powerful because they have fewer cores.
But in terms of like the core themselves, it's
they're absolutely insane. Like we can you can
stick on these little things is outrageous. Wow.
Interesting. I can see this is this how you get
to it, right? You're just like, well, the facts
you're providing are interesting data points
for service that I do not need. Um, yeah. Well,
I looked into servers before and like talking
to artifacts, he was like, yeah, well, you can,
you know, get a server for $2 ,000 and then you
can get your disc and the Ram for it. And I was
always like, that seems pretty ridiculous. Each
one of those is $300 with the CPU. So you just
supply the Ram and the disc. So H one ends up
being, you know, six or $700 instead. That's
a much better proposition. I mean, I've got three
years of three you have rack space, like literally
here in my desk, because I took out the old Roland
JV synth. I've been getting rid of some musical
kit that I haven't been using. We still got all
the modular shit over here. Don't worry. Don't
worry. Longtime fans of the podcast. We still
got lots of wires and synthesizers and stuff.
But the one of the Roland JV synths has gone.
And I've got three U of rack space and an isolated
power strip. So maybe I should set my house on
fire by putting some racks in here. Huh? That
would be cool. Maybe foolish. I don't know. Suddenly
my interest in this crazy scheme has been reinvigorated.
What's the Homer Simpson line? I find your ideas
intriguing and I would like to subscribe to your
newsletter. Is, is that, I think that's how he
ends up in the cold, isn't it? That's the line.
That sounds right. That's the, I feel this is
like something you said, but would know. Um,
so yeah. Okay. Cool. So big, big battery, big
solar array. Does that, does the viability of
it? Um, I mean, you're in, I mean, you're in
the States, but I think you're in a sunnier,
it's sunny away. You are then, you know, England.
No, um, it's actually about the same as where
you are. In Oregon? Yeah. It's mostly cloudy.
He gave away his location now. Now we know where
all his servers are. Somewhere in the very small
state of Oregon. Yep. Yep. Somewhere in Oregon,
near avocados, near coffee. It could be anywhere.
Absolutely anywhere. Yeah. No, it's quite, it's
generally cloudy. It's like cloudy something
like 210 days a year, I want to say. Oh shit.
Okay. So it's like anti Denver. Yeah. Yeah. So
basically what that means is I, my solar panels,
I had to go really overboard in order to actually
be able to cover my energy usage. Yeah. We've
got, I think, uh, I want to say we've got like
10 or 12 on our roof. It's not like a massive,
I perceive it's probably a smaller house than
yours. Cause you know, again, England, um, and
we like, we did not obviously cover up. But I
think again, part of that problem is also battery.
Like we don't, we actually don't have a battery
in the house because the way, at the risk of
becoming gaming nodes, the, the energy subsidy
and government policy podcast. The way it used
to work for getting solar panels in this country
was simply to incentivize it. You sell your power
back to the grid. So every house, probably the
same as you guys, right? Every house has an inverter.
Mine is not under my desk. I think it's in the
toilet there. It's just under the toilet. Great
positioning. But you can fit it where you can.
But it does mean that there wasn't like at the
time a big incentive to put in batteries and
then that came like the technology for batteries
kind of came along like maybe five years after
the original subsidy for solar panels. So a lot
of people that got solar panels got solar panels,
then batteries became good, but it wasn't like
necessarily joined up for everybody. Some people
who got solar panels later got batteries and
solar panels, especially with like electric vehicles.
It makes a lot more sense, right? Yeah, so so
the way my setup works, the solar panels are
not connected to the grid, they connected directly
to the battery pack. And so the battery pack
then basically pushes the power to the house,
but it stops there, it can't push power back
into the grid, which has ups and downs. The good
thing is that, in essence, the house chooses
the battery first. So that's why if the grid
goes out, it's literally zero millisecond turnover
time for like the house to have power because
it's powering from the battery. But you can't
get any payback if you overproduce. Okay. The
guys that set my solar up originally, they forgot
to put the inverter or they they forgot to put
a switch. So when I got hooked up to the power
company and they actually put the grid through
my house, it blew all my solar and all my inverters.
No, no, that's insane. Puerto Rico things Chelsea
Chelsea knows I don't want to talk about Puerto
Rico to be honest with you. I'm still like recovering
That was a tough like six months of my life.
We're roughing it out here, man I'm just still
angry that so So for for dr. Frey over here.
There was a proposal after a hurricane Maria
I think where they were going to in effect get
rid of a central power plant in southern in Puerto
Rico and instead do micro grids because Puerto
Rico has sun basically all the time, right? Except
like at 310 PM, which it pours down rain for
a minute and a half and then it goes back to
sun. So they're gonna set up like 30 micro grids
in the island, all paid for the government. And
it was like totally reasonably cost. And then
there'd be solar panels, you know, all over whatever.
So I don't know what happened but something with
the US government was like absolutely not we
can't get rid of fossil fuels You guys are so
funny and they vetoed it and so now there's an
even worse system now Yeah, so they went back
to the original power plant and they did something
and we were losing power weekly Oh, yeah every
week. That's why I had to get the solar So no
servers in my house I was going to say that was
going to be my next question. Was it, uh, was
the motivation for getting the solar brand and
the, uh, uh, the, the whisper node, uh, test
net set up, or was it just like, you just tired
of not having wifi cause we're sitting in a dark
room. Pretty much. Yeah. It was like, if I needed
to work, I needed power. Yeah. Okay. That figures.
Right. Um, solar is relatively cheap in Puerto
Rico, right? When I talked to my landlord there,
she was like, Oh, yeah, I've got this big old
array and it costs like $1 ,100. I was like,
why? Like 10 % of what it was. I think I paid
like 20k for mine, but I'm like, pretty much
off grid. So it's not too bad. I think 26 panels.
Yeah, yeah, yeah. Big battery for that. So, um,
god damn, I'm just sorry that just that's just
That's back to like small European houses and
small European perspective. You're like, that's
a lot of fucking. There's like a, there's like
a house I cycle past on my way to work most days
where they've obviously decided that they just
want done with paying for power. And they have
literally like all the way down their roof and
then kind of on the side of the house. And I'm
kind of wondering whether it's actually illegal
because doesn't seem like the sort of thing that
somebody would say, yeah, you could just like
string up solar panels inside your house, but
on the side of the house, and then their entire
front garden and then all down the side of the
house visible from the road is just solar panels.
And they've sort of been set up just I think
so a person can like squeeze sideways past them
to get to the rear of the house. But I'm like,
I always like cycle past it. I think like if
you need to like I don't know, like take a sofa
through your back door or carry a ladder around
the back of your house. Like how are you going
to do it? How are you going to do it? Like there's
no, I guess the answer is probably try and carry
it through the house, but like inconvenient.
I mean, I guess they obviously thought of that
and they went, yeah, actually the number of times
I had to carry a sofa through my house, probably
one in 20 years and free and you pay for this
and get free power. So probably it's again, previous
comment don't understand business. There we go.
So yeah, speaking of speaking of not understanding
business, that wasn't really a good setup. But
so Brendan, what made you what made you decide
to go and work with flip side? It's kind of the
perfect marrying of all my web three experience.
So I used to work for Secret Network doing growth
stuff very closely with Tor Bear. uh head of
marketing and I ran the ambassador program over
there and kind of just through that whole experience
saw how projects don't really operate like web
two projects where they are using data to inform
their decisions they're just kind of like throwing
money like all over the place and hoping that
they have roi somewhere so uh yeah and then obviously
got into running whisper node after that validators
Wait, I thought WhisperNode came before then.
I didn't realize WhisperNode was after you started
then. Oh yeah. Whoa, okay. This changes everything
for me. So how did you start with Secret Network
then? Like what was the catalyst there? Man,
I was an ambassador back when they were Enigma
in 2017. Like after the ICO craze back then.
Wow. Okay. So you are, you are OG or as OG as
you can be in the Cosmos kind of space. Yeah.
Yeah. Well, they were, they were one of the first
kind of Cosmos app chains. They were, I think
they had smart contracts second only to Luna,
if I recall. Wow. Yeah. That would make sense
based on that. Yeah. I think, I think that's
about right. Something like that because they,
because they were both them and Luna were on
Uh, like a pre version one stable version of
Cosmism, weren't they? Yep. Exactly. Yeah. It
was, yeah. And Schultze just, uh, learned the
validator game. I didn't know what the hell I
was doing, to be honest, when I first started
running it. Uh, and, uh, yeah. Figured it out
over time. Obviously had Ghost join the team,
uh, at some point in November, 2021. And, uh,
yeah, just kept. learning more about the tech
side of things. And flip side was like this perfect
marrying of data driven growth, which I could
understand the value proposition right away having
kind of worked in growth in web three. And then
they also have an infra arm where they run validators
for the projects that they're doing growth with.
So you are not doing the validator operation
side, you are just doing the data. Okay. Do you
criticize the NoteOps guys? Are you like, you
bunch of fools? I've been doing this for two
years. I know, right? No, they're, they're pretty
good over there. They're like really solid guys.
Um, but it is funny having kind of people that
are way more technically knowledgeable than I
am having, you know, web two sys admin experience,
you know, spitting up all kinds of Docker containers
and over again, over engineering, like everything.
And then they're coming to me asking for like
advice on running Cosmos nodes. And I'm just
like, why? You hit start and make sure you have
a seed and then you're good. Yeah. Yeah. Yeah.
It reminds me of like when, when I mean all the
shit that like I look at now and I look at, um,
the company's past financials. And I go, well,
that was kind of thick. But when we first started
running nodes, it was like straight off the bat.
Like, it was, I think the first nodes that I
ran were when I had an enterprise contract that
was winding down at the time, right? So I was
going from like day job where it was all like,
to be fair, what I've been doing was actually
architecting a system that was completely on
demand. So it was probably the cutting edge platform
-wise of what that business had in their entire
tech stack. But it was like all AWS, all serverless,
on -demand databases, on -demand nightly snapshots,
all of it just completely, just like magically
automate all the things. If nobody touches any
of the websites, basically nothing's going to
cost you any money. uh if it gets a lot of traffic
then suddenly it's going to get expensive kind
of thing blah blah blah like um and it did it
was like a covid response for a lot for a large
company large organization uh project and to
be fair like the costs and everything on it were
pretty good like the tech was not gonna lie pretty
awesome the code was the code was legendary it
was architected by this great guy um and then
came straight out of that into Cosmos and was
just like, yeah, I guess we just run this on
AWS, right? How bad could it be? And it was just
like over -engineered, absolutely everything
overspent on absolutely everything for too long.
And you know, like, I always just tell the story
now of like, for all of the monitoring and automated
monitoring and backup monitoring stuff that we
used to run in that setup, like had this moment
of clarity when, I can't remember which network
it was now. I think it might have been maybe
when Mars. No, it wasn't. No, it was the second,
you know, cyber attack. A monitoring stack got,
um, like, you know, responses that it couldn't
really understand. And the monitoring just like
fell below, yeah, some threshold monitoring,
which went, they really know what's happening.
Um, notify and like, didn't even come through
for page GD because I was asleep and I had like,
or whatever. And I just like, notify me that
something a bit weird happened. Fuck off. Meanwhile,
I had like this little, um, bit of code that
call AWS Lambda. I don't think it's ever actually
cost us any money because it's never breached
the whatever it is million executions you need
to actually pay any money for it. And it just
like, it just says, Hey, I'm alive. Hey, this,
we used to have this on our Cosmos Nodes. I was
just like one, one morning I wrote it in like
five minutes because I was paranoid about Cosmos
Nodes being down and just goes, Hey, I'm alive.
Hey, I'm alive. Hey, I'm alive. And if it doesn't
say, Hey, I'm alive for like three minutes in
a row, just It just called me, just called my
phone. And we have a slightly more complicated
version of this now, but it just calls the ops
shared phone number, which is like gets really
to everybody. And that fucking caught it. That
stupid piece of code. It's like a bash script
at the time. It was like bash script running
in a crud job. Hey, there's nothing wrong with
that. All the other shit, all the over -engineered
stuff we had, we had like, what was it called?
The AWS ops work observability thing on the,
on the boxes and everything. They call us like,
they were like, yeah, the Ram looks okay. I don't
know what the fucking problem is. Like, yeah,
the Ram looks okay. Cause there's no transactions
going through that server. Like, Oh man. So yeah,
that's a very long rambling story to say over
-engineering shit. I feel. I feel that deeply.
I feel that deeply. Do you guys, did you run
like, or do you run? There's like that really
complicated diagram out there for Cosmos with
like the three sentry nodes and like three horcrux
signers and then like a private, a private subnet
and like all this shit because the engineers
at Flipside were actually asking me about that
chart. And I was like, no, man. I'm a small node
operator. Like I can't afford, you know, I can't
afford three century nodes, a validator node
and three Horcrux nodes for every single network
I'm running. That's crazy. Yeah. So first of
all, no, I don't. Second of all, I think that's
strange love. The makers of Horcrux have the
most interesting setup for their Horcrux. They
keep their Horcrux nodes on the node of the signer.
And then they only have one signer look at one
node. so it basically gets rid of the the one
downside of Horcrux which is latency and they
just like pound their own node it's absolutely
insane and the way they do explain to me in the
way like Kubernetes interacts with it that's
off topic but it's interesting and yeah some
wild guys their setup is pretty cool um we used
to run centuries for all the Cosmos nodes networks
we did um that's why ultimately we didn't make
much money Right. Yeah, we were doing it in the
beginning, and then we were like, this is not
sustainable. We're probably losing money at this
point. Yeah, I think that's basically why we
lost money on StarGaze, despite being there from
the start and having good delegations. Because
for the first maybe nine months of the network,
we were running. Hey, look at this guy. we were
running a full century architecture on AWS and
AWS, like, you know, I think we were, I think
we were running for the centuries, like we were
running smaller boxes, but like we're still running
an R5XL. Oh my God. Yeah. Well, cause the, the
cause and whys and chokes, like you just, you
need, you need that is, is brand bottleneck.
You just need like 32 or 64 or somebody can do,
somebody can do the, um, the Ram bottleneck attack.
Do you remember like, were you on the early Juno
test nets? Do you remember the one where me and
JunoSwapBen took down the network while we were
drinking one night and then told all the validators
what we'd done afterward? We just like sent a
message around saying, hey guys, we had a thought.
We tried the thought. We did the thing. Sorry,
we need to restart. We need to regenerate this.
I vaguely remember that. I was like going from
test net. What was it two to three or three to
four one maybe one to two actually it was very
early yeah it was like okay i do vaguely remember
i think it might have been just post juno launch
but like literally like three or four days post
launch something like that yeah um on the test
net um but that that attack is what happened
if you if you're one of the nodes that's ram
is too was too low at least in that first period
of time when cosmism was out in the wild it was
very very easy um, for a developer to work out
a way of triggering that attack, that particular
attack. And that's why a lot of the networks
at the time, um, we shared what we had done with
other Cosmos networks. And that's why a lot of
networks at the time sort of said, Oh, actually,
um, it's not immediately like, it's not really
an attack per se because the transaction is valid
and it can get through. But if you're, if, if
enough of your consensus set doesn't have the
brand. they're going to choke, then the block
will stall. And it's actually quite difficult
and costly to get that back online. Um, and what
we of course found was that when it chokes, some
people start messing with their nodes and their
double sign and the whole thing goes to fuck
really quickly. Um, because they're down or they're
like, why am I down? It's like, well, you're
down because you know, it's choked, but if you
mess with it, it's definitely going to be down
down. Ah, good times. Welcome to the party, Noel.
Hey, yeah, I'm like mid -transit driving back
home to Brisbane from another place, so I stopped
at the pub. What does that sign say? That's Tooey's.
Tooey's new. Yeah, Tooey's. All right, I'm gonna
see if I can find it. And this is, if you're
around, this is the top pub in Urella. but yeah
it's cold as fuck yeah not as cold as like that
other place where i pulled up outside of a cafe
once but it's still pretty cold um was that the
one in america where you turned off the podcast
yeah yeah that one yeah hey brendan what's going
on no wait so you're not in australia yeah yeah
yeah i'm in australia yeah i'm just uh oh man
i'm in transit driving um i always picture australia
as being blazing hot i didn't know it gets cold
there Off or upside down. It's always surprising
when he's actually writes it out Does the was
the water rotate the opposite way when you flush
the toilet kind of thing the opposite to the
northern hemisphere? Yeah Yeah, wait, it actually
does. Yeah, you know that No, I just assumed
it was a joke I thought it was a joke too It
actually does physics hmm, huh? Okay. Yeah, it's
I don't gravity Rama, Americans are stupid, man.
I didn't deserve that. I see you've played knifey
spoonie before. So in the summer time it's hot
and in the winter time it's cold, if you can
imagine. So it's not like, you know, Northern
America winter cold, but it gets to zero and
that's enough to freeze my balls off. yeah i
mean i am wearing shorts right now and it's i
think maybe eight degrees um which is really
shrinking my balls um but yeah what's been happening
i'm a little bit late because uh those road works
so what have we been talking about uh yeah my
data center uh talked about oh the lab puerto
rico schultz and i have been like talking about
the lab in the in the side chat and uh is it
like is it product complete now? Did you get
your second incident? You're good in today? You're
good? No. So funny thing. Yesterday, the network
engineer actually came out for laying the next
line. And I'm in a flag lot. And so like, I'm
basically there's a mode of houses around me.
And I'm looking, I can see out my window and
see down like down the street, where he's looking
at the power line. And he's looking at all confused.
And I was like, Oh, I should I should I should
go see what he's doing. And I walk over there
and he's like, this isn't going to happen. I'm
like, well, what's the problem? Like the fibers
already ran. What, why is this an issue? He goes,
do you see that port up there? We can't fill
that port. We can't fill the port. Like I've
been talking to you guys for months. Like, why
is this, why is this an issue? Um, so I have
to get DSL as the backup internet rather than
fiber. That is not ideal. Yeah. Um, I guess the
engineers are, are. They have to take a cost
analysis on if whether it's worth it and me being
a peon It won't be worth it. So over here if
you're like if you know it if the fiber doesn't
run out the front of your house You have to get
a build but they just charge you that it's like
it'll cost you ten grand Well, I've got fiber
I've got five gig internet right now. It's just
I wanted the backup and the backup would have
been 500 Meg. Oh, so you got your business internet
I have my business internet. Yeah. I was trying
to get another backup. Yeah. Yeah. Yeah. Well,
so I'm not that concerned. I'm a little annoyed,
but it's better than a kick in the nuts, I suppose.
Um, yeah, pretty much. Well, it is a kick in
the nuts, but it's better than a legit kick in
the nuts. Um, yep. So have you like, are you
running anything in there at the moment? That's
like noteworthy or it's just testing stuff. Sorry
about the background though. It's done like on
the side of the highway. I can hear that. Have
you got a beer or you got a coffee? I have a
coffee but I did just notice a dude like walk
behind me and hit that door real quick and then
fuck off again. So it's uh 8 30 in the morning.
But the gentile looked reasonably disheveled
so I am certain he was after a beer. I do have
some prod stuff going right now but not a whole
lot just yet. Ethereum nodes are actively getting
migrated now. Archives are almost entirely migrated
really low priority master servers have been
migrated over or like actual validators haven't
migrated over So you said your emissions probably
you said your aetherium stuff you're moving over.
So are you like having? backup nodes elsewhere
for that Yeah, so we have I think we're running
seven aetherium nodes. So basically we're decommissioning
like the off -site server ones that are in the
US and She migrated me here. So most, most time
if we have us servers, they're on the East coast.
So add in 50 milliseconds of latency, which I'm
willing to, to, to, to squirrel away by saying,
Hey, we, we self host it in our, in our data
center boys. Is there, um, is like, why seven?
Are they all main net? Sorry. What was the question?
You said you have seven Ethereum nodes. Are they
all main net? Why seven? Uh, so a lot of it is
like basically where they are in the world. So,
um, we have three regions, right? So in the two
in each region and then one test net, two test
nets, two tests. So eight nodes. Right. Is it,
but like, are these for validators or are these
for, um, no, no, no, that's our PCs. Yeah. Yeah.
Yeah. Cause it's like, it's like, it's a lot
for a validator. Yeah, yeah, no, no, no, that'd
be that'd be way overkill way overkill. Well,
I'm I'm told that they don't Necessarily work
very well when you've got them load balanced
either. So Because I think like the validator
node subscribes to a a RPC. So anyway, that's
like we've got multiple but they're sort of in
like active backup For yeah for like, you know
SSV and mobile and whatever right? Yeah, right
but to be honest like Ethereum's so stable, I
don't think it's ever fallen over on me. So the
only thing you would really have is the hardware
issue, I guess. It will run out of space, but
it doesn't even use that much space. Is it pretty
lightweight to note, to run? I was in Dubai at
the token conference and in the Solana community,
they like to talk shit on the Ethereum community
and say... The priority is allowing, you know,
decentralization so you can run nodes on toasters.
You can run a theory on a fucking potato if you
shove an SSD in it. Yeah, 100%. Like I, in the
computer science department, I found that there
was an academic blockchain, there was an academic
blockchain project that a bunch of universities
were running. It was a fork of geese and I literally
just got forwarded an email being like, Hey,
this thing is down. And it was from like the
maintainers at some European university saying,
Hey, your university's node is down. And it was
one of those where then this got forwarded on
from like the head of the department to somebody
else eventually washed up on my desk. Cause somebody
went, Hey, he's a blockchain guy. He'll know
how to figure this out. And I basically, yeah.
End up finding a I want to say it was an Ubuntu
18 .3 LTS, something like that. It was a fucking
old server, completely out. It was like EOL LTS
server with just this stupid ass node on it.
And the only reason it had crashed was absolute
potato. And it was like whatever server under
the desk in the IT service help desk room that
they had been prepared to to give these academics
to just casually run a blockchain. They're probably
like, uh -huh, yeah, sure. We're not even gonna
put that in our firewall. We're gonna use Jenny,
the little server that could under that desk
there that's covered in coffee stains. And yeah,
and the only reason it hadn't stopped working
was because the disk was full. And I was just
like, oh, yeah, it's full, the disk full. And
they're like, oh, right. How do we fix it? I
was like, I... I literally, this server is fucked.
We need a new server with a slightly bigger disc,
but it will just work. And yeah, somebody plugged
in another hard drive and just, it continued
fucking working. And I was like, love a fix right
after doing a thing. Amazing. But Ethereum man,
like it, yeah, we'll just run on anything. And
it is so, it's like, you just, it will kind of,
it's like a Cosmos chain without Cosmoism. You're
just like Cosmos chains without Cosmoism. You're
just like, it's gonna keep running who cares
um it's just that it's just that it does use
a lot of space though i think ours are two terabytes
maybe yeah that's probably better right they
even this academic blockchain that was like empty
blocks it was still like i think it had blown
up at night like it busted through like i think
it was a one terabyte drive and yeah yeah i mean
but then you know flip sides like solana guys
like shitting on ethereum you're like well you're
your blockchain is hugely resource intensive
still crashes a fair bit and you know yeah whatever
requires the dark web in order to work correctly
or like 30 percent of their traffic through like
one provider or something like that yeah yeah
which provider is that at the moment i can't
remember the name but isn't it that one that
was like recently purchased do you mean like
one uh like hardware provider or one node provider.
Like one note provider, I think. I think it's
those dudes that we were talking about last month
that got bought for some stupid amount of money.
Sounds about right. And then all their traffic
goes through Jito too. So if basically Jito goes
down, no transactions will hit the chain. Decentralization,
baby. This shit works. I won't, I won't hear
a word against this decentralization. Um, fuck
anyway, um, circling background to cause and
wasm, you know, I feel kind of heartbroken that
cause and wasm is basically going away. I think,
I think the death knell has been, has been wrong.
Yeah, but it's, it's on the hub. It is on the
hub. That doesn't really mean anything. It's
on the hub, but nothing is, nothing's being built
on it that I'm aware of. Well, isn't the hub
going for missionists? I mean, this was literally,
we had this, we had, um, we had mags on the podo
you know and there were we did there were rumblings
about things to come on the uh on the cosmos
hub using cosmism potentially permissionless
cosmism although i guess we'll believe when we
see it right so the uh the sip three for say
uh i don't know if he's talked about this already
is just dropping entirely cosmism in the presumably
in their new um development giga Yeah. I haven't
talked about that, but yeah, I mean, that's kind
of, that's the thing, right? Like so many chains
are just not opting to use cosmopolitanism anymore.
And I don't know if it's for good reason or not.
I mean, considering confio, the maintainers and
creators of cosmopolitanism have been defunded.
It probably is for, for good reason, right? Like,
why would you trust something that's no longer
maintained? Yeah. I think that's basically it.
Isn't it? It's hard. It's like the money's run
out and And that's that. It's just like the rough
economics of the space, but it is disappointing
as somebody who invested quite a lot of time
and effort in Cosmos and its adoption, training
developers, writing documentation, making videos,
building protocols. Like it is a much better
product than the EVM. The EVM feels like boomer
shit compared to Cosmos. Like it really, really
does. Cosmos is just a better tool chain. But
to get to the level of developer tooling where
it would really win people over, it just needed
a lot more money and a lot more investment and
a lot more adoption. And you know what, it's
in microcosm. It's the same fucking problem as
the whole of the cosmos, right? You can sit on
a really great app stack and it doesn't matter
a shit. Like it reminds me of the, I mean, like
the most extreme version of this is like the
move VM. discussion because I know we've been
talking recently about so there's some stuff
happening in Aptos at the moment which is all
about there's been a proposal to reduce the inflation
rate on that network and there's been some vigorous
discussion on whether or not that is a good idea
but underneath all of it there's this question
of like you know product market fit and you know
if there's demand for your tokens a lot of the
other problems resolve themselves. And if you
have use cases, there's demand for your tokens.
And you can have the most excellent smart contract
framework and tooling for developers that are
picky and are prepared to stand a bit of pain
to have a problem they have solved, if that's
safety in the case of Move or if it's just a
comprehensible developer experience in Rust in
the case of Cosmosm. that that doesn't attract
users. Users don't give a shit how it's written.
They just give a shit that it's good or it does
something for them, right? I mean, it doesn't
even give a shit if it's good, right? Because
arguably the reason why so many exploits exist
on Ethereum. Yeah, I mean, like, obviously, cybersecurity
is a pretty tricky and complex subject doubly.
So, you know, block Jones and Tripoli, so unlike
smart contracts, so like, you know, I'm not gonna
say I'm gonna sit here and be like, Oh, well,
if you if you cause and wasm your things, there's
no chance of getting owned. It's like, well,
that's obviously fucking bullshit. But it's like
the functional programming argument. If you have
careful controls over mutable state, you're not
going to have any programming bugs that involve
mutable state. That's the whole point of functional
programming. And you can extend that argument
to smart contract frameworks that have better
guarantees about safety. It's like, well, OK.
you're still going to get wrecked if you're going
to get wrecked, but there's a certain class of
bugs that you're no longer going to have. And
as a developer, you can take that off your pile
of shit to think about. And that's important
if we're talking about programmable money, I
think. It's just that it's not, do you know what
friend of the, I can't remember who it was, but
somebody said a while ago, like about decentralization,
some friend of the podcast. that everybody says
it's important, but nobody will fucking pay for
it. I'll tell you that for nothing. And I can't
remember who said that. Sorry. It's kind of lodged
in my brain, but I think it's a little bit the
same with like smart contract tooling and stuff
like, or application stack tooling. It's like
exactly the same problem. And yeah, I don't know
how we get past it to not just building everything
on the EVM. like Boomers, like ETH Boomers, and
we move on to like, okay, can we find a use case
that doesn't involve every single part of our
application architecture just being whatever
the answer to the question just being whatever
Ethereum does, that would be a good day that
when we move past that, I think. That was a bit
of a rant, sorry. I'll shut up now. Do you think
it's all related to developer tooling or is some
of it just people are after the liquidity, you
know? I mean, I think honestly, Brendan, you
probably know better than I would about that.
But I think you're right. Liquidity is obviously
a big thing. I mean, liquidity is a huge thing
when you look at versus the cosmos, right? But
I think it's also like maybe like also the brand
and mind space and mind share. It's like, why
is it easier to start a company in the Bay Area
than it is to start a company in, you know, even
in London, right? Biggest city in the UK, tech
hub, ostensibly. Much easier to start a company
in Probably SF SF isn't a fair comparison, but
probably most US cities with a tech presence
It would be easier to start a company and get
funding than it would be in in London for a lot
of use cases And it's that Yeah, it's like that
network effect, isn't it? I think you have on
a theory and as well as just the like you say
the pure liquidity, right? I wonder how much
of it also has to do with the for lack of a better
term, fork ability. It seems like EVM lends itself
really well to people just forking each other's
projects and then launching their own based off
of said fork. You saw that a lot with like Phantom.
Phantom had a lot of copycats for their swaps.
At the smart contract level, you mean? Smart
contract and really the entire level. Smart contract
level as well as You see a lot of repeaters and
stuff. Yeah, I guess that's true. I mean, to
be honest, when you were saying that, I also
thought, we're just thinking, well, we're seeing
a lot of networks where they just take the execution
client or whatever from Ethereum. And it feels
like there's enough developer mindshare throughout
the Ethereum stack that people are remixing it.
like kind of an ad ad and for an item, whether
it's smart contracts or bits of the execution
client or the consensus client or whatever. And
that's cool, but they're very incremental changes,
I think is the problem. And there's a lot of,
but it's just VHS Betamax isn't it again. So
I don't know why I'm surprised. But yeah, it's
a tough one to solve. And because I think the
fundamental problem, I guess the root of my concern,
I guess, you know, I don't know what you guys
think about this is that I still don't think
despite that point, they've got to where the
remixing is happening, right and remixing remixing
in the kind of sense of like taking an idea and
making minor modifications to it and creating
a culture around it and discourse around it.
I mean, that is not only the essence of culture,
but it's also the essence of innovation, right,
which is cool. So it's good for Ethereum and
it's good for the wider ecosystem that that remix
culture is happening. But what I don't see is
any like, like Ethereum birthed NFTs, cool. What
are the use cases? Like real use cases has it
birthed since? You know, they did those NFTs,
they did DEXs, they did a lot of those first
things and then... I'm kind of waiting for another
wow moment and I'm not seeing it coming from
that ecosystem. So. Ah, big mistake. They did
soulbound NFTs. You see, there you go. That's,
that's the big thing. It's, it's AI, man. It's
all AI. It's all AI. I mean, I said on the last
podcast, I said like, it is entirely possible
that AI is the use case and I'm wrong. And, and
it's fine. and null score he's like ai i'm out
so this conversation deuces yeah he's like i
want to go look at some hard drives um oh he's
back ah yes all right i missed it all right okay
see you know peace out buddy um text now um Yeah.
I mean, what do you think, what do you think
of the, like, what kind of use cases are exciting
you, Brendan? I know, I know at the moment you've
taken a little bit of a, you've taken a little
bit of a step away, I suppose, from, you're still
running, you're still running Whisper, right?
So you have kind of, but you're not, my understanding
is you're not like doing lots of active business
development to like bring on, you know, it's
for Whisper at the moment, right? Yeah, pretty
much zero. So just running, running the existing
ones I had, even kind of concentrating on a select
few at this point. But yeah, mental bandwidth
is more on the broader ecosystem, trying to catch
back up on what's going on out there. I feel
like with running the validator stuff and running
the day -to -day business, you lose track of
where the market is going, where the space is
going to a degree. Yeah, 100%. I don't know what
your setup is like, but I feel like Maybe this
has got better recently, but like, especially
maybe again, early days, like, do you remember
in like 2021, when none of us even knew how many
tokens we got, and we were all trying to like,
write scripts to work out what our commission
rate was so that we could like not get sent to
jail for like, misreporting taxes. Like, and
it's just like, you look up like six months later,
and you're like, I don't know any launches that
happened. I wonder why, right? Yup. 100%. but
like what are the kind of things you're looking
at now where you're looking at and you say hey
that's pretty exciting because like you say you've
got a little bit more perspective now maybe and
like even if it's like quite a different perspective
yeah i mean i'm only like two weeks in so not
that much of a perspective but really you're
only two weeks done it feels like you've been
oh i agree oh man having a kid my time frames
are just so messed up It's Web3 in general, man.
It's like a day feels like a year, you know?
Yeah. I think some of the AI stuff is interesting.
They're slapping it everywhere, but there was
a project out there where the founder was ex
-Uber and they were building like payment rails
for AI agents. I think that could potentially
be an interesting use case where the blockchain's
kind of confirming the actions that agents are
taking and then taking payments, whether it be
agent to agent or agent to external API. Cause
like, you know, who's going to run the payment
system for all these AI agents out there. Yeah.
Well, I guess it'd be extremely profitable for
anybody that did want to, but it would involve
a fair bit of risk. I guess the real question
would be like, who needs to audit it and when
and by whom, because the easiest solution is
just make it public. And then the next thing
you know, your AI agent has made a really, really
identifiable set of transactions on your behalf
and somebody is coming to your house with a hammer.
Yeah. Well, there's a big push to run a lot of
the stuff in TE's, Schultze. going back to the
secret network days again, coming full circle
there. TEs are a hot subject again, because of
all this AI crap. And I know near protocol, they
were in a bunch of panels and their product,
product lead just kept hammering, hammering away
at TEs and all these AI agents need to be running
in TEs. I, I've said it many times in this podcast,
but near I think, is one it's my favorite blockchain.
Like I think that I think that their vision has
been The most pure if you've seen their white
paper or not the right papers their whiteboarding
series on YouTube They basically interview all
the different founders of different chains and
they talk about like what were the design constraints
and then they basically used all of those whiteboarding
sessions with those founders to develop near
and The things they've done in near I they should
be bigger than it is I mean, maybe they're going
to have kind of a renaissance. It seems like
they have a lot of presence at conferences. They
were everywhere at token in Dubai. Oh, really?
And they're actually a big client of flip sides
as well. We just we just released our own AI
insights product with near. So I'm curious to
see what they'll they'll do with it. I think
if a few dollar do screw screwed away, haven't
they near protocols. So this There's definitely
some like, um, and I guess like Tendermint or
the Hub is maybe a little bit of an example of
this. There are definitely some crypto foundations
that made so much money in their early ICOs or
via VCs or just as the result of their tokenomics
that they're, they're basically too big to fail
at this point, even if they don't execute their
product vision. And that's going to be an interesting
dynamic moving forward. Right. As well. Cause
I mean, there's, there's kind of a little bit
of that way. It's just like the even if they
don't find a use case, they're just going to
have a big treasury, which somebody will buy.
There's still ICOs out there like that. I just
saw someone that worked for Salt Lending the
other day. I didn't even know Salt Lending was
still around. Interesting. I didn't know either.
I mean, yeah, you see a lot of stuff, especially
because academic papers take quite often so many
years to get published. I'm sometimes like I'll
see a thing that I'll see a paper and read it
and then be like I have to go and double check
like when it was first started versus when it
was published because I'll be like is that protocol
still alive or is it just that they took a data
point in 2017 and then it's taken five years
to get in this top journal because it takes a
long time to write like you know these really
like top tier stuff. Or like, is it because that
thing is still fucking going? And it's because
they're still going. They've got loads of money,
and they're trying to get academics to write
papers about them. Or they'll go, hey, you guys
want data on decentralization or something? We'll
give you archive access for free, for no worries.
And we'll give you developers to help you find
the data if you're not very good at coding or
whatever it might be. And you're like, oh, that's
what happened. It takes someone to take us a
bit of scratching. forever looking for a use
case whatever it might be.