not shopping before.
No, the fucking internet just keeps dropping out.
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent
validated teams.
Hello, welcome to Game of Nodes, a weekly podcast on the cosmos from independent
validator streams.
Independent validator streams?
Independent validator teams.
Ah, that was nearly professional for the first time in our history.
We're here as always, well, as mostly always with a very frozen null with
Schultzie and with Todd, who's going to be talking to us about the new release of
Tender Duty 2, the hottest product on the street for the validator that wants to
monitor their nodes.
But first up, we've got some follow-ups for next week from last for next week.
Well, my sense of time has obviously got very screwy from last week.
One thing I've added to that as a kid, one of the things I have the most
difficulty with is telling the difference between like, I did this yesterday
versus I did it tomorrow.
And so I'd be talking about things I'm going to do or I did and I'd be saying
tomorrow I did X, Y and Z.
So I don't know why I did that, but just with you talking about going a little
bit off, I totally relate.
Is there a specific thing where children are less aware of the passage of time
or something, or is it just that we're both bad at remembering which is the
correct word?
There's probably something to do with that.
I mean, children don't get object permanence until they're like, what, four?
So, or maybe let, I don't know.
I don't know, children.
I don't know.
Yeah, I don't know.
Anyway, that was a bit of a weird divergence caused by me.
Doing the verbal equivalent of a typo.
So we've, we've got one of the biggest follow-ups we've got from last week is
a variety of stuff on the Validator League discussion.
Because I gather that's moved on a little bit and they're now restructuring.
Is that right?
The, the, and I think we're possibly even going to have ghost and possibly
Tyborg coming on the show on a future episode actually to talk about how that
thing, how that stuff is evolving.
Is that right, folks?
Yeah, I believe so.
I think next week, but not starting.
Yeah.
And those and eagle eye viewers will have spotted that there is no usurper here
because he is, I think, stuck in a hotel lobby somewhere.
We presume drinking whiskey on his own.
So one of the, one of the other Nulls frozen again.
I think we have just lost him.
Let's, let's move on while Null is frozen to, to talk about tender duty too.
So we've got Todd with us who, so most, most validators have the problem of,
of like, obviously monitoring their nodes.
And I think it's pretty common that people are going to use something like Grafana
or Zabix, Prometheus as a sink into the, into one of those two services.
I guess I'm guessing we're on here mostly using Zabix.
Is that right?
Okay.
Yeah.
For monitoring.
And then the other part of observability is, is obviously notifications when we
go wrong.
That's where Todd comes in with the incredibly useful tender duty tool for
monitoring cosmos validator nodes.
It's used very extensively by people on the show and around the show.
And everyone's very appreciative for it.
And there is a new version that is just out.
So we thought we'd get Todd on the show to talk about what's, what's new in tender duty too.
So yeah, thanks man.
And I'll, I'll, I'll show my screen, but I'll try to narrate thoroughly since I know some
people listen on audio only.
So tender duty kind of, it is one of those things where it's like, I kept
finding myself looking at Mint scan or ping dot pub to see if I was missing blocks.
It's like, Hey, how am I doing on blocks?
And it was like, this is not okay.
There's got to be something for this, right?
So, yeah, so let's see, how do I share my screen?
Oh boy.
All right.
It requires sharing my whole screen, which is a 47 inch monitor.
So I don't know if that's going to work.
Tell you what, I can jump back on from a different computer real quick that, that makes
this easier.
And it looks like Noel's moving again.
So I'll pick it up here in a minute.
Is that all right guys?
Yeah, no worries.
He really just asked me before, is that, you know, before I was frozen?
What do you say now?
Sorry.
It's plenty of technical difficulties with this issue.
So, yeah, before the show started, we were talking about, we were talking about Horcrux
as well, because it obviously has been a very common topic on the show is remote signers
and using that kind of infrastructure to secure your nodes that you don't have to have key
master on a given box.
And sure, so you've finally made the jump, I think maybe the first out of all of the
Game of Nodes crew to make the jump to full Horcrux, right?
Yeah.
So I guess for context, when you run a validator, there's kind of like three stages of key signing.
There's what most people do, which is you leave the key, which is like the way your validators
assigned on the validator machine itself.
And then the validator kind of signs for itself.
And then the second stage is you have a remote signer.
So there's a software called Tenderman Key Management System, which that says then you
have like your node running, and then you have a second node running that points to the
first one that says, hey, I'm going to sign the blocks for them.
And what that does is it kind of gives a layer of indirection in terms of security.
You can keep the remote signer really locked down and then the node less locked down and
you're still secure.
Another benefit with that is if your node goes down, you can just point it to another one
really easily.
And the third solution is you can have multiple signers pointing to multiple nodes to make
sure that you're always signing in some capacity.
So with Horcrux, it's that third level where I can have basically two nodes go down and
still be signing perfectly fine.
Similarly, I can have multiple of the signers go down and still be signing.
So basically it's a method for ensuring that your blocks are just always signed no matter
what, short of something utterly catastrophic.
Right.
Hey, Todd.
Back on different machine.
So we were just talking about Schultz's migration to Horcrux and he's just explained basically
what the benefit of essentially a cluster of threshold signers is over a single box running
something like TMKMS.
So the interesting thing as well is like there's all of the, it's obviously there's a threshold
signer and that's all that's a form of raft and whatnot.
But also just the even to actually create the key matter, you basically need additional
to shard the key in a particular way and stuff like that as well, don't you?
And so like there's quite a lot of steps involved in actually making that a viable production
ready thing you can do, right?
Yeah.
But the, I think the thing that we were also talking about earlier was that the there was
some quite unexpected behavior in terms of what you saw when you actually sort of ran
it in production, right?
Yeah.
So for test net, it was absurdly stable.
They are Horcrux itself is typically pretty CPU bound.
In whenever I run on test nets, because I set up on my tests as first, it ran perfectly
stable.
I could run, I ran it for almost a week because just the test nets and everything were fine.
So I set up on the main net.
I was using AWS instances.
And at around 4am every morning, one of the Horcrux signers would just decide it was time for it to
completely peg out on its resource usage, which would cause all sorts of misblocks.
And it was even inconsistent across which day because it's happened in the last four
days in a row.
Some days it'd be the first signer, some days it'd be the second signer, some days it'd be the third
signer.
It's still ongoing as to what the issue was, but hopefully it is more or less resolved as I've
switched from using AWS instances, which can be kind of on the fly versus using bare
metal machines for the signers as well, which is also cheaper because AWS is absurdly
expensive.
So where do you have your bare metal machines at for Horcrux?
I now have one on Volter, one on OVH Cloud and one on Hetzner.
So one in Helsinki, one in Virginia and one in, I want to say Oregon.
So do you have a whole bare metal machine just for Horcrux signer?
No, I've already got instances for those either way.
So Volter and OVH Cloud hold, they have bare metal machines that can work with secret.
And so I'm basically reusing those to also be used as a signer.
So yeah.
And I think that's the original intent of Horcrux is to have it alongside nodes on actual,
instead of having them on dedicated instances anyway.
So is it?
Yeah.
Well, when Jack used to originally talk about it, he used to say that you'd deploy it next to a node.
Because, you know, we use it in a bit of a D gen way in that the original intent is that you would have a swarm of signers for one network,
not, you know, sharing it across a couple of networks like running multiple signers on one node.
So the original architecture intent was to launch it out next to a node running on an instance somewhere.
And then it, like that signer would just talk to that node and the other signers.
And then the other signers would talk to their node and the other signers.
So it wouldn't be like, so you'd have the lowest latency to the node and the network, but then they could talk to each other.
Right.
So it's like having three centuries, each of which has a signer.
None of the signers, none of them have the keymatter.
So it doesn't matter that you've got three extra things out in the world, right?
Yeah.
So the signers only have a portion of the key matter on there.
It's impossible to put back together unless you've got like them all.
For some reason, my mental model was totally broken.
I thought the whole point was like, it was more like Kafka and ZooKeeper.
So like you deploy three signers to a cluster and then they point at a cluster of centuries where if one of those centuries goes down,
you round rob into the next one or you just round rob in them via proxy anyway.
But the cluster of signers is going, let's roll.
So there's many ways to skin a cat with hall crocs and it just depends on how much of a degenerate you are on the ultimate.
And a lot of people do actually keep their signer on the same machine still.
That's super common.
Yeah.
I was talking to someone about it today about a script that they wrote basically for setting up their machines doing that.
Yeah.
I haven't deployed hall crocs on any main nets yet, but that's how I've got my Ansible setup.
One signer, one node, same box.
So, Schiltzi, I'm not sure if I tagged you yesterday in the chat, but there's Dan B&B.
It's going to be a Kujira validator.
And he had this hectic Terraform scripting to launch everything in one click.
It was crazy.
But I think it was alongside the nodes as well.
I haven't seen that, but it sounds absolutely fascinating.
Terraform is one of those things that where I see it and I'm like, oh, someone knows what they're doing.
And then you look at it and you're like, I don't know what's going on.
They might know what they're doing.
They might not.
I don't know.
I was literally like, say this is the timeline, right?
So, I was talking to him about some stuff and then he asked me about TMKMS, right?
Because I had that script up there that makes it pretty easy.
And I said, well, if you want to know about it, have a look at that, read through it.
I talked to him a couple of times that day, right?
And then he had TMKMS up on the test net and he was pretty happy with it.
And then I said, if you want a better solution, maybe have a look at Horcrux and gave him the link to Horcrux.
Like two days later, he's like, oh man, I've spent the whole day scripting.
You'll be pretty impressed with my setup.
And he sent me the link to his repo on GitHub.
I was like, holy fuck, man.
He's written like 300 lines just for the config file.
And then there's like four other files that like make this thing work.
I was like, holy shit, dude.
You did this today.
And he's like, yeah, start this morning.
Yeah, Terraform is pretty verbose though.
To be fair, like I did a Spaces Within the other day and he is like a DevOps genius.
So he's like new to validating, but like a DevOps is pretty good.
Yeah, I wish he could come on my team.
Yeah, I'm sure he is.
There's definitely a thing though with Ops stuff that a lot of the, especially the automation scripting tools are.
There's a combination of things other than that.
For the purposes of this conversation, they tend to be quite verbose because obviously their configuration management and all that kind of stuff.
But the more pertinent thing maybe for this is that there are really heavy diminishing returns on something that you don't need to DR very often.
And there have definitely been a couple of times in my career where I've spent like a month getting a completely like rock solid set of Terraform scripts or Ansible scripts or whatever for a thing.
Only to then find that, oh no, we need to change this major thing about the way we deploy the software.
And it's like, right, that's basically all in the bin.
Script that.
Yeah, I mean, like you want to deploy like two database instances behind Nginx showing my age a little bit there.
Perhaps I don't know if people are still using it as a reverse proxy all the time.
And you like script all of that blah, blah, blah, blah, happy days.
And then a month later it's like, actually, we're just spending a lot of time on database ops.
Should we just put it on Amazon RDS?
And you're like, yeah, I mean, we can do that.
But we have to rewrite all of our database deployment scripts.
And it's like, yeah, we just do it by the Amazon console this one time.
And then as anybody who's actually deployed any of the, you know, the highly managed AWS RDS instances knows, you deploy it by the console and then you write a script in case it goes down.
And it never fucking goes down ever.
And then so you're just like, and then you know, somebody asked you at conference, how do you how do you manage your, your thingy thingy and you're like, yeah, I mean, we've got like 1200 lines of terraform.
But to be honest, we just deployed it by the console.
So I would love to pretend that we have some really shiny stuff going on.
But that's sort of paranoia really.
So my only comment to him was that he's like, you know, here it is, do you have any comments that I'm like, to an AWS, you're going to be broke real quick.
Yeah, I mean, to be fair, I only want AWS because I got the $1,000 startup credit.
So I was like, yeah, I can just blow through these and nothing matters.
But apparently, apparently what I was using would be I think $200.
A month per signer.
And I was like, honey, doctor my side away anyway, even if it's free, I can't stomach it.
Yeah, I was dollars, but get you far.
No, for us, I think we're going to spray it around digital ocean in multiple locations on digital ocean.
See how that goes.
But I'm doing like, I'm doing stepping stones.
So I was on TM KMS, now I'm like single sign off all cracks pointed at multiple nodes.
And the next step will be like, once I'm comfortable with that, the next step will be working out how I'm going to reliably do multiple signers.
But yeah, it should be pretty, pretty interesting.
But I need to work out my answer woman.
I need to watch the rest of them videos.
Yeah, I mean, there's also like, I think one of the things that I'm quite interested in trying to do is like the whole Akash thing, which obviously been experimenting quite a bit with recently, but I,
I, you can't, well, I mean, you can, but it's going to be hugely complex to deploy.
And you have then whole cracks signers in that containerized environment in like mini pod with the, with the card valid with the Akash with it, so with the validators that are deployed into a cash like by the time you've done that,
you may as well actually just buy a couple of really big metal boxes and actually do it yourself because I bet dollars to notice that we cheaper than running Anakash
Yeah, after your experiments on a cache, they turned me off a little bit.
I'll say that deploying my website on there, not granted, because it requires zero resources,
was ridiculously smooth.
When provider went down, when together went on, it took me all of a minute to redeploy.
You can deploy it with unrelated to websites, but if you have a look at the Cosmos Omnibus
repo, there's an example in there about how to deploy two nodes with a load balancer to
two different providers.
There's that as well.
Interesting.
I was told that that wasn't something that could be done.
You'd have to do it manually.
I'll have to look around in there.
Maybe.
It was a long time ago.
Maybe I've got my wires crossed now, but it'd be interesting to have a look anyway.
Can you only deploy?
Yeah, I don't know, man, because the provider is hosting the deployment, which is a thin
wrap around Kubernetes.
Not sure how it could...
Maybe it's on the same provider.
It might be my fuck up there.
I'm not sure.
If it was a thin provider, it'd be pretty easy to just put nginx in front of two RPCs,
for sure, or whatever it is.
Actually, running RPCs, the difficulty, I think, is supply side.
I think we were talking about this privately.
The tech works surprisingly well.
The marketplace works not the thing, but the supply side of it is a bit more complicated.
Make sure he says sometimes providers go away.
I think you would want to be in a position where you had redundancy.
The cool thing is that because of omnibus, like Tom's great work again, you can just
bootstrap a node from nothing quite quickly.
That is really impressive when you do that.
That was really what's pushed me down into TMKMS, finally adopting that properly and
all that kind of stuff, is because you can literally just spin up RPCs.
Those could be validator nodes that are quite throwaway longer term.
As long as you know where to point them, you're just reloading config somewhere.
You can obviously drive that a variety of ways.
You can script it with Ansible.
You can do whatever on your TMKMS node, whatever.
There is that lingering feeling that the provider might go away.
That's the missing piece for me a little bit, is I haven't really worked out how to monitor
it effectively and I haven't really worked out the pricing.
Again, I think we were talking about it, getting a reliable deployment.
You're sort of $60, $70 ballpark, I found.
That will buy you quite a bit of bare metal.
It will buy you quite a bit of digital ocean.
It will even buy you quite a bit of AWS.
That potentially is a bit of a segue into tender duty.
Now I'm not as unfrazen because one of the things I was thinking about was obviously
one of the ways you can monitor, you can at least have alerts when things go down is tender
duty and since that runs, typically it's run containerized anyway, co-locating that with
a node with its RPC port open only to other containers within the deployment is actually
very viable.
That's pretty cool.
I have to confess, I'm probably the only one not using tender duty on the entire game
of nodes crew.
You're not using it at all?
What are you using?
How do you monitor your shit?
I use Monit and Cron.
I use Monit too, man.
Monit is like the core of my monitoring.
Oh, yeah?
Yeah, I mean, I use inMonit for, I've got crazy Ansible scripts that build all these massive
Monit configs that check everything, man.
We're leveling on my disks.
You know, you can do ware level monitoring on Zabix now too, plug.
Oh, nice.
I mean, it's a Monit, everything's crazy Ansible script.
You just go like install that friggin smart monitoring and it just picks it up and it's
like, oh, now I can do this.
Right, but so my reasoning here is this, right, is that there are lots and lots of things
that can go wrong.
I know that if the shell fails, that box is toast, right?
I know if Monit fails, the box is toast.
Like those are things that are not going to fail.
Cron is not going to fail.
Yeah, you don't necessarily like have to have a toast box to stop signing though.
You can have a toast signer.
Well, sure, but like that's the point of the bash group, which is checking various things
about the box on a short interval and then is going, okay, if any of this is wrong, then
I'm going to immediately hit a page duty endpoint essentially.
So it's not it's not different, significantly different to tend to duty.
It's just a homebrew config involving Monit and a web request via Curl.
So actually what Tinder duty does is is different.
And that's why I wrote it at first.
I was like, you know, there's nothing doing what it does and it needed to happen.
So it subscribes to the blocks on a web socket and it checks for your signature.
And that's where it, you know, that's what the main purpose of it was is to say, hey,
if I missed three blocks, you know, I need to get up and fix something.
I don't care about note health.
I don't care about anything else.
Well, now I do, but it was built for one purpose and only one thing.
And I think the first, the first version that people started using was like 140 lines of
code.
So it was like this tiny little single purpose very Unix like type tool that did one thing.
So and then it's grown quite a bit.
And now it's V2.
It does a lot of stuff.
So and this is stuff people have been asking for and stuff I've been wanting to do as well,
you know, so now it's kind of a full cosmos monitoring tool that goes beyond just, hey,
did I miss the blocks kind of thing?
So yeah, I know that like, all of us thought it was awesome and all of us use it, but all
of us are always like, Hey, man, have you thought about adding this thing?
Like, I'll put it on the list.
And like, Hey, man, have you thought about like making it like easier to look at?
I'll put that on the list.
Hey, man, like, can you come on to this other thing too?
Okay.
Yeah.
So now, you know, it's got a dashboard, a Prometheus exporter, which is still in the works, but
you know, like, Hey, if your node goes down, how long has it been down kind of thing?
Or, you know, how many times did I miss a block where I actually sit in a pre-commit
and the proposer didn't include it?
And that's a big question for me a lot of times if I'm not performing well on a chain,
it's like, is this actually my fault?
Or is this, you know, somebody else out there set to sign blocks at three seconds, you know,
just making me drop.
So just to explain that for people who are listening, what's because this is the thing
we've seen on several networks, right?
The three second signing.
What's the reason that that's causing you to miss blocks?
Because the proposer is setting a limited time before that they will propose a block
at 67%.
So normally if you wait, I think it's five seconds after your first pre-commit or something
like that until you actually propose the block.
And if you crank those numbers down, you can reduce block time on a chain, but you're also
going to have a lot of validators missing blocks.
I think this might actually be a suspect that this is causing a problem on secret network
right now because their average block time is like 5.2 seconds, which is a hair faster
than I'd expect.
So somebody out there speeding up the block.
So someone with a lot of like voting weight to a lot of blocks.
I haven't dug into it, but at some point I will and probably reach out to them individually
and say, Hey,
Yeah, we've seen that on some
Todd, I think I lost your audio.
Did you guys just see me or battery went flat?
No, we can't hear you, man.
So I mean, I love tender duty.
It's like so light.
So one thing I want to know is like when Todd gets back on here is so there's a new configuration.
Well, he's completely rewritten it, right?
So now it's like, I think it's even a different language now, or maybe it's still go, but
he's added in extra features.
And what I'm going to want to know is like, can you turn off the Prometheus exporter, etc.
to try and keep it as light as possible?
So I use ZabEx, I'll probably like query it in different ways with ZabEx through the
different ways you can actually send commands through ZabEx to your server.
So you know, things like the Prometheus exporter, I won't want.
But also like I'm wondering if it's multiple network configuration in the one file because
you used to have to launch multiple instances and configure all your networks separately.
And up until recently, I thought that meant that you had to bring them all down when you
wanted to change the configuration and then bring them all back up.
But apparently with doccontainment, I just figured this out the other day, you can actually
just update your configuration file and then re-up them into daemons and it'll just update
them and you can use a flag that'll drop all the dead containers and delete them.
So, daemon Hill, race car driver.
Did you guys hear me now?
Yeah, man, you're back now.
I've just had so much technical trouble today.
Sorry, I just took the liberty to ramble some shit while you weren't here.
Oh, no, no, totally.
And actually all this pronounced daemon really weird too.
Incredibly relevant because...
Because...
Oh boy.
Apparently, I can't say daemon or daemon.
I always call it daemon.
So, daemon is a name.
It's just an old English name and daemon is like a malicious spirit.
So you guys are going to laugh because Chrome didn't have access to record my screen and
it wants me to restart.
Well, five minutes.
That's what I'm going to do, man.
I'm just going to explain what's going on with it and if you're interested in checking
it out, I've got a...
Do you have a working page somewhere because Nol can probably throw it up on the screen?
Yeah, actually.
You know what?
I've got to publish my notes.
Share the link in the private chat and then Nol can throw it up.
So anyways, you were saying that one of the pain points was one chain per config, right?
And so now that's...
You can do multiple chains on one running instance.
The dashboard is very much inspired by...
Oh, boy.
Batteries.
Welcome to summer.
Anyways, the display on there is very much inspired by what's going on on ping.pub except
it is quite wide and you can actually scroll over.
I think I've got...I want to say I share...keep the last 512 blocks so you can see where you
missed, where you signed, the different colors.
When you propose isn't a big deal, but I thought it was cool to be able to see where and when
I proposed to see how often it was happening.
My dashboard is incredibly boring because there's nothing fun happening here.
A lot of my testing I've been doing with other validators.
They've been very helpful.
They don't know it.
But...
That's a low-key flex if I've ever heard one.
What's that?
That's a low-key flex if I've ever heard one.
My network is boring because I missed so few blocks.
I did miss a block on Terri Classic there, it looks like.
That's great.
The dashboard is something that I wanted just because I never really had that information.
When something fails, you get a nice little orange alert message and you can click on it
and see the details.
This is set up for public use.
Normally if it would have a log stream below showing, hey, this RPC node is not responding
or something like that.
The dashboard is a dashboard.
There's not much to it.
Just...
Then, like I mentioned, there's a pre-MTSN point.
If my screen share were working, I'd show you the beginnings of a Grafana dashboard,
but I can't.
Each thing is configurable now.
You can alert to Discord, Telegram, or page or duty.
It's configurable per chain.
You can have different alert destinations per each chain.
Like I said, this is done.
This is essentially done.
I'm working on docs and then I'm going to tag it as a beta probably in an hour or so
of ending this podcast.
You can do...
I didn't realize that it had the notification stuff because I was looking at what's it called,
Half-Life, which I know some of the other GamerNodes folks use for Discord notifications.
I hadn't realized because I think, like I said about the Akash stuff has started to motivate
me to move to...
Even on our servers, probably dockerizing and actually deploying everything using our
same mount strategy, but starting to use sort of dockerized containers.
Tendermint has been on my list for a while to work out what the gap is between what we're
doing and how good it could be.
I think that might be the killer feature for us there.
How does that work?
Is it just possible to basically configure and send push notifications then to additional
sources?
Yeah.
I mean, if you want to alert to Discord, you just set up a web hook on a channel and
it can post in.
Then telegram is kind of a pain to find the group ID for the chat, but you can get an
API key and then invite the bot into a chat room and it can send stuff.
PagerDuty, obviously, API key.
The one benefit to using PagerDuty is you can run multiple instances and it'll deduplicate
alerts based on a key that I set when I set an alarm.
You can have redundant, tender duties.
Redundancy and your redundancy.
You could have a tender duty on the node and you have a tender duty somewhere else pointing
an RPC and then it would deduplicate them.
Yeah, because it uses the same key.
It updates the same message, right?
Oh, my God.
Did it my mic cut out again?
Yeah, just like an SID you're muted.
Crazy.
Wow, this new Macbook isn't entirely stable.
Yeah, man.
So it dedupts based on a key that you set and depending on the type of alert it's sending,
I set different keys, but I think I use the Valkons address, the consensus public key
address as the key for mistblock alerts.
So pager duty won't resend an alert with the same key, but you can resolve an alert automatically
with the same key.
So it's nothing special I'm doing, just something they do for us already.
But yeah, see what else is new about it?
How long did it take you to develop the first tender duty?
About four hours.
What a machine.
Well, I mean, it's not hard.
You just subscribe to a web socket and look for a key in the resulting JSON.
So that was super easy.
It's when you start adding edge cases and configurability and you break something and
then you're like, oh, well, I got to fix that.
So yeah.
So with those four hours, you saved probably thousands of man hours.
That's pretty cool.
Yeah.
But yeah, since then I've put weeks of work into it, but especially now I'm about seven
eight days into this effort.
So that's a lot of work.
That's a significant amount of time.
As an aside, as somebody who comes from primarily a high level programming background, I'm in
the grand scheme of things, relatively new to Rust and I guess I've read a bit of Go
here and there in the past.
But this is one of those things I think that people maybe from, like I say, me who have
moved from more of a high level programming background to those of you that aren't programmers,
it means closer high level, generally high level of abstraction.
So I think that's why, where assembly is the lowest level of not abstraction at all, really,
up to literally like whatever you see in your web browser, I guess is the way of explaining
it.
But so the cool thing about the first version, which I think I remember seeing at the time,
because this is I think why I end up just being like, I don't need to listen for the
web socket.
So I'm just going to like, not to go to a bash script in five minutes.
But I think I remember seeing that first version and it is like essentially, yeah, the stuff
you can't do in a bash script because it's a web socket subscription goes got an API
for that.
It's like a few calls and then a bit of JSON parsing and then boom.
But the really cool thing is for somebody from like a high level program background,
you look at it and you go like, ah, this is essentially a small script, a Go file that
is essentially being treated like a scripting language, and it's going to emit a binary,
which then you can pretty much run everywhere.
And in addition, you also supplied a Docker file with it.
So you literally can run it everywhere with Docker.
But it's one of those things that like, I guess I hadn't really thought about until I
saw I think it was your script for tender duty and something I went back to a little
while ago because somebody from Terra was asking how to listen for a particular type
of emitted Cosmosm event.
And I remember that Jan's Alex, who has done quite a bit of core work on Juno, had written
like a tiny, yeah, again, like a 30 line Go file that you can obviously compile to a
binary and be like, here you are, listen for whatever you want to listen for.
Yeah.
Yeah, actually, you know, it's insane.
For Tinder, men is crazy easy because you just call into their RPC library.
And so it's like, hey, I want to get this.
It's like four lines of code.
You know, so the API is well defined there.
The annoying part though is where you end up with different like some stuffs protobufs,
some stuff is JSON.
If it's JSON, everything's a string.
So now you've got, if you're in a strongly typed language, that's a problem.
You know, now you've got to deal with reflection and casting and all that fun stuff.
So you know, RPC isn't great.
GRPC client is awesome.
I mean, you can do everything, but it's that requires a little bit more knowledge to program
against.
So again, I'll expose my ignorance to this.
What's the difference between RPC and GRPC in this case?
RPC is a web protocol, returns JSON.
GRPC, it's a web two protocol.
I mean, HTTP protocol, but it all relies on protobuf, which is a binary.
Yeah.
It's the binary format at the core of, you know, ABCI, the blockchain interface databases.
Yeah.
Okay.
Obviously protobuf is quite deeply embedded in Cosmos and like former life doing data,
data infrastructure.
Protobuf is one of the things, is one of those things like Avro, I guess like Avro you see
as well, but like protobuf is pretty common as a serialization mechanism.
Yeah.
I think one of the big differences is that Avro, generally you can embed the definitions
with the message in protobuf.
You don't do that.
Yeah, I think in my, I think the most patient for, sorry.
No.
Can one of you guys explain what protobuf is so that people who are listening don't
know exactly what you're talking about?
I don't know.
So like JSON is like structured data, right?
Yeah.
So what is protobuf?
Protobuf is an efficient data messaging protocol.
You define what type everything is and then it builds, I don't know, I don't know if you
guys are familiar with like ABI's from Ethereum or EOSIO world where it's, you've got an ABI
and everything's binary and you pack it in.
That's basically what protobuf does.
It's a way to see ABI's, but it takes it another step where you've got all these
binary definitions and ways of compressing stuff and then it adds the ability to build
ABI's around it automatically.
So when you have a type, you can say, hey, this is how you set it.
This is how you get it with just a regular function and then it'll build out the GRPC
APIs and then even on TenderMint, the Rust APIs.
And is that type like basically it makes a smaller message like less data transfer and
stuff like that as well quicker?
It's just more overhead on both sides.
Yeah.
For like a string, you get a null byte and then the length of the string.
And then if a field is mandatory, you just get null byte between each one.
Yeah.
So it's super efficient that way.
Yeah.
While we're talking about these interfaces, what's an LCD now?
I call it Rust.
Yeah.
In the documentation, it's called Rust.
But I have no fucking clue what LCD means in this context.
Like clientDemon.
It's from before when the clientDemon was split out from the TenderMintDemon, right?
So it would talk to TenderMint over, I believe, a GRPC socket and then it was like, well,
why do we have this kind of thing?
So it's just basically another layer to make it easier for people to talk to it?
Yeah.
If you understand Rust and not GRPC.
Well, it exposes everything in GRPC, almost everything.
But you could alternatively just talk to GRPC instead.
If you know how.
Yes, that's the big question.
It is something that takes a little bit of time to get your head around, I think.
But it's so core if you're going to do any kind of dev on Cosmos, I believe, having
a good understanding of how it uses protobuf is pretty key.
Yeah.
This is kind of insane thing that happens if you actually define anything new in protobuf
and then just kind of like you're able to run it.
It just generates all this like loads of stuff and you're like, whoa.
So yeah.
It's, I'm still not 100% again, not 100% used to how that works in Cosmos, but it does give
you, it does make you look like you were typing very, very quickly when you do get to do that.
So that's not, I know it's not even my jokes.
It's something Jaby said because he's a funny man and I am.
He was like, look how fast I'm typing.
And it was just like generating all this protobuf stuff.
But I guess that the, it'd be quite interesting.
I have never really thought about how it must work under the hood for the arbitrary types
for, well, I know a little bit of how it works for the arbitrary types that you generate
from smart contracts, but obviously they're then exposed via a underlying protobuf message
in WASMD, which is then available to the client, but you essentially do an extra layer of misdirection,
don't you, at the smart contract level because you're doing that execute contract and then
passing along the stuff.
And I think this was actually a sticking point we're talking about orth Z for smart contracts
because technically, technically, if I remember correctly, the every smart contract that I've
ever seen that runs in Cosmos on Cosmwazn expects a top level JSON object as its message
because then you can unpack the types in Rust.
You were using the, what's it called the SERDI library.
In this case, we're for those listening, SERDI means serialization, deserialization.
So think code, something happening through code into JSON and then back out of JSON,
something like that SERDI.
But the, yeah, apparently that actually the underlying code makes no assumptions about
the shape.
So even though in practice, it would be pretty hard to work with anything that didn't say,
oh, by the way, I'm a message of this type arriving at the smart contract.
Apparently, that's not actually an assumption that holds at WASM D level and it was causing
some trouble.
Yeah, you would have to like wrap the original message in JSON inside the message.
So it's all fun and games.
Anyways, a huge thank you to Cosmos's grants people and gave me a grant to do this rewrite
and it's something I wanted to do for a while and it helps a lot of people.
So I hope people find it useful and I don't have anything else to say about it.
How was the grant process?
Did you have to go through like interview panels or did you write up, you know, how's
it going?
Yeah, I did an interview, video call the interview and put together a proposal that I submitted.
I went through a couple rounds of review.
I think it's easier because I said, hey, I don't want anything up front.
If I don't deliver, you don't pay me and if you're not happy, then okay, it was worth
the chance.
So I don't know what the final review process looks like.
You know, I'm not going to call them up until I'm done, but I'm close.
I mean, yeah, I think that of all the things that we've seen recently that are kind of
up for grants and things, tender duty is one of the most actually pragmatically useful
things.
So I'm sure it's not going to be a problem when it comes to it.
I just noticed we've had a question in the chat from Adam Burke says, I'll put it up
on the screen, but I'll read it out for those listening on podcast players.
Super baby technical question.
If I wanted to get information about specific chain value data ranking, block time, any
kind of data available on a block explorer, how would I do that?
The short version is via RPC and the long version.
Yeah, I mean, there's an API for that.
Could be a new catchphrase.
Yeah, API for that.
Yeah, PC.
It really depends on what you're trying to get, but yeah, the command line makes it
pretty easy.
I cheat a lot.
I will use the command line tool with a sniffer and catch what requested is actually sending
because that's so much faster than trying to read the code and figure out what the queries
are.
And so that's usually my go to trick.
I tend to use Cosm.js and do the same thing, like just like grep through the code for Cosm.js
on GitHub to find a function signature that looks like the thing that I'm trying to accomplish
and then just fire it off from the console in the browser and just see what it looks
like in the network inspector.
And obviously, part of it is encoded always, but you kind of tend to get an idea.
You're like, oh, right.
I kind of see it's doing this thing.
Okay, cool.
Yeah, you know, and then the types can get weird depending on if you're using REST or
RPC, but I have found a trick with most endpoints.
You can do a raw A, B, C, I query over RPC and then it will give you protobuf back.
And then you can just use the native types from, you know, whatever X modules you're
working against directly out of the RPC.
And it's like three steps instead of trying to figure out how to handle all this JSON
and reflection and stuff.
So that's a pretty cool trick too.
So for the dashboard stuff, are you using it?
Presumably it's all JavaScript, right?
Or TypeScript, maybe.
It's almost all go.
The dashboard is a little bit of JavaScript.
You know, the little grid is all written in Canvas.
So that's kind of nasty, but it's fast and it's light.
And, you know...
So Canvas is, for people who don't know, Canvas is like a library, right?
Canvas is a part of the HTML5 spec.
It's you're just drawing with coordinates.
So let's say, you know, make a rectangle from X, Y, you know, and then draw a line from
here, you know, set my color.
So is that dynamic as well?
Or like...
So when you say that, is that setting out the grid in the page like that?
Or is that using the individual elements like that?
Because, you know, for setting out your page, I thought you would have used something like
CSS Grids or something like that?
Yeah, you know, a lot of people have built stuff using like tables and grids and stuff.
This is...it repaints the whole image every time it refreshes though.
So it's big loop over an array of...
A map of arrays, so basically.
So would that be considered light or heavy in terms of resource?
It's super light.
It sips resources.
It's a lot faster than I'm including, you know, a giant graphing library.
You know, I have a few that I really like.
I just didn't find any that would do this kind of grid thing.
So this was an experiment that worked.
So theoretically, like, you could create a similar dashboard to like what Krozeness has
and Bro and Bro.
That's a lot lighter and quicker to load using sort of the process that you've already used
for TenderGD2 and building that web interface, right?
But you just have to use some graphing libraries and stuff to make it a little bit more, you know...
I'm not sure what you want to see.
You talking about like a missed block thing for all the validators or...?
I'll bring it up.
Keep talking, I'm slow.
Yeah, you know, the stuff I did with Canvas was super simple.
You know, it's just a series of sequential rectangles.
Yeah, so I think we're getting into the weeds a tiny bit.
So yeah, because you're looking at like a complex like Grafana UI.
So behind the scenes, like a lot of this is probably being drawn with a graphing library.
The reason it's slow with Grafana is...
Well, depending on what you're feeding it from is probably the source interval from Prometheus.
So it's a combination of BlockTime X, whatever the refresh rate is, plus any throttling,
depending on what tier you're on for Grafana.
Like if you had the same UI tooling that Grafana is using, I don't know, D3 or something probably,
and you had a WebSocket feeding it, like live data coming in at the speed of the data,
it would just be real time.
So it's not really a light client, heavy client, because almost every client now that has any
client has any kind of animation, right, and it's JavaScript is not light, but by definition.
But what Todd's done is used the lowest level version of that tool, which is Canvas.
So at some point, almost all this stuff eventually is Canvas or an abstraction upon HTML5 Canvas,
probably.
So by going all the way down to that, you can write something that is painful to write
but performant rather than having to sort of import a bunch of graphing libraries to
do one thing, if you just want to do one thing.
But if you want to do that one thing and you don't want to rewrite all that stuff from
scratch, then that's probably when you bring in a graphing library, right?
Yeah, I just didn't find any cool graphs that did what I wanted.
So I'm not above bringing in e-charts, which I love, but it's huge, right?
So I wasn't trying to be efficient, it's just a side effect.
It just came out that way.
Cool.
Well, I've noticed we're honing up towards an hour now.
Obviously, we lost about five, 10 minutes due to technical issues at the top of the show.
But should we move on to, we've got a couple of Ask Gamer Nodes questions.
So should we do those?
Just before we do, I just want to ask Todd one more thing, and it'll be real quick.
How do you turn on and off the log, which I know I've seen you with a screenshot of
the log, but looking at the one that I've put up?
Everything is done through a config file, just a YAML file, and it's got a variable for it.
Hide log or something like that, yes or no.
It's pretty well documented.
I mean, I put comments on everything.
Yeah, I saw the log flash up when the page was loading and then disappear, so I must
be like a shortcut or something to turn it on.
Yeah, it's checking out.
It's making a rest call to see if it should hide that or not, because it won't get logs
streamed over the WebSocket if it's disabled.
Cool.
So yeah, I guess move on, DeFray, to Ask Gamer Nodes.
Have we had any decent questions this week or any questions at all?
Only it will.
We've had a couple, but one of the ones we have hanging from the other week is ICS, right?
Because I stirred up quite a bit about ICS, and then we've been talking about it quite
a bit on the show, and there was a lot said about it when we went and asked, and then I
wasn't here that episode afterwards, so we didn't actually talk about it.
But the other thing, there was actually an interesting discussion from Whispernode, which
we got tagged in.
So I think that there is an implication there that we were supposed to discuss it, which
was about DapChains and whether or not they are a good thing in Cosmos, which is, I think
is quite an interesting question.
So essentially, the premise here is, do you need a single application chain to run a single
app, right?
Or can you just...
So yeah, I've been under a rock, and I don't validate Cosmos, so thank you, PupMos.
So the comment for those of you listening to podcast players, almost one hour in, no
mention of Prop72, the willpower is admirable from...
I don't even really also know what Prop72 is.
I read something yesterday, but I just...
Yeah, so I think the context for why we were tagged in this is Prop72, and I think as usual,
Shorthy is probably going to have to explain to us using small words what Prop72 is all
about.
But so the thing that we were specifically tagged in before I tag in Shorthy to tell
us all what the hell is actually going on was essentially whether or not DapChains,
i.e. chains that run only one application, are actually a good idea.
And the quote of the tweet that we were specifically tagged in was, several convos in a couple
of days, my opinion right now is that DapChains and Cosmos dilate human and human, and we
have technical and capital capital in the Cosmos ecosystem and competition amongst
L1s is ripping it apart and will continue to do so because there is not enough of any
of the three to go around yet.
That I think is a really interesting take.
And so I'm like, game on, let's talk about it.
But Shorthy, this is Prop72, is it?
Yeah, you know, to be honest, I haven't really reviewed this too much either.
I've seen that there's a lot of drama around it right now, but I myself haven't yet engaged
with it.
Yeah.
Okay, so there's your answer, PopMars.
The reason we didn't engage with Prop72 until an hour in is because we don't really know
what Prop72 is.
No one knows.
It gives a fuck.
I mean, I guess I can tell DR the drama from what I understand.
Beautiful.
I just don't know the specific of this proposal.
So the drama, I believe, really stems around Gravity Bridge, if you boil it down all the
way, because Gravity Bridge originally was a proposal in Cosmos to implement it.
And then rather than doing something to provide value back to Adam, they created their own
chain, right?
And not only do they create their own chain, they also...
I don't know the specifics, but I think they airdrop to iron holders and stuff that was
just considered very controversial.
And I think that this is considered like Gravity Bridge v2.
So basically giving away a huge portion of the community pool to what will effectively
become something that doesn't necessarily bring value back to Adam.
That's my understanding of where the drama's at right now.
Right.
Is there some...
Was some VC slash Lido slash...
There was some other thing that people were upset about?
Ooh.
I know there's two aspects to...
So my understanding is this prop should have been broken up into two props.
There should have been like...
I want to say a liquid-saking side of it and maybe a DeFi side.
And instead of breaking them up into several ones, they've combined them together.
And my understanding is that the DeFi side is considered acceptable and the liquid-staking
is considered creating more monoliths.
Because Lido has, of course, basically taken over all the theorem-staking.
There's something like 60% of all the theorem-staking goes through Lido at this point.
And the take from my understanding, again, this is from very cursory reading, is that
effectively, you'd be funding another monolith to exist.
Whenever the issue...
One of the issues, of course, being that there's already two competitors here, right?
You have Quicksilver, which is already going to be doing liquid-staking.
You have persistence.
And then you also have Celestia, which is also be providing industry and security already.
And so there's all these competitors already that exist.
So why spend, I want to say, like $1.1 million to do something that's already coming to be?
Ah, so it's quite literally bringing Lido into it.
Interesting.
So a comment said that the proposal is promoting Lido to build on ICS-Secure cosmogen chain.
Interesting.
Okay.
Right.
Okay.
That is interesting.
So I don't really know why that's hyper-controversial other than the Gravity Bridge, I guess, precedent.
Adam Burke's made an interesting comment in the comments saying,
Dapp on its own chain is no longer a Dapp anyway, but an App chain.
Immo.
Yeah.
That's fair.
And I mean, I guess that's the thing, isn't it?
I guess the question is whether the thing that is being objected to here...
Because there's actually...
Because there's two separate things on there.
The idea that you don't have App chains is only possible if you build on, say, a Cosmos
and Substrate or something like that, so you can have lots of Apps on one chain.
Otherwise, if you want to write it in Go, then you should probably just write it in
the SDK.
Right?
So there's...
I think there's quite a few things going on here.
Right?
I mean, if you were building a project tomorrow, right, I think that everybody in this chat
would go about it differently, right?
Because we will come from different language backgrounds.
Like, Todd, you would go native, right, presumably.
Yeah.
Who knows what's going to happen, though, with Cosmos and Go?
I mean...
Well, I mean, Cosmos and Go is just not going to have the mind share, I don't think.
It's going to be kind of bloated compared to Rust or C++, right?
So...
So, I'll say just like, are they going to reimplement all the contracts that they've
already implemented and had audited?
Like...
No.
Without being able to use those like a library, it kind of seems like it just won't get any
much...
And who knows, man?
No land looks really neat, too.
I mean, I looked at some of those contracts and it's like super simple.
So...
But I don't know yet.
Yeah.
What I would love is Rust SDK plus Cosmosm, but...
See, for some of us Rust is too intimidating, man.
Now, I started getting into it and I was a love of it and then I ran into the Barrow
Checker and we won a few rounds and I lost.
So...
What editor are you using?
I was using C-Lion, so...
Because I've got Rust Analyzer, LSP server and Emacs and it just like...
When I screw up the Barrow Checking, it goes...
No.
It goes, you've done a bad thing and you just go sit in the corner while I throw compiler
errors at you until you're very, very, very, very, very in pain.
So...
I can tell myself, I'll learn it once I have time.
Yeah.
I think one of the biggest things is just again, coming from a high level language, the package
management is just like...
It's like all the good things about NPM, most of the bad things removed and it's just very,
very straightforward for a systems programming language, package management experience.
It's really good.
And then the DevTooling is pretty decent, but the best thing is the fact that you can
have macros for serialization, deserialization and that sort of stuff where you literally
go like, here's my enum, give me it and Jason.
In my opinion, macros are one of the worst things that it brought over from C++, right?
So that world, because they're opaque.
Yeah.
I mean, the rule of...
You know what the rule of list macros is?
You don't need a macro, right?
That's the motto.
But there's definitely...
There's a point where the hammer and the nail are no longer working and you need to just
improvise some kind of rail gun type situation to drive the nail.
And at that point, you do need a macro.
And there's definitely something in like Rust where you go like, oh, right.
I can see that this must have been implemented via a macro.
Do I have a desire to write my own?
No.
But it's definitely resolving a problem that I had, I guess.
So yeah, I don't know.
I guess the thing that resonated for me with that comment that came in, I guess, was whether
or not there's an argument at this stage.
And I guess this is the core value out of a chain like Juno or secret is that you should
be able to...
Less so secret, I guess, because in order to maintain the privacy guarantees, you have
to stay on secret.
But the idea that you can bootstrap a product and then move it off when you're ready, the
thing that kind of resonated with me about this comment was that the idea that there
are so few devs in the space that splitting up mind share onto loads of app chains is
like potentially cross purposes.
But then it's kind of back to what you were saying about Unix and the original tender
duty, right?
Do one thing, do it well.
And that's the other reasoning behind an app chain, right?
Yeah.
And the one thing that kind of struck me when I heard him say it was, I think the whole
point of Cosmos is this democratization of blockchains and IBC has really helped realize
this new internet model.
And I think it's great.
Is it resulting in a bunch of shut chains?
Yeah.
I don't know.
So yeah, and Pop was just saying Lido would dramatically restructure the validator set
and be detrimental to delegator suffrage.
I have to say, I don't know enough about how Lido works in order to sort of talk about
that any further, to be honest.
But so my understanding is though Lido bringing the chain online, they're going to use ICS,
right?
So this is presumably intended to be the first out in the wild use of interchange security?
I have no idea there, but the idea of flooding the voting power resulting in a reduction in
delegator suffrage I think is really interesting.
I mean, the irony here of course is that we've had a conversation here in this chat probably
every single time we've met to talk about how we could resolve the fact that the top
five of a chain of most Cosmos genes can help the network.
Yeah, flood through the validator set doesn't seem like necessarily a problem to me.
I think that kind of sounds like a good.
Yeah.
I actually want to address the comment earlier before as well about the brain drain.
So I think that I think it's a really apt point, but I think that it's a little bit.
The conversation is unfortunately nuanced.
There's actually a Cosmos SDK that I expect out that effectively I want to develop as
an SDK because it could benefit all chains.
The problem is there's no funding for it.
So I've gone to several different building funders, I guess, grantees, and everyone's
like, this is an amazing idea.
I think you should profit.
We cannot fund you for it.
If you want to create your own chain that only does this, great.
If you want to build it on Adam or on Juno or on Osmosis, we can't fund that.
We can't fund it at all.
So really the only recourse for getting funding is building your own chain.
And it does have that side effect of there is that brain drain.
But I think that a more interesting side effect is that, not side effect, but observation
is that if you look at most Cosmos chains, get hubs, they have like last six months,
they've done no development.
So it's kind of hard to claim there's a brain drain whenever they aren't actually doing
any development.
So I'm curious now, what was your idea about the SDK?
Oh, basically, I mean, it's fairly simple.
The idea is essentially, if you've used Venmo or Zell or Cache app or whatever, a request
feature.
So create another module where you can essentially wrap any other transaction in a request.
So let's say you're shooting down your validator and you put out a 30-day notice saying, hey,
I'm shooting down my validator.
You could then send out a request to all your delegators with an undeligated or a redelegate
transaction, which they would then accept and then be able to go through and redelegate
their funds without having to go through any sort of rigmarole around staking.
Or even if, let's just say you and I went out to coffee or whatever, and I paid for
lunch, I could send you a one Juneau request to pay me back.
Just something simple like that, but wrapping all transactions.
So kind of like AuthC does for allowances, kind of just a way of flipping the flipping
that I want to make this type of message of Cosmos message type instead saying, I would
like you to acknowledge this by sending me back the Cosmos message type I've requested.
Yeah, exactly right.
And the idea would even be that you could add a chain of events so be able to send to
multiple people, you just send an adjacent blog and it would give it out to everyone
that it needs to go out to and then they could just respond and execute it.
Yeah, that's kind of cool.
But as I guess you've touched on the incentives are quite skewed at the moment, aren't they?
Which this is actually, the interesting thing is that as well, launching a smart contract
on a smart contract chain in Cosmos is also economically not as beneficial.
And especially if you want to get investor funding like for how social and stuff like
we are bootstrapping it ourselves.
But I know that it also makes sense to launch on a chain where there is a large user base
already rather than trying to bootstrap your own.
But if you were going after investment money, it will be a lot easier to do it as a single
shot chain with only one permissioned smart contract running on it.
And I think there have been a few projects that have moved over to Cosmos from Terra that
are basically going that route where they're going to just bootstrap as their own chain
like one shot and they're going to bootstrap with probably almost no users and maybe die
in that wilderness because their investors are like, got to be a blockchain.
So it cuts both ways, doesn't it?
Yeah, I mean, my vision for why Juno is great is effectively my imagining of what a chain
should do or what an app should do is it should use Juno as or secret or in a generalized
smart contract platform, create their platform, create their community, create their base.
And if they find that they need to expand out, then they can go do that.
And my vision of what Juno should be is effectively that sandbox for lack of a better term to create
that community, especially when it already has such a strong community.
And so if there ends up being that desire where you find some reason to bounce out, that's
when you do it.
I think it's kind of a, the question is like a short term to long term goals to me.
I don't think that necessarily all new chains are short term oriented.
I certainly don't want to make that claim.
But I think that if you're going for like a really long term project, build somewhere
that already exists and expand if you need to versus if you need cash now, you would
just create your own chain and then dump on retail effectively.
Yeah.
So moving on to the interchange security bit, because that seems related to this 72 thing.
So the background to interchange security and why we're bringing up on the show is because
I said, I went on Twitter and just went, okay, don't really understand why we need
interchange security because it's not really hard to get a validator set.
And if your chain doesn't work out, that's too bad.
But like finding validators, admittedly potentially varying qualities is not actually a problem
that has existed for some time.
And so I was like, what is the actual use case?
What is the problem that ICS is solving?
And I guess the thing that I guess I wasn't really considering as part of this is from
somebody who's more used to working in a smart contract environment.
I'm used to having what Shorthy just described, which is an on ramp for an idea into a community.
But if you're building stuff at chain level, at SDK level, then maybe ICS is a better fit
for that.
And Twitter was very quick to tell me I was wrong in about 15 different ways.
And so I'm going to read out some of the responses that were very good on why we need
interchange security.
So I'm going to just do some call outs here.
PhonicG made some really, really good points.
I can't put them all in full because it would take me ages to read out a thread around sort
of market cap and stuff like that.
Well, XBT also said having a small market cap is dangerous when starting out.
And then Simon Water, friend of the show from Confio from Cosmosm, said that he essentially
backed up that this was on a longer thread.
Saying a small market cap means you can buy out the chain.
It's an important attack factor to consider in a bear market or for new projects.
And I think, again, especially tail end of last year, a lot of the chains that were launching,
hua hua, right?
Huge market cap.
So you look at that and you go, where does interchange security fit into this when a
literal meme chain can have a multi-hundred million dollar market cap?
It becomes more obvious in a bear market.
So thank you, Simon, for pointing that out.
Eric Opp said there's pros and cons to ICS and chains should decide and added a hot take
on the end, right?
And this one I like.
Interchange security is Atom's last chance to stay relevant after gravity decks and gravity
bridge failure.
If ICS isn't a hit, then it's over.
Spicy hot take.
So we continue on there, the Freya.
I've got a comment just about the market caps and buying out a chain.
I think that it probably would come cost prohibitive even for a shit chain because once you start
to buy all the tokens specifically with how a decks functions, you are going to get cost
prohibitive pretty quick.
You can't just keep buying the tokens without the market cap rising significantly.
But that's why they halted Terra.
I mean, it was happening.
Terra worked differently though.
It was like a mechanism where you cash in the UST and it was a new look.
That's right now.
Terra Classic, you can't register a new validator.
You can't delegate new tokens to a validator.
You can redelegate, but otherwise that was all a hard fork that went into the code because
once it devalued, it was possible to take over.
So yeah, but it devalued to basically nothing.
And the distribution of tokens is all over the place because of the way that that mint
printing module was working when you were cashing in UST.
But I think for regular chains, most of them aren't even on a sex, but even on a CAX, once
you start to try and buy up a meaningful amount of tokens, it's just going to push the price
sky high unless you do it over a really, really, really long time, which I think then people
would start to notice before you got anywhere near a big enough amount to actually do some
damage to a chain.
I wonder if you, if right, it's okay, like if I, okay, let's imagine I have loads of
money and I'm a bad guy, I'm a real bad dude.
I would imagine the way to go about it would be to try and over the counter purchase from
founders of a chain, but go to the niche individually and just gamble that they didn't tell each
other that they were doing it out of either embarrassment or sneakiness.
And then just see if you can get 33% because a lot of chains do have that much vested in
the founding team.
You know, so that might be possible, but imagine this right.
What's the point?
I'm imagining I've got my imagination hat on.
So if you go and get all this effort right to get all this token and spend all this money
to get all this token, why are you going to maliciously attack a chain that you own all
the token for?
If you drain the chain, you can't possibly fucking sell it without killing the price
of it anyway.
It just doesn't make any sense.
But I think that's the point though, right?
So like, D, DYDX is coming to Cosmos, right?
They're probably going to be, as soon as they spin up $100 million project.
But there's already other DeFi projects.
Let's say they're worth 1 million.
They might just buy all the tokens just to literally kill the project.
I think that's the very concern rather than some guy deciding that he likes the project
enough to kill it.
It's a computing entity that says, this is a really great product and they could take
our share.
We can't have that.
I submit to you that if a project's worth a million dollars, it's worth killing.
I mean, in crypto, yeah.
In normal startup land, I would feel pretty proud to start a million dollar company.
I'd be like, this is kind of nice.
Yeah, million dollar company mate.
Maybe before the crash.
So I just want to pull out a few comments we've had in the chat.
So Putmar says, interchain security comes with a sacrifice of self sovereignty, which
try and save very quickly.
That's quite tricky.
A lot of S's in that one.
Gravity Bridge is a prime candidate for interchain security because its token does not provide
sufficient economic security to the collateral it holds.
Juno.Love says, Fortis was worth 35 billion.
My brain just halted.
Is that a billion?
Yeah, it's a.
Yeah, on paper.
By the way, do you like my imagination hat?
This is my imagination hat.
And it looks very special.
Yeah, it's very sensational.
And then the variety of people in the chat have basically been advocating different DGEN
strategies for attacking a chain.
So we've obviously started something with that line of questioning.
So a final couple of points I had from other people, but read out in my voice so that the
clever points will attach themselves in your mind to me.
But Trill and Juno News said that ICS shouldn't be viewed as an end in itself.
You should have the set as interim when you go when you move to an app chain and then
you can get different new validators.
So I think what they're referring to there is that there's a second version of ICS coming
later that allows you to layer some validates from one set and have new validators.
So that's an interesting development as and when it comes.
And then the final point here, this is a good one, is from H. Devalence, who's one of the
founders of Penumbra.
And one of those people who's very, very clever and whenever you use crypto libraries in Rust,
you see their GitHub handle come up all the time to remind you that you are just a small
brain grug developer.
But they said app specific chains can have their own security and be fully self-reinforced
and pay the cost of that, or they can be secured by the hub or an ICS chain.
And this is the really powerful part.
They can choose somewhere right in the middle and they can change their choice at any time.
And that is a reference again to layering.
So yeah, those are all of the tepids to hot takes that I thought were worth mentioning
on the ICS thing.
I'm kind of going to go and say that I think I stand corrected on ICS.
I think there is a use case for it.
I think I just didn't see it initially because I was blinked by Cosmosm and by only building
on top of smart contract chains.
Well, yeah, it's like horizontal scaling, right?
You can only scale so big on Cosmosm before it's bloated and can't do shit.
Well, I mean, the other side of the conversation is calling Interchain Security worthless,
calling Polkadot worthless?
Yes.
That was the deep desire of the fray when he said that comment originally is that he
was throwing shade at Polkadot indirectly.