1
00:00:00,000 --> 00:00:01,240
not shopping before.

2
00:00:02,280 --> 00:00:04,320
No, the fucking internet just keeps dropping out.

3
00:00:07,600 --> 00:00:12,640
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent

4
00:00:12,640 --> 00:00:13,640
validated teams.

5
00:00:19,360 --> 00:00:24,760
Hello, welcome to Game of Nodes, a weekly podcast on the cosmos from independent

6
00:00:24,760 --> 00:00:25,960
validator streams.

7
00:00:26,680 --> 00:00:28,160
Independent validator streams?

8
00:00:28,200 --> 00:00:29,680
Independent validator teams.

9
00:00:29,680 --> 00:00:32,560
Ah, that was nearly professional for the first time in our history.

10
00:00:34,040 --> 00:00:40,840
We're here as always, well, as mostly always with a very frozen null with

11
00:00:40,840 --> 00:00:46,840
Schultzie and with Todd, who's going to be talking to us about the new release of

12
00:00:46,840 --> 00:00:51,360
Tender Duty 2, the hottest product on the street for the validator that wants to

13
00:00:51,360 --> 00:00:52,520
monitor their nodes.

14
00:00:53,600 --> 00:00:59,080
But first up, we've got some follow-ups for next week from last for next week.

15
00:00:59,080 --> 00:01:03,920
Well, my sense of time has obviously got very screwy from last week.

16
00:01:05,560 --> 00:01:08,640
One thing I've added to that as a kid, one of the things I have the most

17
00:01:08,640 --> 00:01:12,680
difficulty with is telling the difference between like, I did this yesterday

18
00:01:12,680 --> 00:01:13,840
versus I did it tomorrow.

19
00:01:13,960 --> 00:01:17,120
And so I'd be talking about things I'm going to do or I did and I'd be saying

20
00:01:17,280 --> 00:01:18,960
tomorrow I did X, Y and Z.

21
00:01:19,400 --> 00:01:24,880
So I don't know why I did that, but just with you talking about going a little

22
00:01:24,880 --> 00:01:26,000
bit off, I totally relate.

23
00:01:26,000 --> 00:01:30,680
Is there a specific thing where children are less aware of the passage of time

24
00:01:30,680 --> 00:01:34,640
or something, or is it just that we're both bad at remembering which is the

25
00:01:34,640 --> 00:01:35,280
correct word?

26
00:01:38,600 --> 00:01:40,880
There's probably something to do with that.

27
00:01:40,880 --> 00:01:44,280
I mean, children don't get object permanence until they're like, what, four?

28
00:01:44,320 --> 00:01:47,200
So, or maybe let, I don't know.

29
00:01:47,200 --> 00:01:48,120
I don't know, children.

30
00:01:48,120 --> 00:01:48,640
I don't know.

31
00:01:49,160 --> 00:01:50,760
Yeah, I don't know.

32
00:01:51,640 --> 00:01:55,120
Anyway, that was a bit of a weird divergence caused by me.

33
00:01:55,120 --> 00:01:57,920
Doing the verbal equivalent of a typo.

34
00:01:58,920 --> 00:02:03,640
So we've, we've got one of the biggest follow-ups we've got from last week is

35
00:02:03,640 --> 00:02:09,760
a variety of stuff on the Validator League discussion.

36
00:02:10,280 --> 00:02:15,120
Because I gather that's moved on a little bit and they're now restructuring.

37
00:02:15,360 --> 00:02:16,080
Is that right?

38
00:02:16,080 --> 00:02:22,400
The, the, and I think we're possibly even going to have ghost and possibly

39
00:02:22,400 --> 00:02:27,360
Tyborg coming on the show on a future episode actually to talk about how that

40
00:02:27,360 --> 00:02:29,000
thing, how that stuff is evolving.

41
00:02:29,000 --> 00:02:29,920
Is that right, folks?

42
00:02:32,920 --> 00:02:34,160
Yeah, I believe so.

43
00:02:34,160 --> 00:02:37,600
I think next week, but not starting.

44
00:02:39,120 --> 00:02:39,600
Yeah.

45
00:02:39,800 --> 00:02:45,000
And those and eagle eye viewers will have spotted that there is no usurper here

46
00:02:45,000 --> 00:02:47,760
because he is, I think, stuck in a hotel lobby somewhere.

47
00:02:47,760 --> 00:02:49,760
We presume drinking whiskey on his own.

48
00:02:49,760 --> 00:02:55,040
So one of the, one of the other Nulls frozen again.

49
00:02:59,040 --> 00:03:00,400
I think we have just lost him.

50
00:03:00,400 --> 00:03:08,720
Let's, let's move on while Null is frozen to, to talk about tender duty too.

51
00:03:08,720 --> 00:03:15,760
So we've got Todd with us who, so most, most validators have the problem of,

52
00:03:15,760 --> 00:03:20,360
of like, obviously monitoring their nodes.

53
00:03:20,360 --> 00:03:26,160
And I think it's pretty common that people are going to use something like Grafana

54
00:03:26,160 --> 00:03:31,760
or Zabix, Prometheus as a sink into the, into one of those two services.

55
00:03:31,760 --> 00:03:36,360
I guess I'm guessing we're on here mostly using Zabix.

56
00:03:36,360 --> 00:03:37,760
Is that right?

57
00:03:37,760 --> 00:03:39,360
Okay.

58
00:03:39,360 --> 00:03:39,960
Yeah.

59
00:03:39,960 --> 00:03:40,960
For monitoring.

60
00:03:40,960 --> 00:03:45,560
And then the other part of observability is, is obviously notifications when we

61
00:03:45,560 --> 00:03:46,560
go wrong.

62
00:03:46,560 --> 00:03:51,360
That's where Todd comes in with the incredibly useful tender duty tool for

63
00:03:51,360 --> 00:03:54,360
monitoring cosmos validator nodes.

64
00:03:54,360 --> 00:04:00,560
It's used very extensively by people on the show and around the show.

65
00:04:00,560 --> 00:04:02,560
And everyone's very appreciative for it.

66
00:04:02,560 --> 00:04:05,960
And there is a new version that is just out.

67
00:04:05,960 --> 00:04:11,360
So we thought we'd get Todd on the show to talk about what's, what's new in tender duty too.

68
00:04:11,360 --> 00:04:12,760
So yeah, thanks man.

69
00:04:12,760 --> 00:04:17,360
And I'll, I'll, I'll show my screen, but I'll try to narrate thoroughly since I know some

70
00:04:17,360 --> 00:04:19,360
people listen on audio only.

71
00:04:19,360 --> 00:04:26,760
So tender duty kind of, it is one of those things where it's like, I kept

72
00:04:26,760 --> 00:04:32,160
finding myself looking at Mint scan or ping dot pub to see if I was missing blocks.

73
00:04:32,160 --> 00:04:34,360
It's like, Hey, how am I doing on blocks?

74
00:04:34,360 --> 00:04:37,160
And it was like, this is not okay.

75
00:04:37,160 --> 00:04:38,960
There's got to be something for this, right?

76
00:04:38,960 --> 00:04:47,760
So, yeah, so let's see, how do I share my screen?

77
00:04:47,760 --> 00:04:49,960
Oh boy.

78
00:04:49,960 --> 00:04:51,960
All right.

79
00:04:51,960 --> 00:05:00,760
It requires sharing my whole screen, which is a 47 inch monitor.

80
00:05:00,760 --> 00:05:01,960
So I don't know if that's going to work.

81
00:05:01,960 --> 00:05:07,760
Tell you what, I can jump back on from a different computer real quick that, that makes

82
00:05:07,760 --> 00:05:09,760
this easier.

83
00:05:09,760 --> 00:05:11,160
And it looks like Noel's moving again.

84
00:05:11,160 --> 00:05:14,560
So I'll pick it up here in a minute.

85
00:05:14,560 --> 00:05:15,760
Is that all right guys?

86
00:05:15,760 --> 00:05:19,360
Yeah, no worries.

87
00:05:19,360 --> 00:05:28,360
He really just asked me before, is that, you know, before I was frozen?

88
00:05:28,360 --> 00:05:29,160
What do you say now?

89
00:05:29,160 --> 00:05:35,360
Sorry.

90
00:05:35,360 --> 00:05:38,760
It's plenty of technical difficulties with this issue.

91
00:05:38,760 --> 00:05:45,560
So, yeah, before the show started, we were talking about, we were talking about Horcrux

92
00:05:45,560 --> 00:05:52,640
as well, because it obviously has been a very common topic on the show is remote signers

93
00:05:52,640 --> 00:05:57,360
and using that kind of infrastructure to secure your nodes that you don't have to have key

94
00:05:57,360 --> 00:06:00,360
master on a given box.

95
00:06:00,360 --> 00:06:06,360
And sure, so you've finally made the jump, I think maybe the first out of all of the

96
00:06:06,360 --> 00:06:11,360
Game of Nodes crew to make the jump to full Horcrux, right?

97
00:06:11,360 --> 00:06:12,360
Yeah.

98
00:06:12,360 --> 00:06:19,360
So I guess for context, when you run a validator, there's kind of like three stages of key signing.

99
00:06:19,360 --> 00:06:24,360
There's what most people do, which is you leave the key, which is like the way your validators

100
00:06:24,360 --> 00:06:27,360
assigned on the validator machine itself.

101
00:06:27,360 --> 00:06:30,360
And then the validator kind of signs for itself.

102
00:06:30,360 --> 00:06:33,360
And then the second stage is you have a remote signer.

103
00:06:33,360 --> 00:06:37,360
So there's a software called Tenderman Key Management System, which that says then you

104
00:06:37,360 --> 00:06:41,360
have like your node running, and then you have a second node running that points to the

105
00:06:41,360 --> 00:06:45,360
first one that says, hey, I'm going to sign the blocks for them.

106
00:06:45,360 --> 00:06:50,360
And what that does is it kind of gives a layer of indirection in terms of security.

107
00:06:50,360 --> 00:06:56,360
You can keep the remote signer really locked down and then the node less locked down and

108
00:06:56,360 --> 00:06:58,360
you're still secure.

109
00:06:58,360 --> 00:07:01,360
Another benefit with that is if your node goes down, you can just point it to another one

110
00:07:01,360 --> 00:07:02,360
really easily.

111
00:07:02,360 --> 00:07:09,360
And the third solution is you can have multiple signers pointing to multiple nodes to make

112
00:07:09,360 --> 00:07:12,360
sure that you're always signing in some capacity.

113
00:07:12,360 --> 00:07:19,360
So with Horcrux, it's that third level where I can have basically two nodes go down and

114
00:07:19,360 --> 00:07:21,360
still be signing perfectly fine.

115
00:07:21,360 --> 00:07:25,360
Similarly, I can have multiple of the signers go down and still be signing.

116
00:07:25,360 --> 00:07:30,360
So basically it's a method for ensuring that your blocks are just always signed no matter

117
00:07:30,360 --> 00:07:33,360
what, short of something utterly catastrophic.

118
00:07:33,360 --> 00:07:35,360
Right.

119
00:07:35,360 --> 00:07:37,360
Hey, Todd.

120
00:07:37,360 --> 00:07:41,360
Back on different machine.

121
00:07:41,360 --> 00:07:50,360
So we were just talking about Schultz's migration to Horcrux and he's just explained basically

122
00:07:50,360 --> 00:08:00,360
what the benefit of essentially a cluster of threshold signers is over a single box running

123
00:08:00,360 --> 00:08:03,360
something like TMKMS.

124
00:08:03,360 --> 00:08:10,360
So the interesting thing as well is like there's all of the, it's obviously there's a threshold

125
00:08:10,360 --> 00:08:13,360
signer and that's all that's a form of raft and whatnot.

126
00:08:13,360 --> 00:08:20,360
But also just the even to actually create the key matter, you basically need additional

127
00:08:20,360 --> 00:08:24,360
to shard the key in a particular way and stuff like that as well, don't you?

128
00:08:24,360 --> 00:08:31,360
And so like there's quite a lot of steps involved in actually making that a viable production

129
00:08:31,360 --> 00:08:33,360
ready thing you can do, right?

130
00:08:33,360 --> 00:08:35,360
Yeah.

131
00:08:35,360 --> 00:08:40,360
But the, I think the thing that we were also talking about earlier was that the there was

132
00:08:40,360 --> 00:08:45,360
some quite unexpected behavior in terms of what you saw when you actually sort of ran

133
00:08:45,360 --> 00:08:47,360
it in production, right?

134
00:08:47,360 --> 00:08:49,360
Yeah.

135
00:08:49,360 --> 00:08:52,360
So for test net, it was absurdly stable.

136
00:08:52,360 --> 00:08:58,360
They are Horcrux itself is typically pretty CPU bound.

137
00:08:58,360 --> 00:09:03,360
In whenever I run on test nets, because I set up on my tests as first, it ran perfectly

138
00:09:03,360 --> 00:09:04,360
stable.

139
00:09:04,360 --> 00:09:07,360
I could run, I ran it for almost a week because just the test nets and everything were fine.

140
00:09:07,360 --> 00:09:10,360
So I set up on the main net.

141
00:09:10,360 --> 00:09:12,360
I was using AWS instances.

142
00:09:12,360 --> 00:09:17,360
And at around 4am every morning, one of the Horcrux signers would just decide it was time for it to

143
00:09:17,360 --> 00:09:23,360
completely peg out on its resource usage, which would cause all sorts of misblocks.

144
00:09:23,360 --> 00:09:26,360
And it was even inconsistent across which day because it's happened in the last four

145
00:09:26,360 --> 00:09:28,360
days in a row.

146
00:09:28,360 --> 00:09:31,360
Some days it'd be the first signer, some days it'd be the second signer, some days it'd be the third

147
00:09:31,360 --> 00:09:33,360
signer.

148
00:09:33,360 --> 00:09:38,360
It's still ongoing as to what the issue was, but hopefully it is more or less resolved as I've

149
00:09:38,360 --> 00:09:44,360
switched from using AWS instances, which can be kind of on the fly versus using bare

150
00:09:44,360 --> 00:09:50,360
metal machines for the signers as well, which is also cheaper because AWS is absurdly

151
00:09:50,360 --> 00:09:52,360
expensive.

152
00:09:52,360 --> 00:09:56,360
So where do you have your bare metal machines at for Horcrux?

153
00:09:56,360 --> 00:10:00,360
I now have one on Volter, one on OVH Cloud and one on Hetzner.

154
00:10:00,360 --> 00:10:06,360
So one in Helsinki, one in Virginia and one in, I want to say Oregon.

155
00:10:06,360 --> 00:10:14,360
So do you have a whole bare metal machine just for Horcrux signer?

156
00:10:14,360 --> 00:10:17,360
No, I've already got instances for those either way.

157
00:10:17,360 --> 00:10:23,360
So Volter and OVH Cloud hold, they have bare metal machines that can work with secret.

158
00:10:23,360 --> 00:10:29,360
And so I'm basically reusing those to also be used as a signer.

159
00:10:29,360 --> 00:10:31,360
So yeah.

160
00:10:31,360 --> 00:10:42,360
And I think that's the original intent of Horcrux is to have it alongside nodes on actual,

161
00:10:42,360 --> 00:10:46,360
instead of having them on dedicated instances anyway.

162
00:10:46,360 --> 00:10:48,360
So is it?

163
00:10:48,360 --> 00:10:50,360
Yeah.

164
00:10:50,360 --> 00:10:58,360
Well, when Jack used to originally talk about it, he used to say that you'd deploy it next to a node.

165
00:10:58,360 --> 00:11:09,360
Because, you know, we use it in a bit of a D gen way in that the original intent is that you would have a swarm of signers for one network,

166
00:11:09,360 --> 00:11:16,360
not, you know, sharing it across a couple of networks like running multiple signers on one node.

167
00:11:16,360 --> 00:11:30,360
So the original architecture intent was to launch it out next to a node running on an instance somewhere.

168
00:11:30,360 --> 00:11:36,360
And then it, like that signer would just talk to that node and the other signers.

169
00:11:36,360 --> 00:11:41,360
And then the other signers would talk to their node and the other signers.

170
00:11:41,360 --> 00:11:48,360
So it wouldn't be like, so you'd have the lowest latency to the node and the network, but then they could talk to each other.

171
00:11:48,360 --> 00:11:49,360
Right.

172
00:11:49,360 --> 00:11:54,360
So it's like having three centuries, each of which has a signer.

173
00:11:54,360 --> 00:11:57,360
None of the signers, none of them have the keymatter.

174
00:11:57,360 --> 00:12:01,360
So it doesn't matter that you've got three extra things out in the world, right?

175
00:12:01,360 --> 00:12:02,360
Yeah.

176
00:12:02,360 --> 00:12:06,360
So the signers only have a portion of the key matter on there.

177
00:12:06,360 --> 00:12:10,360
It's impossible to put back together unless you've got like them all.

178
00:12:10,360 --> 00:12:13,360
For some reason, my mental model was totally broken.

179
00:12:13,360 --> 00:12:18,360
I thought the whole point was like, it was more like Kafka and ZooKeeper.

180
00:12:18,360 --> 00:12:30,360
So like you deploy three signers to a cluster and then they point at a cluster of centuries where if one of those centuries goes down,

181
00:12:30,360 --> 00:12:36,360
you round rob into the next one or you just round rob in them via proxy anyway.

182
00:12:36,360 --> 00:12:41,360
But the cluster of signers is going, let's roll.

183
00:12:41,360 --> 00:12:50,360
So there's many ways to skin a cat with hall crocs and it just depends on how much of a degenerate you are on the ultimate.

184
00:12:50,360 --> 00:12:55,360
And a lot of people do actually keep their signer on the same machine still.

185
00:12:55,360 --> 00:12:56,360
That's super common.

186
00:12:56,360 --> 00:12:57,360
Yeah.

187
00:12:57,360 --> 00:13:02,360
I was talking to someone about it today about a script that they wrote basically for setting up their machines doing that.

188
00:13:02,360 --> 00:13:03,360
Yeah.

189
00:13:03,360 --> 00:13:14,360
I haven't deployed hall crocs on any main nets yet, but that's how I've got my Ansible setup.

190
00:13:14,360 --> 00:13:20,360
One signer, one node, same box.

191
00:13:20,360 --> 00:13:30,360
So, Schiltzi, I'm not sure if I tagged you yesterday in the chat, but there's Dan B&B.

192
00:13:30,360 --> 00:13:34,360
It's going to be a Kujira validator.

193
00:13:34,360 --> 00:13:41,360
And he had this hectic Terraform scripting to launch everything in one click.

194
00:13:41,360 --> 00:13:43,360
It was crazy.

195
00:13:43,360 --> 00:13:47,360
But I think it was alongside the nodes as well.

196
00:13:47,360 --> 00:13:50,360
I haven't seen that, but it sounds absolutely fascinating.

197
00:13:50,360 --> 00:13:54,360
Terraform is one of those things that where I see it and I'm like, oh, someone knows what they're doing.

198
00:13:54,360 --> 00:13:56,360
And then you look at it and you're like, I don't know what's going on.

199
00:13:56,360 --> 00:13:57,360
They might know what they're doing.

200
00:13:57,360 --> 00:13:58,360
They might not.

201
00:13:58,360 --> 00:13:59,360
I don't know.

202
00:13:59,360 --> 00:14:04,360
I was literally like, say this is the timeline, right?

203
00:14:04,360 --> 00:14:10,360
So, I was talking to him about some stuff and then he asked me about TMKMS, right?

204
00:14:10,360 --> 00:14:16,360
Because I had that script up there that makes it pretty easy.

205
00:14:16,360 --> 00:14:19,360
And I said, well, if you want to know about it, have a look at that, read through it.

206
00:14:19,360 --> 00:14:21,360
I talked to him a couple of times that day, right?

207
00:14:21,360 --> 00:14:25,360
And then he had TMKMS up on the test net and he was pretty happy with it.

208
00:14:25,360 --> 00:14:33,360
And then I said, if you want a better solution, maybe have a look at Horcrux and gave him the link to Horcrux.

209
00:14:33,360 --> 00:14:37,360
Like two days later, he's like, oh man, I've spent the whole day scripting.

210
00:14:37,360 --> 00:14:40,360
You'll be pretty impressed with my setup.

211
00:14:40,360 --> 00:14:45,360
And he sent me the link to his repo on GitHub.

212
00:14:45,360 --> 00:14:46,360
I was like, holy fuck, man.

213
00:14:46,360 --> 00:14:50,360
He's written like 300 lines just for the config file.

214
00:14:50,360 --> 00:14:56,360
And then there's like four other files that like make this thing work.

215
00:14:56,360 --> 00:14:58,360
I was like, holy shit, dude.

216
00:14:58,360 --> 00:15:00,360
You did this today.

217
00:15:00,360 --> 00:15:04,360
And he's like, yeah, start this morning.

218
00:15:04,360 --> 00:15:09,360
Yeah, Terraform is pretty verbose though.

219
00:15:09,360 --> 00:15:16,360
To be fair, like I did a Spaces Within the other day and he is like a DevOps genius.

220
00:15:16,360 --> 00:15:25,360
So he's like new to validating, but like a DevOps is pretty good.

221
00:15:25,360 --> 00:15:28,360
Yeah, I wish he could come on my team.

222
00:15:28,360 --> 00:15:30,360
Yeah, I'm sure he is.

223
00:15:30,360 --> 00:15:38,360
There's definitely a thing though with Ops stuff that a lot of the, especially the automation scripting tools are.

224
00:15:38,360 --> 00:15:42,360
There's a combination of things other than that.

225
00:15:42,360 --> 00:15:50,360
For the purposes of this conversation, they tend to be quite verbose because obviously their configuration management and all that kind of stuff.

226
00:15:50,360 --> 00:16:02,360
But the more pertinent thing maybe for this is that there are really heavy diminishing returns on something that you don't need to DR very often.

227
00:16:02,360 --> 00:16:13,360
And there have definitely been a couple of times in my career where I've spent like a month getting a completely like rock solid set of Terraform scripts or Ansible scripts or whatever for a thing.

228
00:16:13,360 --> 00:16:19,360
Only to then find that, oh no, we need to change this major thing about the way we deploy the software.

229
00:16:19,360 --> 00:16:22,360
And it's like, right, that's basically all in the bin.

230
00:16:22,360 --> 00:16:24,360
Script that.

231
00:16:24,360 --> 00:16:34,360
Yeah, I mean, like you want to deploy like two database instances behind Nginx showing my age a little bit there.

232
00:16:34,360 --> 00:16:37,360
Perhaps I don't know if people are still using it as a reverse proxy all the time.

233
00:16:37,360 --> 00:16:42,360
And you like script all of that blah, blah, blah, blah, happy days.

234
00:16:42,360 --> 00:16:46,360
And then a month later it's like, actually, we're just spending a lot of time on database ops.

235
00:16:46,360 --> 00:16:49,360
Should we just put it on Amazon RDS?

236
00:16:49,360 --> 00:16:54,360
And you're like, yeah, I mean, we can do that.

237
00:16:54,360 --> 00:16:59,360
But we have to rewrite all of our database deployment scripts.

238
00:16:59,360 --> 00:17:03,360
And it's like, yeah, we just do it by the Amazon console this one time.

239
00:17:03,360 --> 00:17:16,360
And then as anybody who's actually deployed any of the, you know, the highly managed AWS RDS instances knows, you deploy it by the console and then you write a script in case it goes down.

240
00:17:16,360 --> 00:17:19,360
And it never fucking goes down ever.

241
00:17:19,360 --> 00:17:29,360
And then so you're just like, and then you know, somebody asked you at conference, how do you how do you manage your, your thingy thingy and you're like, yeah, I mean, we've got like 1200 lines of terraform.

242
00:17:29,360 --> 00:17:32,360
But to be honest, we just deployed it by the console.

243
00:17:32,360 --> 00:17:38,360
So I would love to pretend that we have some really shiny stuff going on.

244
00:17:38,360 --> 00:17:41,360
But that's sort of paranoia really.

245
00:17:41,360 --> 00:17:57,360
So my only comment to him was that he's like, you know, here it is, do you have any comments that I'm like, to an AWS, you're going to be broke real quick.

246
00:17:57,360 --> 00:18:01,360
Yeah, I mean, to be fair, I only want AWS because I got the $1,000 startup credit.

247
00:18:01,360 --> 00:18:04,360
So I was like, yeah, I can just blow through these and nothing matters.

248
00:18:04,360 --> 00:18:10,360
But apparently, apparently what I was using would be I think $200.

249
00:18:10,360 --> 00:18:12,360
A month per signer.

250
00:18:12,360 --> 00:18:19,360
And I was like, honey, doctor my side away anyway, even if it's free, I can't stomach it.

251
00:18:19,360 --> 00:18:22,360
Yeah, I was dollars, but get you far.

252
00:18:22,360 --> 00:18:30,360
No, for us, I think we're going to spray it around digital ocean in multiple locations on digital ocean.

253
00:18:30,360 --> 00:18:31,360
See how that goes.

254
00:18:31,360 --> 00:18:35,360
But I'm doing like, I'm doing stepping stones.

255
00:18:35,360 --> 00:18:42,360
So I was on TM KMS, now I'm like single sign off all cracks pointed at multiple nodes.

256
00:18:42,360 --> 00:18:51,360
And the next step will be like, once I'm comfortable with that, the next step will be working out how I'm going to reliably do multiple signers.

257
00:18:51,360 --> 00:18:56,360
But yeah, it should be pretty, pretty interesting.

258
00:18:56,360 --> 00:18:58,360
But I need to work out my answer woman.

259
00:18:58,360 --> 00:19:02,360
I need to watch the rest of them videos.

260
00:19:02,360 --> 00:19:14,360
Yeah, I mean, there's also like, I think one of the things that I'm quite interested in trying to do is like the whole Akash thing, which obviously been experimenting quite a bit with recently, but I,

261
00:19:14,360 --> 00:19:22,360
I, you can't, well, I mean, you can, but it's going to be hugely complex to deploy.

262
00:19:22,360 --> 00:19:33,360
And you have then whole cracks signers in that containerized environment in like mini pod with the, with the card valid with the Akash with it, so with the validators that are deployed into a cash like by the time you've done that,

263
00:19:33,360 --> 00:19:43,360
you may as well actually just buy a couple of really big metal boxes and actually do it yourself because I bet dollars to notice that we cheaper than running Anakash

264
00:19:43,360 --> 00:19:53,360
Yeah, after your experiments on a cache, they turned me off a little bit.

265
00:19:53,360 --> 00:20:02,000
I'll say that deploying my website on there, not granted, because it requires zero resources,

266
00:20:02,000 --> 00:20:04,880
was ridiculously smooth.

267
00:20:04,880 --> 00:20:09,840
When provider went down, when together went on, it took me all of a minute to redeploy.

268
00:20:09,840 --> 00:20:18,240
You can deploy it with unrelated to websites, but if you have a look at the Cosmos Omnibus

269
00:20:18,240 --> 00:20:23,880
repo, there's an example in there about how to deploy two nodes with a load balancer to

270
00:20:23,880 --> 00:20:25,880
two different providers.

271
00:20:25,880 --> 00:20:27,600
There's that as well.

272
00:20:27,600 --> 00:20:28,600
Interesting.

273
00:20:28,600 --> 00:20:32,440
I was told that that wasn't something that could be done.

274
00:20:32,440 --> 00:20:34,160
You'd have to do it manually.

275
00:20:34,160 --> 00:20:36,960
I'll have to look around in there.

276
00:20:36,960 --> 00:20:37,960
Maybe.

277
00:20:37,960 --> 00:20:38,960
It was a long time ago.

278
00:20:38,960 --> 00:20:44,000
Maybe I've got my wires crossed now, but it'd be interesting to have a look anyway.

279
00:20:44,000 --> 00:20:46,000
Can you only deploy?

280
00:20:46,000 --> 00:20:55,720
Yeah, I don't know, man, because the provider is hosting the deployment, which is a thin

281
00:20:55,720 --> 00:20:59,280
wrap around Kubernetes.

282
00:20:59,280 --> 00:21:02,280
Not sure how it could...

283
00:21:02,280 --> 00:21:03,560
Maybe it's on the same provider.

284
00:21:03,560 --> 00:21:04,560
It might be my fuck up there.

285
00:21:04,560 --> 00:21:05,560
I'm not sure.

286
00:21:05,560 --> 00:21:10,240
If it was a thin provider, it'd be pretty easy to just put nginx in front of two RPCs,

287
00:21:10,240 --> 00:21:13,320
for sure, or whatever it is.

288
00:21:13,320 --> 00:21:20,440
Actually, running RPCs, the difficulty, I think, is supply side.

289
00:21:20,440 --> 00:21:23,960
I think we were talking about this privately.

290
00:21:23,960 --> 00:21:27,200
The tech works surprisingly well.

291
00:21:27,200 --> 00:21:31,280
The marketplace works not the thing, but the supply side of it is a bit more complicated.

292
00:21:31,280 --> 00:21:36,120
Make sure he says sometimes providers go away.

293
00:21:36,120 --> 00:21:40,240
I think you would want to be in a position where you had redundancy.

294
00:21:40,240 --> 00:21:46,640
The cool thing is that because of omnibus, like Tom's great work again, you can just

295
00:21:46,640 --> 00:21:51,520
bootstrap a node from nothing quite quickly.

296
00:21:51,520 --> 00:21:54,000
That is really impressive when you do that.

297
00:21:54,000 --> 00:22:00,440
That was really what's pushed me down into TMKMS, finally adopting that properly and

298
00:22:00,440 --> 00:22:05,800
all that kind of stuff, is because you can literally just spin up RPCs.

299
00:22:05,800 --> 00:22:10,320
Those could be validator nodes that are quite throwaway longer term.

300
00:22:10,320 --> 00:22:16,880
As long as you know where to point them, you're just reloading config somewhere.

301
00:22:16,880 --> 00:22:19,880
You can obviously drive that a variety of ways.

302
00:22:19,880 --> 00:22:20,880
You can script it with Ansible.

303
00:22:20,880 --> 00:22:28,080
You can do whatever on your TMKMS node, whatever.

304
00:22:28,080 --> 00:22:34,760
There is that lingering feeling that the provider might go away.

305
00:22:34,760 --> 00:22:38,120
That's the missing piece for me a little bit, is I haven't really worked out how to monitor

306
00:22:38,120 --> 00:22:44,680
it effectively and I haven't really worked out the pricing.

307
00:22:44,680 --> 00:22:49,080
Again, I think we were talking about it, getting a reliable deployment.

308
00:22:49,080 --> 00:22:55,160
You're sort of $60, $70 ballpark, I found.

309
00:22:55,160 --> 00:22:57,000
That will buy you quite a bit of bare metal.

310
00:22:57,000 --> 00:22:58,840
It will buy you quite a bit of digital ocean.

311
00:22:58,840 --> 00:23:02,720
It will even buy you quite a bit of AWS.

312
00:23:02,720 --> 00:23:07,760
That potentially is a bit of a segue into tender duty.

313
00:23:07,760 --> 00:23:14,240
Now I'm not as unfrazen because one of the things I was thinking about was obviously

314
00:23:14,240 --> 00:23:18,400
one of the ways you can monitor, you can at least have alerts when things go down is tender

315
00:23:18,400 --> 00:23:25,920
duty and since that runs, typically it's run containerized anyway, co-locating that with

316
00:23:25,920 --> 00:23:34,080
a node with its RPC port open only to other containers within the deployment is actually

317
00:23:34,080 --> 00:23:36,240
very viable.

318
00:23:36,240 --> 00:23:38,960
That's pretty cool.

319
00:23:38,960 --> 00:23:45,840
I have to confess, I'm probably the only one not using tender duty on the entire game

320
00:23:45,840 --> 00:23:46,840
of nodes crew.

321
00:23:46,840 --> 00:23:48,560
You're not using it at all?

322
00:23:48,560 --> 00:23:49,560
What are you using?

323
00:23:49,560 --> 00:23:52,360
How do you monitor your shit?

324
00:23:52,360 --> 00:23:57,880
I use Monit and Cron.

325
00:23:57,880 --> 00:23:59,800
I use Monit too, man.

326
00:23:59,800 --> 00:24:03,360
Monit is like the core of my monitoring.

327
00:24:03,360 --> 00:24:04,360
Oh, yeah?

328
00:24:04,360 --> 00:24:12,280
Yeah, I mean, I use inMonit for, I've got crazy Ansible scripts that build all these massive

329
00:24:12,280 --> 00:24:14,680
Monit configs that check everything, man.

330
00:24:14,680 --> 00:24:16,680
We're leveling on my disks.

331
00:24:16,680 --> 00:24:23,480
You know, you can do ware level monitoring on Zabix now too, plug.

332
00:24:23,480 --> 00:24:24,480
Oh, nice.

333
00:24:24,480 --> 00:24:27,920
I mean, it's a Monit, everything's crazy Ansible script.

334
00:24:27,920 --> 00:24:37,080
You just go like install that friggin smart monitoring and it just picks it up and it's

335
00:24:37,080 --> 00:24:39,320
like, oh, now I can do this.

336
00:24:39,320 --> 00:24:44,160
Right, but so my reasoning here is this, right, is that there are lots and lots of things

337
00:24:44,160 --> 00:24:45,480
that can go wrong.

338
00:24:45,480 --> 00:24:50,960
I know that if the shell fails, that box is toast, right?

339
00:24:50,960 --> 00:24:53,400
I know if Monit fails, the box is toast.

340
00:24:53,400 --> 00:24:55,760
Like those are things that are not going to fail.

341
00:24:55,760 --> 00:24:58,800
Cron is not going to fail.

342
00:24:58,800 --> 00:25:04,840
Yeah, you don't necessarily like have to have a toast box to stop signing though.

343
00:25:04,840 --> 00:25:06,600
You can have a toast signer.

344
00:25:06,600 --> 00:25:12,160
Well, sure, but like that's the point of the bash group, which is checking various things

345
00:25:12,160 --> 00:25:18,960
about the box on a short interval and then is going, okay, if any of this is wrong, then

346
00:25:18,960 --> 00:25:22,880
I'm going to immediately hit a page duty endpoint essentially.

347
00:25:22,880 --> 00:25:29,240
So it's not it's not different, significantly different to tend to duty.

348
00:25:29,240 --> 00:25:33,400
It's just a homebrew config involving Monit and a web request via Curl.

349
00:25:33,400 --> 00:25:37,080
So actually what Tinder duty does is is different.

350
00:25:37,080 --> 00:25:39,640
And that's why I wrote it at first.

351
00:25:39,640 --> 00:25:48,160
I was like, you know, there's nothing doing what it does and it needed to happen.

352
00:25:48,160 --> 00:25:56,240
So it subscribes to the blocks on a web socket and it checks for your signature.

353
00:25:56,240 --> 00:26:00,240
And that's where it, you know, that's what the main purpose of it was is to say, hey,

354
00:26:00,240 --> 00:26:04,400
if I missed three blocks, you know, I need to get up and fix something.

355
00:26:04,400 --> 00:26:05,800
I don't care about note health.

356
00:26:05,800 --> 00:26:07,560
I don't care about anything else.

357
00:26:07,560 --> 00:26:13,680
Well, now I do, but it was built for one purpose and only one thing.

358
00:26:13,680 --> 00:26:18,920
And I think the first, the first version that people started using was like 140 lines of

359
00:26:18,920 --> 00:26:20,240
code.

360
00:26:20,240 --> 00:26:28,240
So it was like this tiny little single purpose very Unix like type tool that did one thing.

361
00:26:28,240 --> 00:26:32,960
So and then it's grown quite a bit.

362
00:26:32,960 --> 00:26:34,200
And now it's V2.

363
00:26:34,200 --> 00:26:36,240
It does a lot of stuff.

364
00:26:36,240 --> 00:26:41,440
So and this is stuff people have been asking for and stuff I've been wanting to do as well,

365
00:26:41,440 --> 00:26:49,400
you know, so now it's kind of a full cosmos monitoring tool that goes beyond just, hey,

366
00:26:49,400 --> 00:26:52,440
did I miss the blocks kind of thing?

367
00:26:52,440 --> 00:26:58,880
So yeah, I know that like, all of us thought it was awesome and all of us use it, but all

368
00:26:58,880 --> 00:27:03,040
of us are always like, Hey, man, have you thought about adding this thing?

369
00:27:03,040 --> 00:27:06,480
Like, I'll put it on the list.

370
00:27:06,480 --> 00:27:10,640
And like, Hey, man, have you thought about like making it like easier to look at?

371
00:27:10,640 --> 00:27:12,200
I'll put that on the list.

372
00:27:12,200 --> 00:27:16,160
Hey, man, like, can you come on to this other thing too?

373
00:27:16,160 --> 00:27:17,160
Okay.

374
00:27:17,160 --> 00:27:18,160
Yeah.

375
00:27:18,160 --> 00:27:27,040
So now, you know, it's got a dashboard, a Prometheus exporter, which is still in the works, but

376
00:27:27,040 --> 00:27:31,880
you know, like, Hey, if your node goes down, how long has it been down kind of thing?

377
00:27:31,880 --> 00:27:37,360
Or, you know, how many times did I miss a block where I actually sit in a pre-commit

378
00:27:37,360 --> 00:27:38,960
and the proposer didn't include it?

379
00:27:38,960 --> 00:27:45,360
And that's a big question for me a lot of times if I'm not performing well on a chain,

380
00:27:45,360 --> 00:27:48,640
it's like, is this actually my fault?

381
00:27:48,640 --> 00:27:55,680
Or is this, you know, somebody else out there set to sign blocks at three seconds, you know,

382
00:27:55,680 --> 00:27:57,280
just making me drop.

383
00:27:57,280 --> 00:28:03,800
So just to explain that for people who are listening, what's because this is the thing

384
00:28:03,800 --> 00:28:05,600
we've seen on several networks, right?

385
00:28:05,600 --> 00:28:07,440
The three second signing.

386
00:28:07,440 --> 00:28:11,520
What's the reason that that's causing you to miss blocks?

387
00:28:11,520 --> 00:28:18,800
Because the proposer is setting a limited time before that they will propose a block

388
00:28:18,800 --> 00:28:20,520
at 67%.

389
00:28:20,520 --> 00:28:30,120
So normally if you wait, I think it's five seconds after your first pre-commit or something

390
00:28:30,120 --> 00:28:34,240
like that until you actually propose the block.

391
00:28:34,240 --> 00:28:39,040
And if you crank those numbers down, you can reduce block time on a chain, but you're also

392
00:28:39,040 --> 00:28:41,960
going to have a lot of validators missing blocks.

393
00:28:41,960 --> 00:28:48,040
I think this might actually be a suspect that this is causing a problem on secret network

394
00:28:48,040 --> 00:28:56,320
right now because their average block time is like 5.2 seconds, which is a hair faster

395
00:28:56,320 --> 00:28:58,080
than I'd expect.

396
00:28:58,080 --> 00:29:02,600
So somebody out there speeding up the block.

397
00:29:02,600 --> 00:29:09,120
So someone with a lot of like voting weight to a lot of blocks.

398
00:29:09,120 --> 00:29:15,200
I haven't dug into it, but at some point I will and probably reach out to them individually

399
00:29:15,200 --> 00:29:16,200
and say, Hey,

400
00:29:16,200 --> 00:29:21,200
Yeah, we've seen that on some

401
00:29:21,200 --> 00:29:30,480
Todd, I think I lost your audio.

402
00:29:30,480 --> 00:29:36,080
Did you guys just see me or battery went flat?

403
00:29:36,080 --> 00:29:42,760
No, we can't hear you, man.

404
00:29:42,760 --> 00:29:45,680
So I mean, I love tender duty.

405
00:29:45,680 --> 00:29:47,160
It's like so light.

406
00:29:47,160 --> 00:29:54,800
So one thing I want to know is like when Todd gets back on here is so there's a new configuration.

407
00:29:54,800 --> 00:29:57,240
Well, he's completely rewritten it, right?

408
00:29:57,240 --> 00:30:02,200
So now it's like, I think it's even a different language now, or maybe it's still go, but

409
00:30:02,200 --> 00:30:03,560
he's added in extra features.

410
00:30:03,560 --> 00:30:08,600
And what I'm going to want to know is like, can you turn off the Prometheus exporter, etc.

411
00:30:08,600 --> 00:30:11,120
to try and keep it as light as possible?

412
00:30:11,120 --> 00:30:18,240
So I use ZabEx, I'll probably like query it in different ways with ZabEx through the

413
00:30:18,240 --> 00:30:24,520
different ways you can actually send commands through ZabEx to your server.

414
00:30:24,520 --> 00:30:28,720
So you know, things like the Prometheus exporter, I won't want.

415
00:30:28,720 --> 00:30:33,880
But also like I'm wondering if it's multiple network configuration in the one file because

416
00:30:33,880 --> 00:30:40,160
you used to have to launch multiple instances and configure all your networks separately.

417
00:30:40,160 --> 00:30:44,880
And up until recently, I thought that meant that you had to bring them all down when you

418
00:30:44,880 --> 00:30:48,040
wanted to change the configuration and then bring them all back up.

419
00:30:48,040 --> 00:30:52,560
But apparently with doccontainment, I just figured this out the other day, you can actually

420
00:30:52,560 --> 00:30:58,560
just update your configuration file and then re-up them into daemons and it'll just update

421
00:30:58,560 --> 00:31:04,320
them and you can use a flag that'll drop all the dead containers and delete them.

422
00:31:04,320 --> 00:31:09,720
So, daemon Hill, race car driver.

423
00:31:09,720 --> 00:31:12,400
Did you guys hear me now?

424
00:31:12,400 --> 00:31:14,160
Yeah, man, you're back now.

425
00:31:14,160 --> 00:31:17,360
I've just had so much technical trouble today.

426
00:31:17,360 --> 00:31:21,080
Sorry, I just took the liberty to ramble some shit while you weren't here.

427
00:31:21,080 --> 00:31:22,080
Oh, no, no, totally.

428
00:31:22,080 --> 00:31:25,600
And actually all this pronounced daemon really weird too.

429
00:31:25,600 --> 00:31:30,480
Incredibly relevant because...

430
00:31:30,480 --> 00:31:31,480
Because...

431
00:31:31,480 --> 00:31:32,480
Oh boy.

432
00:31:32,480 --> 00:31:36,800
Apparently, I can't say daemon or daemon.

433
00:31:36,800 --> 00:31:38,440
I always call it daemon.

434
00:31:38,440 --> 00:31:40,640
So, daemon is a name.

435
00:31:40,640 --> 00:31:50,320
It's just an old English name and daemon is like a malicious spirit.

436
00:31:50,320 --> 00:31:58,120
So you guys are going to laugh because Chrome didn't have access to record my screen and

437
00:31:58,120 --> 00:31:59,920
it wants me to restart.

438
00:31:59,920 --> 00:32:01,920
Well, five minutes.

439
00:32:01,920 --> 00:32:05,800
That's what I'm going to do, man.

440
00:32:05,800 --> 00:32:11,160
I'm just going to explain what's going on with it and if you're interested in checking

441
00:32:11,160 --> 00:32:16,760
it out, I've got a...

442
00:32:16,760 --> 00:32:21,600
Do you have a working page somewhere because Nol can probably throw it up on the screen?

443
00:32:21,600 --> 00:32:22,600
Yeah, actually.

444
00:32:22,600 --> 00:32:23,600
You know what?

445
00:32:23,600 --> 00:32:25,280
I've got to publish my notes.

446
00:32:25,280 --> 00:32:31,520
Share the link in the private chat and then Nol can throw it up.

447
00:32:31,520 --> 00:32:46,960
So anyways, you were saying that one of the pain points was one chain per config, right?

448
00:32:46,960 --> 00:32:50,040
And so now that's...

449
00:32:50,040 --> 00:32:54,040
You can do multiple chains on one running instance.

450
00:32:54,040 --> 00:33:02,640
The dashboard is very much inspired by...

451
00:33:02,640 --> 00:33:11,640
Oh, boy.

452
00:33:11,640 --> 00:33:17,640
Batteries.

453
00:33:17,640 --> 00:33:27,640
Welcome to summer.

454
00:33:27,640 --> 00:33:37,240
Anyways, the display on there is very much inspired by what's going on on ping.pub except

455
00:33:37,240 --> 00:33:41,600
it is quite wide and you can actually scroll over.

456
00:33:41,600 --> 00:33:50,560
I think I've got...I want to say I share...keep the last 512 blocks so you can see where you

457
00:33:50,560 --> 00:33:57,720
missed, where you signed, the different colors.

458
00:33:57,720 --> 00:34:01,480
When you propose isn't a big deal, but I thought it was cool to be able to see where and when

459
00:34:01,480 --> 00:34:05,640
I proposed to see how often it was happening.

460
00:34:05,640 --> 00:34:11,880
My dashboard is incredibly boring because there's nothing fun happening here.

461
00:34:11,880 --> 00:34:18,080
A lot of my testing I've been doing with other validators.

462
00:34:18,080 --> 00:34:19,080
They've been very helpful.

463
00:34:19,080 --> 00:34:20,600
They don't know it.

464
00:34:20,600 --> 00:34:21,600
But...

465
00:34:21,600 --> 00:34:25,720
That's a low-key flex if I've ever heard one.

466
00:34:25,720 --> 00:34:27,720
What's that?

467
00:34:27,720 --> 00:34:31,120
That's a low-key flex if I've ever heard one.

468
00:34:31,120 --> 00:34:33,880
My network is boring because I missed so few blocks.

469
00:34:33,880 --> 00:34:38,800
I did miss a block on Terri Classic there, it looks like.

470
00:34:38,800 --> 00:34:43,040
That's great.

471
00:34:43,040 --> 00:34:50,400
The dashboard is something that I wanted just because I never really had that information.

472
00:34:50,400 --> 00:34:57,920
When something fails, you get a nice little orange alert message and you can click on it

473
00:34:57,920 --> 00:35:00,280
and see the details.

474
00:35:00,280 --> 00:35:07,800
This is set up for public use.

475
00:35:07,800 --> 00:35:16,880
Normally if it would have a log stream below showing, hey, this RPC node is not responding

476
00:35:16,880 --> 00:35:24,600
or something like that.

477
00:35:24,600 --> 00:35:26,680
The dashboard is a dashboard.

478
00:35:26,680 --> 00:35:28,760
There's not much to it.

479
00:35:28,760 --> 00:35:30,360
Just...

480
00:35:30,360 --> 00:35:35,720
Then, like I mentioned, there's a pre-MTSN point.

481
00:35:35,720 --> 00:35:41,480
If my screen share were working, I'd show you the beginnings of a Grafana dashboard,

482
00:35:41,480 --> 00:35:45,920
but I can't.

483
00:35:45,920 --> 00:35:47,880
Each thing is configurable now.

484
00:35:47,880 --> 00:35:57,960
You can alert to Discord, Telegram, or page or duty.

485
00:35:57,960 --> 00:36:01,360
It's configurable per chain.

486
00:36:01,360 --> 00:36:06,400
You can have different alert destinations per each chain.

487
00:36:06,400 --> 00:36:08,600
Like I said, this is done.

488
00:36:08,600 --> 00:36:09,600
This is essentially done.

489
00:36:09,600 --> 00:36:14,400
I'm working on docs and then I'm going to tag it as a beta probably in an hour or so

490
00:36:14,400 --> 00:36:18,480
of ending this podcast.

491
00:36:18,480 --> 00:36:19,480
You can do...

492
00:36:19,480 --> 00:36:27,480
I didn't realize that it had the notification stuff because I was looking at what's it called,

493
00:36:27,480 --> 00:36:35,920
Half-Life, which I know some of the other GamerNodes folks use for Discord notifications.

494
00:36:35,920 --> 00:36:41,320
I hadn't realized because I think, like I said about the Akash stuff has started to motivate

495
00:36:41,320 --> 00:36:44,120
me to move to...

496
00:36:44,120 --> 00:36:49,520
Even on our servers, probably dockerizing and actually deploying everything using our

497
00:36:49,520 --> 00:36:53,160
same mount strategy, but starting to use sort of dockerized containers.

498
00:36:53,160 --> 00:37:00,920
Tendermint has been on my list for a while to work out what the gap is between what we're

499
00:37:00,920 --> 00:37:03,600
doing and how good it could be.

500
00:37:03,600 --> 00:37:07,800
I think that might be the killer feature for us there.

501
00:37:07,800 --> 00:37:09,000
How does that work?

502
00:37:09,000 --> 00:37:14,280
Is it just possible to basically configure and send push notifications then to additional

503
00:37:14,280 --> 00:37:15,280
sources?

504
00:37:15,280 --> 00:37:16,280
Yeah.

505
00:37:16,280 --> 00:37:22,240
I mean, if you want to alert to Discord, you just set up a web hook on a channel and

506
00:37:22,240 --> 00:37:23,240
it can post in.

507
00:37:23,240 --> 00:37:31,320
Then telegram is kind of a pain to find the group ID for the chat, but you can get an

508
00:37:31,320 --> 00:37:38,760
API key and then invite the bot into a chat room and it can send stuff.

509
00:37:38,760 --> 00:37:42,280
PagerDuty, obviously, API key.

510
00:37:42,280 --> 00:37:47,880
The one benefit to using PagerDuty is you can run multiple instances and it'll deduplicate

511
00:37:47,880 --> 00:37:54,200
alerts based on a key that I set when I set an alarm.

512
00:37:54,200 --> 00:37:58,240
You can have redundant, tender duties.

513
00:37:58,240 --> 00:38:00,120
Redundancy and your redundancy.

514
00:38:00,120 --> 00:38:05,200
You could have a tender duty on the node and you have a tender duty somewhere else pointing

515
00:38:05,200 --> 00:38:09,200
an RPC and then it would deduplicate them.

516
00:38:09,200 --> 00:38:12,680
Yeah, because it uses the same key.

517
00:38:12,680 --> 00:38:20,200
It updates the same message, right?

518
00:38:20,200 --> 00:38:22,200
Oh, my God.

519
00:38:22,200 --> 00:38:24,520
Did it my mic cut out again?

520
00:38:24,520 --> 00:38:28,480
Yeah, just like an SID you're muted.

521
00:38:28,480 --> 00:38:29,480
Crazy.

522
00:38:29,480 --> 00:38:35,520
Wow, this new Macbook isn't entirely stable.

523
00:38:35,520 --> 00:38:38,520
Yeah, man.

524
00:38:38,520 --> 00:38:45,600
So it dedupts based on a key that you set and depending on the type of alert it's sending,

525
00:38:45,600 --> 00:38:54,800
I set different keys, but I think I use the Valkons address, the consensus public key

526
00:38:54,800 --> 00:39:01,000
address as the key for mistblock alerts.

527
00:39:01,000 --> 00:39:10,560
So pager duty won't resend an alert with the same key, but you can resolve an alert automatically

528
00:39:10,560 --> 00:39:13,000
with the same key.

529
00:39:13,000 --> 00:39:19,000
So it's nothing special I'm doing, just something they do for us already.

530
00:39:19,000 --> 00:39:25,480
But yeah, see what else is new about it?

531
00:39:25,480 --> 00:39:29,320
How long did it take you to develop the first tender duty?

532
00:39:29,320 --> 00:39:31,520
About four hours.

533
00:39:31,520 --> 00:39:34,200
What a machine.

534
00:39:34,200 --> 00:39:36,480
Well, I mean, it's not hard.

535
00:39:36,480 --> 00:39:43,000
You just subscribe to a web socket and look for a key in the resulting JSON.

536
00:39:43,000 --> 00:39:45,080
So that was super easy.

537
00:39:45,080 --> 00:39:52,720
It's when you start adding edge cases and configurability and you break something and

538
00:39:52,720 --> 00:39:55,960
then you're like, oh, well, I got to fix that.

539
00:39:55,960 --> 00:39:58,760
So yeah.

540
00:39:58,760 --> 00:40:02,800
So with those four hours, you saved probably thousands of man hours.

541
00:40:02,800 --> 00:40:03,800
That's pretty cool.

542
00:40:03,800 --> 00:40:04,800
Yeah.

543
00:40:04,800 --> 00:40:15,240
But yeah, since then I've put weeks of work into it, but especially now I'm about seven

544
00:40:15,240 --> 00:40:19,200
eight days into this effort.

545
00:40:19,200 --> 00:40:21,440
So that's a lot of work.

546
00:40:21,440 --> 00:40:24,840
That's a significant amount of time.

547
00:40:24,840 --> 00:40:30,400
As an aside, as somebody who comes from primarily a high level programming background, I'm in

548
00:40:30,400 --> 00:40:36,160
the grand scheme of things, relatively new to Rust and I guess I've read a bit of Go

549
00:40:36,160 --> 00:40:37,360
here and there in the past.

550
00:40:37,360 --> 00:40:43,800
But this is one of those things I think that people maybe from, like I say, me who have

551
00:40:43,800 --> 00:40:48,520
moved from more of a high level programming background to those of you that aren't programmers,

552
00:40:48,520 --> 00:40:52,760
it means closer high level, generally high level of abstraction.

553
00:40:52,760 --> 00:40:59,760
So I think that's why, where assembly is the lowest level of not abstraction at all, really,

554
00:40:59,760 --> 00:41:04,440
up to literally like whatever you see in your web browser, I guess is the way of explaining

555
00:41:04,440 --> 00:41:05,440
it.

556
00:41:05,440 --> 00:41:10,760
But so the cool thing about the first version, which I think I remember seeing at the time,

557
00:41:10,760 --> 00:41:13,800
because this is I think why I end up just being like, I don't need to listen for the

558
00:41:13,800 --> 00:41:14,880
web socket.

559
00:41:14,880 --> 00:41:18,080
So I'm just going to like, not to go to a bash script in five minutes.

560
00:41:18,080 --> 00:41:23,680
But I think I remember seeing that first version and it is like essentially, yeah, the stuff

561
00:41:23,680 --> 00:41:28,120
you can't do in a bash script because it's a web socket subscription goes got an API

562
00:41:28,120 --> 00:41:29,120
for that.

563
00:41:29,120 --> 00:41:32,880
It's like a few calls and then a bit of JSON parsing and then boom.

564
00:41:32,880 --> 00:41:36,880
But the really cool thing is for somebody from like a high level program background,

565
00:41:36,880 --> 00:41:41,520
you look at it and you go like, ah, this is essentially a small script, a Go file that

566
00:41:41,520 --> 00:41:45,960
is essentially being treated like a scripting language, and it's going to emit a binary,

567
00:41:45,960 --> 00:41:49,880
which then you can pretty much run everywhere.

568
00:41:49,880 --> 00:41:52,880
And in addition, you also supplied a Docker file with it.

569
00:41:52,880 --> 00:41:55,760
So you literally can run it everywhere with Docker.

570
00:41:55,760 --> 00:41:59,560
But it's one of those things that like, I guess I hadn't really thought about until I

571
00:41:59,560 --> 00:42:05,440
saw I think it was your script for tender duty and something I went back to a little

572
00:42:05,440 --> 00:42:10,640
while ago because somebody from Terra was asking how to listen for a particular type

573
00:42:10,640 --> 00:42:12,720
of emitted Cosmosm event.

574
00:42:12,720 --> 00:42:18,840
And I remember that Jan's Alex, who has done quite a bit of core work on Juno, had written

575
00:42:18,840 --> 00:42:25,160
like a tiny, yeah, again, like a 30 line Go file that you can obviously compile to a

576
00:42:25,160 --> 00:42:27,960
binary and be like, here you are, listen for whatever you want to listen for.

577
00:42:27,960 --> 00:42:28,960
Yeah.

578
00:42:28,960 --> 00:42:30,560
Yeah, actually, you know, it's insane.

579
00:42:30,560 --> 00:42:36,880
For Tinder, men is crazy easy because you just call into their RPC library.

580
00:42:36,880 --> 00:42:39,400
And so it's like, hey, I want to get this.

581
00:42:39,400 --> 00:42:41,360
It's like four lines of code.

582
00:42:41,360 --> 00:42:47,400
You know, so the API is well defined there.

583
00:42:47,400 --> 00:42:56,440
The annoying part though is where you end up with different like some stuffs protobufs,

584
00:42:56,440 --> 00:43:00,320
some stuff is JSON.

585
00:43:00,320 --> 00:43:03,200
If it's JSON, everything's a string.

586
00:43:03,200 --> 00:43:07,680
So now you've got, if you're in a strongly typed language, that's a problem.

587
00:43:07,680 --> 00:43:12,480
You know, now you've got to deal with reflection and casting and all that fun stuff.

588
00:43:12,480 --> 00:43:15,240
So you know, RPC isn't great.

589
00:43:15,240 --> 00:43:19,800
GRPC client is awesome.

590
00:43:19,800 --> 00:43:26,640
I mean, you can do everything, but it's that requires a little bit more knowledge to program

591
00:43:26,640 --> 00:43:27,640
against.

592
00:43:27,640 --> 00:43:31,800
So again, I'll expose my ignorance to this.

593
00:43:31,800 --> 00:43:34,600
What's the difference between RPC and GRPC in this case?

594
00:43:34,600 --> 00:43:40,040
RPC is a web protocol, returns JSON.

595
00:43:40,040 --> 00:43:44,480
GRPC, it's a web two protocol.

596
00:43:44,480 --> 00:43:50,240
I mean, HTTP protocol, but it all relies on protobuf, which is a binary.

597
00:43:50,240 --> 00:43:51,240
Yeah.

598
00:43:51,240 --> 00:43:58,320
It's the binary format at the core of, you know, ABCI, the blockchain interface databases.

599
00:43:58,320 --> 00:43:59,320
Yeah.

600
00:43:59,320 --> 00:44:00,320
Okay.

601
00:44:00,320 --> 00:44:08,400
Obviously protobuf is quite deeply embedded in Cosmos and like former life doing data,

602
00:44:08,400 --> 00:44:09,400
data infrastructure.

603
00:44:09,400 --> 00:44:13,600
Protobuf is one of the things, is one of those things like Avro, I guess like Avro you see

604
00:44:13,600 --> 00:44:17,960
as well, but like protobuf is pretty common as a serialization mechanism.

605
00:44:17,960 --> 00:44:18,960
Yeah.

606
00:44:18,960 --> 00:44:24,200
I think one of the big differences is that Avro, generally you can embed the definitions

607
00:44:24,200 --> 00:44:26,760
with the message in protobuf.

608
00:44:26,760 --> 00:44:27,760
You don't do that.

609
00:44:27,760 --> 00:44:31,440
Yeah, I think in my, I think the most patient for, sorry.

610
00:44:31,440 --> 00:44:32,440
No.

611
00:44:32,440 --> 00:44:36,960
Can one of you guys explain what protobuf is so that people who are listening don't

612
00:44:36,960 --> 00:44:40,880
know exactly what you're talking about?

613
00:44:40,880 --> 00:44:43,440
I don't know.

614
00:44:43,440 --> 00:44:46,280
So like JSON is like structured data, right?

615
00:44:46,280 --> 00:44:47,280
Yeah.

616
00:44:47,280 --> 00:44:48,280
Yeah.

617
00:44:48,280 --> 00:44:50,000
So what is protobuf?

618
00:44:50,000 --> 00:44:57,040
Protobuf is an efficient data messaging protocol.

619
00:44:57,040 --> 00:45:06,640
You define what type everything is and then it builds, I don't know, I don't know if you

620
00:45:06,640 --> 00:45:15,080
guys are familiar with like ABI's from Ethereum or EOSIO world where it's, you've got an ABI

621
00:45:15,080 --> 00:45:18,320
and everything's binary and you pack it in.

622
00:45:18,320 --> 00:45:20,560
That's basically what protobuf does.

623
00:45:20,560 --> 00:45:26,760
It's a way to see ABI's, but it takes it another step where you've got all these

624
00:45:26,760 --> 00:45:34,080
binary definitions and ways of compressing stuff and then it adds the ability to build

625
00:45:34,080 --> 00:45:36,040
ABI's around it automatically.

626
00:45:36,040 --> 00:45:42,400
So when you have a type, you can say, hey, this is how you set it.

627
00:45:42,400 --> 00:45:49,560
This is how you get it with just a regular function and then it'll build out the GRPC

628
00:45:49,560 --> 00:45:55,160
APIs and then even on TenderMint, the Rust APIs.

629
00:45:55,160 --> 00:46:01,920
And is that type like basically it makes a smaller message like less data transfer and

630
00:46:01,920 --> 00:46:03,360
stuff like that as well quicker?

631
00:46:03,360 --> 00:46:06,520
It's just more overhead on both sides.

632
00:46:06,520 --> 00:46:07,520
Yeah.

633
00:46:07,520 --> 00:46:11,720
For like a string, you get a null byte and then the length of the string.

634
00:46:11,720 --> 00:46:18,960
And then if a field is mandatory, you just get null byte between each one.

635
00:46:18,960 --> 00:46:20,200
Yeah.

636
00:46:20,200 --> 00:46:22,840
So it's super efficient that way.

637
00:46:22,840 --> 00:46:23,840
Yeah.

638
00:46:23,840 --> 00:46:30,000
While we're talking about these interfaces, what's an LCD now?

639
00:46:30,000 --> 00:46:31,560
I call it Rust.

640
00:46:31,560 --> 00:46:32,800
Yeah.

641
00:46:32,800 --> 00:46:36,960
In the documentation, it's called Rust.

642
00:46:36,960 --> 00:46:39,960
But I have no fucking clue what LCD means in this context.

643
00:46:39,960 --> 00:46:42,640
Like clientDemon.

644
00:46:42,640 --> 00:46:49,520
It's from before when the clientDemon was split out from the TenderMintDemon, right?

645
00:46:49,520 --> 00:46:58,040
So it would talk to TenderMint over, I believe, a GRPC socket and then it was like, well,

646
00:46:58,040 --> 00:47:01,800
why do we have this kind of thing?

647
00:47:01,800 --> 00:47:06,840
So it's just basically another layer to make it easier for people to talk to it?

648
00:47:06,840 --> 00:47:07,840
Yeah.

649
00:47:07,840 --> 00:47:10,440
If you understand Rust and not GRPC.

650
00:47:10,440 --> 00:47:16,080
Well, it exposes everything in GRPC, almost everything.

651
00:47:16,080 --> 00:47:20,960
But you could alternatively just talk to GRPC instead.

652
00:47:20,960 --> 00:47:22,760
If you know how.

653
00:47:22,760 --> 00:47:26,480
Yes, that's the big question.

654
00:47:26,480 --> 00:47:31,000
It is something that takes a little bit of time to get your head around, I think.

655
00:47:31,000 --> 00:47:35,880
But it's so core if you're going to do any kind of dev on Cosmos, I believe, having

656
00:47:35,880 --> 00:47:41,440
a good understanding of how it uses protobuf is pretty key.

657
00:47:41,440 --> 00:47:42,440
Yeah.

658
00:47:42,440 --> 00:47:48,360
This is kind of insane thing that happens if you actually define anything new in protobuf

659
00:47:48,360 --> 00:47:50,600
and then just kind of like you're able to run it.

660
00:47:50,600 --> 00:47:55,120
It just generates all this like loads of stuff and you're like, whoa.

661
00:47:55,120 --> 00:47:57,120
So yeah.

662
00:47:57,120 --> 00:48:04,040
It's, I'm still not 100% again, not 100% used to how that works in Cosmos, but it does give

663
00:48:04,040 --> 00:48:09,600
you, it does make you look like you were typing very, very quickly when you do get to do that.

664
00:48:09,600 --> 00:48:13,120
So that's not, I know it's not even my jokes.

665
00:48:13,120 --> 00:48:16,520
It's something Jaby said because he's a funny man and I am.

666
00:48:16,520 --> 00:48:18,400
He was like, look how fast I'm typing.

667
00:48:18,400 --> 00:48:23,200
And it was just like generating all this protobuf stuff.

668
00:48:23,200 --> 00:48:30,880
But I guess that the, it'd be quite interesting.

669
00:48:30,880 --> 00:48:34,720
I have never really thought about how it must work under the hood for the arbitrary types

670
00:48:34,720 --> 00:48:38,560
for, well, I know a little bit of how it works for the arbitrary types that you generate

671
00:48:38,560 --> 00:48:45,240
from smart contracts, but obviously they're then exposed via a underlying protobuf message

672
00:48:45,240 --> 00:48:53,680
in WASMD, which is then available to the client, but you essentially do an extra layer of misdirection,

673
00:48:53,680 --> 00:48:58,680
don't you, at the smart contract level because you're doing that execute contract and then

674
00:48:58,680 --> 00:49:00,160
passing along the stuff.

675
00:49:00,160 --> 00:49:06,640
And I think this was actually a sticking point we're talking about orth Z for smart contracts

676
00:49:06,640 --> 00:49:15,800
because technically, technically, if I remember correctly, the every smart contract that I've

677
00:49:15,800 --> 00:49:25,240
ever seen that runs in Cosmos on Cosmwazn expects a top level JSON object as its message

678
00:49:25,240 --> 00:49:27,400
because then you can unpack the types in Rust.

679
00:49:27,400 --> 00:49:33,280
You were using the, what's it called the SERDI library.

680
00:49:33,280 --> 00:49:38,960
In this case, we're for those listening, SERDI means serialization, deserialization.

681
00:49:38,960 --> 00:49:45,960
So think code, something happening through code into JSON and then back out of JSON,

682
00:49:45,960 --> 00:49:49,240
something like that SERDI.

683
00:49:49,240 --> 00:49:53,400
But the, yeah, apparently that actually the underlying code makes no assumptions about

684
00:49:53,400 --> 00:49:55,160
the shape.

685
00:49:55,160 --> 00:49:58,920
So even though in practice, it would be pretty hard to work with anything that didn't say,

686
00:49:58,920 --> 00:50:03,440
oh, by the way, I'm a message of this type arriving at the smart contract.

687
00:50:03,440 --> 00:50:08,000
Apparently, that's not actually an assumption that holds at WASM D level and it was causing

688
00:50:08,000 --> 00:50:09,000
some trouble.

689
00:50:09,000 --> 00:50:16,720
Yeah, you would have to like wrap the original message in JSON inside the message.

690
00:50:16,720 --> 00:50:20,480
So it's all fun and games.

691
00:50:20,480 --> 00:50:31,320
Anyways, a huge thank you to Cosmos's grants people and gave me a grant to do this rewrite

692
00:50:31,320 --> 00:50:39,480
and it's something I wanted to do for a while and it helps a lot of people.

693
00:50:39,480 --> 00:50:49,640
So I hope people find it useful and I don't have anything else to say about it.

694
00:50:49,640 --> 00:50:51,960
How was the grant process?

695
00:50:51,960 --> 00:50:56,520
Did you have to go through like interview panels or did you write up, you know, how's

696
00:50:56,520 --> 00:50:57,520
it going?

697
00:50:57,520 --> 00:51:03,080
Yeah, I did an interview, video call the interview and put together a proposal that I submitted.

698
00:51:03,080 --> 00:51:05,240
I went through a couple rounds of review.

699
00:51:05,240 --> 00:51:13,360
I think it's easier because I said, hey, I don't want anything up front.

700
00:51:13,360 --> 00:51:17,960
If I don't deliver, you don't pay me and if you're not happy, then okay, it was worth

701
00:51:17,960 --> 00:51:20,200
the chance.

702
00:51:20,200 --> 00:51:23,520
So I don't know what the final review process looks like.

703
00:51:23,520 --> 00:51:30,640
You know, I'm not going to call them up until I'm done, but I'm close.

704
00:51:30,640 --> 00:51:37,040
I mean, yeah, I think that of all the things that we've seen recently that are kind of

705
00:51:37,040 --> 00:51:41,080
up for grants and things, tender duty is one of the most actually pragmatically useful

706
00:51:41,080 --> 00:51:42,080
things.

707
00:51:42,080 --> 00:51:44,440
So I'm sure it's not going to be a problem when it comes to it.

708
00:51:44,440 --> 00:51:50,480
I just noticed we've had a question in the chat from Adam Burke says, I'll put it up

709
00:51:50,480 --> 00:51:54,840
on the screen, but I'll read it out for those listening on podcast players.

710
00:51:54,840 --> 00:51:56,880
Super baby technical question.

711
00:51:56,880 --> 00:52:01,160
If I wanted to get information about specific chain value data ranking, block time, any

712
00:52:01,160 --> 00:52:04,680
kind of data available on a block explorer, how would I do that?

713
00:52:04,680 --> 00:52:10,680
The short version is via RPC and the long version.

714
00:52:10,680 --> 00:52:18,160
Yeah, I mean, there's an API for that.

715
00:52:18,160 --> 00:52:20,880
Could be a new catchphrase.

716
00:52:20,880 --> 00:52:24,400
Yeah, API for that.

717
00:52:24,400 --> 00:52:26,400
Yeah, PC.

718
00:52:26,400 --> 00:52:34,840
It really depends on what you're trying to get, but yeah, the command line makes it

719
00:52:34,840 --> 00:52:35,840
pretty easy.

720
00:52:35,840 --> 00:52:37,760
I cheat a lot.

721
00:52:37,760 --> 00:52:52,000
I will use the command line tool with a sniffer and catch what requested is actually sending

722
00:52:52,000 --> 00:52:57,320
because that's so much faster than trying to read the code and figure out what the queries

723
00:52:57,320 --> 00:52:58,320
are.

724
00:52:58,320 --> 00:53:02,520
And so that's usually my go to trick.

725
00:53:02,520 --> 00:53:07,720
I tend to use Cosm.js and do the same thing, like just like grep through the code for Cosm.js

726
00:53:07,720 --> 00:53:15,400
on GitHub to find a function signature that looks like the thing that I'm trying to accomplish

727
00:53:15,400 --> 00:53:19,720
and then just fire it off from the console in the browser and just see what it looks

728
00:53:19,720 --> 00:53:21,240
like in the network inspector.

729
00:53:21,240 --> 00:53:27,720
And obviously, part of it is encoded always, but you kind of tend to get an idea.

730
00:53:27,720 --> 00:53:28,720
You're like, oh, right.

731
00:53:28,720 --> 00:53:30,840
I kind of see it's doing this thing.

732
00:53:30,840 --> 00:53:31,840
Okay, cool.

733
00:53:31,840 --> 00:53:37,440
Yeah, you know, and then the types can get weird depending on if you're using REST or

734
00:53:37,440 --> 00:53:44,920
RPC, but I have found a trick with most endpoints.

735
00:53:44,920 --> 00:53:51,000
You can do a raw A, B, C, I query over RPC and then it will give you protobuf back.

736
00:53:51,000 --> 00:53:56,200
And then you can just use the native types from, you know, whatever X modules you're

737
00:53:56,200 --> 00:54:00,200
working against directly out of the RPC.

738
00:54:00,200 --> 00:54:06,360
And it's like three steps instead of trying to figure out how to handle all this JSON

739
00:54:06,360 --> 00:54:08,680
and reflection and stuff.

740
00:54:08,680 --> 00:54:12,520
So that's a pretty cool trick too.

741
00:54:12,520 --> 00:54:15,040
So for the dashboard stuff, are you using it?

742
00:54:15,040 --> 00:54:16,840
Presumably it's all JavaScript, right?

743
00:54:16,840 --> 00:54:19,720
Or TypeScript, maybe.

744
00:54:19,720 --> 00:54:21,320
It's almost all go.

745
00:54:21,320 --> 00:54:27,240
The dashboard is a little bit of JavaScript.

746
00:54:27,240 --> 00:54:33,120
You know, the little grid is all written in Canvas.

747
00:54:33,120 --> 00:54:38,400
So that's kind of nasty, but it's fast and it's light.

748
00:54:38,400 --> 00:54:40,400
And, you know...

749
00:54:40,400 --> 00:54:48,240
So Canvas is, for people who don't know, Canvas is like a library, right?

750
00:54:48,240 --> 00:54:51,080
Canvas is a part of the HTML5 spec.

751
00:54:51,080 --> 00:54:54,360
It's you're just drawing with coordinates.

752
00:54:54,360 --> 00:55:01,000
So let's say, you know, make a rectangle from X, Y, you know, and then draw a line from

753
00:55:01,000 --> 00:55:04,240
here, you know, set my color.

754
00:55:04,240 --> 00:55:07,160
So is that dynamic as well?

755
00:55:07,160 --> 00:55:08,160
Or like...

756
00:55:08,160 --> 00:55:12,840
So when you say that, is that setting out the grid in the page like that?

757
00:55:12,840 --> 00:55:15,520
Or is that using the individual elements like that?

758
00:55:15,520 --> 00:55:19,440
Because, you know, for setting out your page, I thought you would have used something like

759
00:55:19,440 --> 00:55:21,800
CSS Grids or something like that?

760
00:55:21,800 --> 00:55:27,080
Yeah, you know, a lot of people have built stuff using like tables and grids and stuff.

761
00:55:27,080 --> 00:55:33,200
This is...it repaints the whole image every time it refreshes though.

762
00:55:33,200 --> 00:55:38,320
So it's big loop over an array of...

763
00:55:38,320 --> 00:55:44,040
A map of arrays, so basically.

764
00:55:44,040 --> 00:55:48,600
So would that be considered light or heavy in terms of resource?

765
00:55:48,600 --> 00:55:49,680
It's super light.

766
00:55:49,680 --> 00:55:51,480
It sips resources.

767
00:55:51,480 --> 00:55:55,320
It's a lot faster than I'm including, you know, a giant graphing library.

768
00:55:55,320 --> 00:55:58,480
You know, I have a few that I really like.

769
00:55:58,480 --> 00:56:03,080
I just didn't find any that would do this kind of grid thing.

770
00:56:03,080 --> 00:56:08,720
So this was an experiment that worked.

771
00:56:08,720 --> 00:56:17,160
So theoretically, like, you could create a similar dashboard to like what Krozeness has

772
00:56:17,160 --> 00:56:19,520
and Bro and Bro.

773
00:56:19,520 --> 00:56:24,360
That's a lot lighter and quicker to load using sort of the process that you've already used

774
00:56:24,360 --> 00:56:28,560
for TenderGD2 and building that web interface, right?

775
00:56:28,560 --> 00:56:33,840
But you just have to use some graphing libraries and stuff to make it a little bit more, you know...

776
00:56:33,840 --> 00:56:36,240
I'm not sure what you want to see.

777
00:56:36,240 --> 00:56:41,160
You talking about like a missed block thing for all the validators or...?

778
00:56:41,160 --> 00:56:48,440
I'll bring it up.

779
00:56:48,440 --> 00:56:50,280
Keep talking, I'm slow.

780
00:56:50,280 --> 00:56:53,880
Yeah, you know, the stuff I did with Canvas was super simple.

781
00:56:53,880 --> 00:56:58,600
You know, it's just a series of sequential rectangles.

782
00:56:58,600 --> 00:57:02,560
Yeah, so I think we're getting into the weeds a tiny bit.

783
00:57:02,560 --> 00:57:08,040
So yeah, because you're looking at like a complex like Grafana UI.

784
00:57:08,040 --> 00:57:15,400
So behind the scenes, like a lot of this is probably being drawn with a graphing library.

785
00:57:15,400 --> 00:57:19,880
The reason it's slow with Grafana is...

786
00:57:19,880 --> 00:57:25,840
Well, depending on what you're feeding it from is probably the source interval from Prometheus.

787
00:57:25,840 --> 00:57:32,520
So it's a combination of BlockTime X, whatever the refresh rate is, plus any throttling,

788
00:57:32,520 --> 00:57:35,120
depending on what tier you're on for Grafana.

789
00:57:35,120 --> 00:57:44,160
Like if you had the same UI tooling that Grafana is using, I don't know, D3 or something probably,

790
00:57:44,160 --> 00:57:51,640
and you had a WebSocket feeding it, like live data coming in at the speed of the data,

791
00:57:51,640 --> 00:57:55,240
it would just be real time.

792
00:57:55,240 --> 00:58:04,040
So it's not really a light client, heavy client, because almost every client now that has any

793
00:58:04,040 --> 00:58:12,320
client has any kind of animation, right, and it's JavaScript is not light, but by definition.

794
00:58:12,320 --> 00:58:19,000
But what Todd's done is used the lowest level version of that tool, which is Canvas.

795
00:58:19,000 --> 00:58:26,360
So at some point, almost all this stuff eventually is Canvas or an abstraction upon HTML5 Canvas,

796
00:58:26,360 --> 00:58:27,840
probably.

797
00:58:27,840 --> 00:58:33,480
So by going all the way down to that, you can write something that is painful to write

798
00:58:33,480 --> 00:58:40,480
but performant rather than having to sort of import a bunch of graphing libraries to

799
00:58:40,480 --> 00:58:43,480
do one thing, if you just want to do one thing.

800
00:58:43,480 --> 00:58:47,120
But if you want to do that one thing and you don't want to rewrite all that stuff from

801
00:58:47,120 --> 00:58:50,480
scratch, then that's probably when you bring in a graphing library, right?

802
00:58:50,480 --> 00:58:54,920
Yeah, I just didn't find any cool graphs that did what I wanted.

803
00:58:54,920 --> 00:59:08,000
So I'm not above bringing in e-charts, which I love, but it's huge, right?

804
00:59:08,000 --> 00:59:12,520
So I wasn't trying to be efficient, it's just a side effect.

805
00:59:12,520 --> 00:59:14,840
It just came out that way.

806
00:59:14,840 --> 00:59:15,840
Cool.

807
00:59:15,840 --> 00:59:20,080
Well, I've noticed we're honing up towards an hour now.

808
00:59:20,080 --> 00:59:25,840
Obviously, we lost about five, 10 minutes due to technical issues at the top of the show.

809
00:59:25,840 --> 00:59:32,240
But should we move on to, we've got a couple of Ask Gamer Nodes questions.

810
00:59:32,240 --> 00:59:35,720
So should we do those?

811
00:59:35,720 --> 00:59:40,400
Just before we do, I just want to ask Todd one more thing, and it'll be real quick.

812
00:59:40,400 --> 00:59:46,920
How do you turn on and off the log, which I know I've seen you with a screenshot of

813
00:59:46,920 --> 00:59:50,280
the log, but looking at the one that I've put up?

814
00:59:50,280 --> 00:59:59,560
Everything is done through a config file, just a YAML file, and it's got a variable for it.

815
00:59:59,560 --> 01:00:02,400
Hide log or something like that, yes or no.

816
01:00:02,400 --> 01:00:03,400
It's pretty well documented.

817
01:00:03,400 --> 01:00:06,000
I mean, I put comments on everything.

818
01:00:06,000 --> 01:00:11,800
Yeah, I saw the log flash up when the page was loading and then disappear, so I must

819
01:00:11,800 --> 01:00:14,400
be like a shortcut or something to turn it on.

820
01:00:14,400 --> 01:00:16,440
Yeah, it's checking out.

821
01:00:16,440 --> 01:00:22,000
It's making a rest call to see if it should hide that or not, because it won't get logs

822
01:00:22,000 --> 01:00:26,640
streamed over the WebSocket if it's disabled.

823
01:00:26,640 --> 01:00:27,920
Cool.

824
01:00:27,920 --> 01:00:32,160
So yeah, I guess move on, DeFray, to Ask Gamer Nodes.

825
01:00:32,160 --> 01:00:39,160
Have we had any decent questions this week or any questions at all?

826
01:00:39,160 --> 01:00:40,160
Only it will.

827
01:00:40,160 --> 01:00:48,240
We've had a couple, but one of the ones we have hanging from the other week is ICS, right?

828
01:00:48,240 --> 01:00:53,920
Because I stirred up quite a bit about ICS, and then we've been talking about it quite

829
01:00:53,920 --> 01:01:00,480
a bit on the show, and there was a lot said about it when we went and asked, and then I

830
01:01:00,480 --> 01:01:06,840
wasn't here that episode afterwards, so we didn't actually talk about it.

831
01:01:06,840 --> 01:01:13,800
But the other thing, there was actually an interesting discussion from Whispernode, which

832
01:01:13,800 --> 01:01:16,160
we got tagged in.

833
01:01:16,160 --> 01:01:22,560
So I think that there is an implication there that we were supposed to discuss it, which

834
01:01:22,560 --> 01:01:30,440
was about DapChains and whether or not they are a good thing in Cosmos, which is, I think

835
01:01:30,440 --> 01:01:31,600
is quite an interesting question.

836
01:01:31,600 --> 01:01:45,440
So essentially, the premise here is, do you need a single application chain to run a single

837
01:01:45,440 --> 01:01:46,600
app, right?

838
01:01:46,600 --> 01:01:48,800
Or can you just...

839
01:01:48,800 --> 01:01:55,600
So yeah, I've been under a rock, and I don't validate Cosmos, so thank you, PupMos.

840
01:01:55,600 --> 01:02:00,560
So the comment for those of you listening to podcast players, almost one hour in, no

841
01:02:00,560 --> 01:02:06,080
mention of Prop72, the willpower is admirable from...

842
01:02:06,080 --> 01:02:09,040
I don't even really also know what Prop72 is.

843
01:02:09,040 --> 01:02:12,040
I read something yesterday, but I just...

844
01:02:12,040 --> 01:02:19,120
Yeah, so I think the context for why we were tagged in this is Prop72, and I think as usual,

845
01:02:19,120 --> 01:02:23,920
Shorthy is probably going to have to explain to us using small words what Prop72 is all

846
01:02:23,920 --> 01:02:24,920
about.

847
01:02:24,920 --> 01:02:30,480
But so the thing that we were specifically tagged in before I tag in Shorthy to tell

848
01:02:30,480 --> 01:02:34,840
us all what the hell is actually going on was essentially whether or not DapChains,

849
01:02:34,840 --> 01:02:39,960
i.e. chains that run only one application, are actually a good idea.

850
01:02:39,960 --> 01:02:48,680
And the quote of the tweet that we were specifically tagged in was, several convos in a couple

851
01:02:48,680 --> 01:02:54,840
of days, my opinion right now is that DapChains and Cosmos dilate human and human, and we

852
01:02:54,840 --> 01:03:00,920
have technical and capital capital in the Cosmos ecosystem and competition amongst

853
01:03:00,920 --> 01:03:07,040
L1s is ripping it apart and will continue to do so because there is not enough of any

854
01:03:07,040 --> 01:03:11,080
of the three to go around yet.

855
01:03:11,080 --> 01:03:14,440
That I think is a really interesting take.

856
01:03:14,440 --> 01:03:18,720
And so I'm like, game on, let's talk about it.

857
01:03:18,720 --> 01:03:23,560
But Shorthy, this is Prop72, is it?

858
01:03:23,560 --> 01:03:29,280
Yeah, you know, to be honest, I haven't really reviewed this too much either.

859
01:03:29,280 --> 01:03:34,360
I've seen that there's a lot of drama around it right now, but I myself haven't yet engaged

860
01:03:34,360 --> 01:03:37,360
with it.

861
01:03:37,360 --> 01:03:38,360
Yeah.

862
01:03:38,360 --> 01:03:40,560
Okay, so there's your answer, PopMars.

863
01:03:40,560 --> 01:03:45,600
The reason we didn't engage with Prop72 until an hour in is because we don't really know

864
01:03:45,600 --> 01:03:47,600
what Prop72 is.

865
01:03:47,600 --> 01:03:48,600
No one knows.

866
01:03:48,600 --> 01:03:51,600
It gives a fuck.

867
01:03:51,600 --> 01:03:56,960
I mean, I guess I can tell DR the drama from what I understand.

868
01:03:56,960 --> 01:03:57,960
Beautiful.

869
01:03:57,960 --> 01:03:59,680
I just don't know the specific of this proposal.

870
01:03:59,680 --> 01:04:04,680
So the drama, I believe, really stems around Gravity Bridge, if you boil it down all the

871
01:04:04,680 --> 01:04:10,960
way, because Gravity Bridge originally was a proposal in Cosmos to implement it.

872
01:04:10,960 --> 01:04:17,080
And then rather than doing something to provide value back to Adam, they created their own

873
01:04:17,080 --> 01:04:18,080
chain, right?

874
01:04:18,080 --> 01:04:21,480
And not only do they create their own chain, they also...

875
01:04:21,480 --> 01:04:26,160
I don't know the specifics, but I think they airdrop to iron holders and stuff that was

876
01:04:26,160 --> 01:04:28,840
just considered very controversial.

877
01:04:28,840 --> 01:04:32,040
And I think that this is considered like Gravity Bridge v2.

878
01:04:32,040 --> 01:04:42,280
So basically giving away a huge portion of the community pool to what will effectively

879
01:04:42,280 --> 01:04:46,640
become something that doesn't necessarily bring value back to Adam.

880
01:04:46,640 --> 01:04:49,560
That's my understanding of where the drama's at right now.

881
01:04:49,560 --> 01:04:50,840
Right.

882
01:04:50,840 --> 01:04:53,480
Is there some...

883
01:04:53,480 --> 01:04:57,280
Was some VC slash Lido slash...

884
01:04:57,280 --> 01:05:02,320
There was some other thing that people were upset about?

885
01:05:02,320 --> 01:05:04,120
Ooh.

886
01:05:04,120 --> 01:05:06,880
I know there's two aspects to...

887
01:05:06,880 --> 01:05:12,200
So my understanding is this prop should have been broken up into two props.

888
01:05:12,200 --> 01:05:13,280
There should have been like...

889
01:05:13,280 --> 01:05:19,760
I want to say a liquid-saking side of it and maybe a DeFi side.

890
01:05:19,760 --> 01:05:23,200
And instead of breaking them up into several ones, they've combined them together.

891
01:05:23,200 --> 01:05:27,800
And my understanding is that the DeFi side is considered acceptable and the liquid-staking

892
01:05:27,800 --> 01:05:33,080
is considered creating more monoliths.

893
01:05:33,080 --> 01:05:36,880
Because Lido has, of course, basically taken over all the theorem-staking.

894
01:05:36,880 --> 01:05:41,360
There's something like 60% of all the theorem-staking goes through Lido at this point.

895
01:05:41,360 --> 01:05:47,520
And the take from my understanding, again, this is from very cursory reading, is that

896
01:05:47,520 --> 01:05:51,320
effectively, you'd be funding another monolith to exist.

897
01:05:51,320 --> 01:05:52,720
Whenever the issue...

898
01:05:52,720 --> 01:05:55,640
One of the issues, of course, being that there's already two competitors here, right?

899
01:05:55,640 --> 01:05:59,240
You have Quicksilver, which is already going to be doing liquid-staking.

900
01:05:59,240 --> 01:06:01,240
You have persistence.

901
01:06:01,240 --> 01:06:05,400
And then you also have Celestia, which is also be providing industry and security already.

902
01:06:05,400 --> 01:06:08,280
And so there's all these competitors already that exist.

903
01:06:08,280 --> 01:06:15,200
So why spend, I want to say, like $1.1 million to do something that's already coming to be?

904
01:06:15,200 --> 01:06:20,960
Ah, so it's quite literally bringing Lido into it.

905
01:06:20,960 --> 01:06:21,960
Interesting.

906
01:06:21,960 --> 01:06:29,120
So a comment said that the proposal is promoting Lido to build on ICS-Secure cosmogen chain.

907
01:06:29,120 --> 01:06:30,120
Interesting.

908
01:06:30,120 --> 01:06:31,120
Okay.

909
01:06:31,120 --> 01:06:32,120
Right.

910
01:06:32,120 --> 01:06:33,120
Okay.

911
01:06:33,120 --> 01:06:37,480
That is interesting.

912
01:06:37,480 --> 01:06:45,160
So I don't really know why that's hyper-controversial other than the Gravity Bridge, I guess, precedent.

913
01:06:45,160 --> 01:06:51,120
Adam Burke's made an interesting comment in the comments saying,

914
01:06:51,120 --> 01:06:55,480
Dapp on its own chain is no longer a Dapp anyway, but an App chain.

915
01:06:55,480 --> 01:06:56,480
Immo.

916
01:06:56,480 --> 01:06:57,480
Yeah.

917
01:06:57,480 --> 01:06:58,480
That's fair.

918
01:06:58,480 --> 01:07:00,560
And I mean, I guess that's the thing, isn't it?

919
01:07:00,560 --> 01:07:08,800
I guess the question is whether the thing that is being objected to here...

920
01:07:08,800 --> 01:07:09,800
Because there's actually...

921
01:07:09,800 --> 01:07:10,800
Because there's two separate things on there.

922
01:07:10,800 --> 01:07:17,800
The idea that you don't have App chains is only possible if you build on, say, a Cosmos

923
01:07:17,800 --> 01:07:23,000
and Substrate or something like that, so you can have lots of Apps on one chain.

924
01:07:23,000 --> 01:07:27,320
Otherwise, if you want to write it in Go, then you should probably just write it in

925
01:07:27,320 --> 01:07:28,320
the SDK.

926
01:07:28,320 --> 01:07:29,320
Right?

927
01:07:29,320 --> 01:07:30,320
So there's...

928
01:07:30,320 --> 01:07:31,320
I think there's quite a few things going on here.

929
01:07:31,320 --> 01:07:32,320
Right?

930
01:07:32,320 --> 01:07:37,320
I mean, if you were building a project tomorrow, right, I think that everybody in this chat

931
01:07:37,320 --> 01:07:39,080
would go about it differently, right?

932
01:07:39,080 --> 01:07:40,880
Because we will come from different language backgrounds.

933
01:07:40,880 --> 01:07:44,240
Like, Todd, you would go native, right, presumably.

934
01:07:44,240 --> 01:07:45,240
Yeah.

935
01:07:45,240 --> 01:07:50,680
Who knows what's going to happen, though, with Cosmos and Go?

936
01:07:50,680 --> 01:07:51,680
I mean...

937
01:07:51,680 --> 01:07:57,520
Well, I mean, Cosmos and Go is just not going to have the mind share, I don't think.

938
01:07:57,520 --> 01:08:01,760
It's going to be kind of bloated compared to Rust or C++, right?

939
01:08:01,760 --> 01:08:02,760
So...

940
01:08:02,760 --> 01:08:06,360
So, I'll say just like, are they going to reimplement all the contracts that they've

941
01:08:06,360 --> 01:08:08,560
already implemented and had audited?

942
01:08:08,560 --> 01:08:09,560
Like...

943
01:08:09,560 --> 01:08:10,560
No.

944
01:08:10,560 --> 01:08:16,040
Without being able to use those like a library, it kind of seems like it just won't get any

945
01:08:16,040 --> 01:08:17,040
much...

946
01:08:17,040 --> 01:08:18,040
And who knows, man?

947
01:08:18,040 --> 01:08:20,760
No land looks really neat, too.

948
01:08:20,760 --> 01:08:26,040
I mean, I looked at some of those contracts and it's like super simple.

949
01:08:26,040 --> 01:08:27,040
So...

950
01:08:27,040 --> 01:08:29,960
But I don't know yet.

951
01:08:29,960 --> 01:08:31,800
Yeah.

952
01:08:31,800 --> 01:08:38,760
What I would love is Rust SDK plus Cosmosm, but...

953
01:08:38,760 --> 01:08:42,800
See, for some of us Rust is too intimidating, man.

954
01:08:42,800 --> 01:08:46,080
Now, I started getting into it and I was a love of it and then I ran into the Barrow

955
01:08:46,080 --> 01:08:49,760
Checker and we won a few rounds and I lost.

956
01:08:49,760 --> 01:08:50,760
So...

957
01:08:50,760 --> 01:08:52,880
What editor are you using?

958
01:08:52,880 --> 01:08:56,880
I was using C-Lion, so...

959
01:08:56,880 --> 01:09:04,120
Because I've got Rust Analyzer, LSP server and Emacs and it just like...

960
01:09:04,120 --> 01:09:07,720
When I screw up the Barrow Checking, it goes...

961
01:09:07,720 --> 01:09:08,720
No.

962
01:09:08,720 --> 01:09:15,040
It goes, you've done a bad thing and you just go sit in the corner while I throw compiler

963
01:09:15,040 --> 01:09:21,160
errors at you until you're very, very, very, very, very in pain.

964
01:09:21,160 --> 01:09:22,160
So...

965
01:09:22,160 --> 01:09:27,040
I can tell myself, I'll learn it once I have time.

966
01:09:27,040 --> 01:09:28,040
Yeah.

967
01:09:28,040 --> 01:09:33,040
I think one of the biggest things is just again, coming from a high level language, the package

968
01:09:33,040 --> 01:09:35,280
management is just like...

969
01:09:35,280 --> 01:09:41,800
It's like all the good things about NPM, most of the bad things removed and it's just very,

970
01:09:41,800 --> 01:09:46,480
very straightforward for a systems programming language, package management experience.

971
01:09:46,480 --> 01:09:48,480
It's really good.

972
01:09:48,480 --> 01:09:55,800
And then the DevTooling is pretty decent, but the best thing is the fact that you can

973
01:09:55,800 --> 01:09:59,880
have macros for serialization, deserialization and that sort of stuff where you literally

974
01:09:59,880 --> 01:10:08,120
go like, here's my enum, give me it and Jason.

975
01:10:08,120 --> 01:10:14,160
In my opinion, macros are one of the worst things that it brought over from C++, right?

976
01:10:14,160 --> 01:10:17,600
So that world, because they're opaque.

977
01:10:17,600 --> 01:10:18,600
Yeah.

978
01:10:18,600 --> 01:10:20,160
I mean, the rule of...

979
01:10:20,160 --> 01:10:22,520
You know what the rule of list macros is?

980
01:10:22,520 --> 01:10:24,240
You don't need a macro, right?

981
01:10:24,240 --> 01:10:25,480
That's the motto.

982
01:10:25,480 --> 01:10:27,920
But there's definitely...

983
01:10:27,920 --> 01:10:35,720
There's a point where the hammer and the nail are no longer working and you need to just

984
01:10:35,720 --> 01:10:41,000
improvise some kind of rail gun type situation to drive the nail.

985
01:10:41,000 --> 01:10:43,440
And at that point, you do need a macro.

986
01:10:43,440 --> 01:10:47,080
And there's definitely something in like Rust where you go like, oh, right.

987
01:10:47,080 --> 01:10:50,120
I can see that this must have been implemented via a macro.

988
01:10:50,120 --> 01:10:52,560
Do I have a desire to write my own?

989
01:10:52,560 --> 01:10:53,800
No.

990
01:10:53,800 --> 01:11:01,360
But it's definitely resolving a problem that I had, I guess.

991
01:11:01,360 --> 01:11:04,680
So yeah, I don't know.

992
01:11:04,680 --> 01:11:09,160
I guess the thing that resonated for me with that comment that came in, I guess, was whether

993
01:11:09,160 --> 01:11:12,160
or not there's an argument at this stage.

994
01:11:12,160 --> 01:11:20,160
And I guess this is the core value out of a chain like Juno or secret is that you should

995
01:11:20,160 --> 01:11:21,160
be able to...

996
01:11:21,160 --> 01:11:28,280
Less so secret, I guess, because in order to maintain the privacy guarantees, you have

997
01:11:28,280 --> 01:11:30,720
to stay on secret.

998
01:11:30,720 --> 01:11:35,920
But the idea that you can bootstrap a product and then move it off when you're ready, the

999
01:11:35,920 --> 01:11:39,680
thing that kind of resonated with me about this comment was that the idea that there

1000
01:11:39,680 --> 01:11:43,800
are so few devs in the space that splitting up mind share onto loads of app chains is

1001
01:11:43,800 --> 01:11:46,280
like potentially cross purposes.

1002
01:11:46,280 --> 01:11:51,240
But then it's kind of back to what you were saying about Unix and the original tender

1003
01:11:51,240 --> 01:11:52,240
duty, right?

1004
01:11:52,240 --> 01:11:53,520
Do one thing, do it well.

1005
01:11:53,520 --> 01:11:58,200
And that's the other reasoning behind an app chain, right?

1006
01:11:58,200 --> 01:11:59,200
Yeah.

1007
01:11:59,200 --> 01:12:06,720
And the one thing that kind of struck me when I heard him say it was, I think the whole

1008
01:12:06,720 --> 01:12:16,720
point of Cosmos is this democratization of blockchains and IBC has really helped realize

1009
01:12:16,720 --> 01:12:19,800
this new internet model.

1010
01:12:19,800 --> 01:12:24,960
And I think it's great.

1011
01:12:24,960 --> 01:12:28,040
Is it resulting in a bunch of shut chains?

1012
01:12:28,040 --> 01:12:30,040
Yeah.

1013
01:12:30,040 --> 01:12:33,600
I don't know.

1014
01:12:33,600 --> 01:12:43,360
So yeah, and Pop was just saying Lido would dramatically restructure the validator set

1015
01:12:43,360 --> 01:12:45,880
and be detrimental to delegator suffrage.

1016
01:12:45,880 --> 01:12:53,560
I have to say, I don't know enough about how Lido works in order to sort of talk about

1017
01:12:53,560 --> 01:12:56,800
that any further, to be honest.

1018
01:12:56,800 --> 01:13:02,720
But so my understanding is though Lido bringing the chain online, they're going to use ICS,

1019
01:13:02,720 --> 01:13:03,720
right?

1020
01:13:03,720 --> 01:13:16,200
So this is presumably intended to be the first out in the wild use of interchange security?

1021
01:13:16,200 --> 01:13:25,360
I have no idea there, but the idea of flooding the voting power resulting in a reduction in

1022
01:13:25,360 --> 01:13:28,280
delegator suffrage I think is really interesting.

1023
01:13:28,280 --> 01:13:33,720
I mean, the irony here of course is that we've had a conversation here in this chat probably

1024
01:13:33,720 --> 01:13:40,920
every single time we've met to talk about how we could resolve the fact that the top

1025
01:13:40,920 --> 01:13:46,560
five of a chain of most Cosmos genes can help the network.

1026
01:13:46,560 --> 01:13:49,760
Yeah, flood through the validator set doesn't seem like necessarily a problem to me.

1027
01:13:49,760 --> 01:13:52,080
I think that kind of sounds like a good.

1028
01:13:52,080 --> 01:13:53,080
Yeah.

1029
01:13:53,080 --> 01:13:54,080
Yeah.

1030
01:13:54,080 --> 01:14:05,160
I actually want to address the comment earlier before as well about the brain drain.

1031
01:14:05,160 --> 01:14:09,760
So I think that I think it's a really apt point, but I think that it's a little bit.

1032
01:14:09,760 --> 01:14:12,840
The conversation is unfortunately nuanced.

1033
01:14:12,840 --> 01:14:19,840
There's actually a Cosmos SDK that I expect out that effectively I want to develop as

1034
01:14:19,840 --> 01:14:22,840
an SDK because it could benefit all chains.

1035
01:14:22,840 --> 01:14:25,320
The problem is there's no funding for it.

1036
01:14:25,320 --> 01:14:32,280
So I've gone to several different building funders, I guess, grantees, and everyone's

1037
01:14:32,280 --> 01:14:33,680
like, this is an amazing idea.

1038
01:14:33,680 --> 01:14:35,400
I think you should profit.

1039
01:14:35,400 --> 01:14:36,680
We cannot fund you for it.

1040
01:14:36,680 --> 01:14:39,920
If you want to create your own chain that only does this, great.

1041
01:14:39,920 --> 01:14:47,800
If you want to build it on Adam or on Juno or on Osmosis, we can't fund that.

1042
01:14:47,800 --> 01:14:49,080
We can't fund it at all.

1043
01:14:49,080 --> 01:14:52,360
So really the only recourse for getting funding is building your own chain.

1044
01:14:52,360 --> 01:14:56,320
And it does have that side effect of there is that brain drain.

1045
01:14:56,320 --> 01:15:01,480
But I think that a more interesting side effect is that, not side effect, but observation

1046
01:15:01,480 --> 01:15:07,840
is that if you look at most Cosmos chains, get hubs, they have like last six months,

1047
01:15:07,840 --> 01:15:09,440
they've done no development.

1048
01:15:09,440 --> 01:15:14,240
So it's kind of hard to claim there's a brain drain whenever they aren't actually doing

1049
01:15:14,240 --> 01:15:15,440
any development.

1050
01:15:15,440 --> 01:15:21,840
So I'm curious now, what was your idea about the SDK?

1051
01:15:21,840 --> 01:15:26,640
Oh, basically, I mean, it's fairly simple.

1052
01:15:26,640 --> 01:15:32,160
The idea is essentially, if you've used Venmo or Zell or Cache app or whatever, a request

1053
01:15:32,160 --> 01:15:33,160
feature.

1054
01:15:33,160 --> 01:15:39,400
So create another module where you can essentially wrap any other transaction in a request.

1055
01:15:39,400 --> 01:15:44,840
So let's say you're shooting down your validator and you put out a 30-day notice saying, hey,

1056
01:15:44,840 --> 01:15:46,640
I'm shooting down my validator.

1057
01:15:46,640 --> 01:15:51,840
You could then send out a request to all your delegators with an undeligated or a redelegate

1058
01:15:51,840 --> 01:15:55,680
transaction, which they would then accept and then be able to go through and redelegate

1059
01:15:55,680 --> 01:15:59,360
their funds without having to go through any sort of rigmarole around staking.

1060
01:15:59,360 --> 01:16:03,920
Or even if, let's just say you and I went out to coffee or whatever, and I paid for

1061
01:16:03,920 --> 01:16:10,320
lunch, I could send you a one Juneau request to pay me back.

1062
01:16:10,320 --> 01:16:15,320
Just something simple like that, but wrapping all transactions.

1063
01:16:15,320 --> 01:16:24,120
So kind of like AuthC does for allowances, kind of just a way of flipping the flipping

1064
01:16:24,120 --> 01:16:29,720
that I want to make this type of message of Cosmos message type instead saying, I would

1065
01:16:29,720 --> 01:16:37,480
like you to acknowledge this by sending me back the Cosmos message type I've requested.

1066
01:16:37,480 --> 01:16:39,120
Yeah, exactly right.

1067
01:16:39,120 --> 01:16:42,760
And the idea would even be that you could add a chain of events so be able to send to

1068
01:16:42,760 --> 01:16:47,160
multiple people, you just send an adjacent blog and it would give it out to everyone

1069
01:16:47,160 --> 01:16:51,440
that it needs to go out to and then they could just respond and execute it.

1070
01:16:51,440 --> 01:16:55,400
Yeah, that's kind of cool.

1071
01:16:55,400 --> 01:17:04,520
But as I guess you've touched on the incentives are quite skewed at the moment, aren't they?

1072
01:17:04,520 --> 01:17:16,680
Which this is actually, the interesting thing is that as well, launching a smart contract

1073
01:17:16,680 --> 01:17:22,920
on a smart contract chain in Cosmos is also economically not as beneficial.

1074
01:17:22,920 --> 01:17:27,440
And especially if you want to get investor funding like for how social and stuff like

1075
01:17:27,440 --> 01:17:30,600
we are bootstrapping it ourselves.

1076
01:17:30,600 --> 01:17:35,200
But I know that it also makes sense to launch on a chain where there is a large user base

1077
01:17:35,200 --> 01:17:39,040
already rather than trying to bootstrap your own.

1078
01:17:39,040 --> 01:17:45,720
But if you were going after investment money, it will be a lot easier to do it as a single

1079
01:17:45,720 --> 01:17:51,280
shot chain with only one permissioned smart contract running on it.

1080
01:17:51,280 --> 01:17:58,560
And I think there have been a few projects that have moved over to Cosmos from Terra that

1081
01:17:58,560 --> 01:18:02,640
are basically going that route where they're going to just bootstrap as their own chain

1082
01:18:02,640 --> 01:18:09,400
like one shot and they're going to bootstrap with probably almost no users and maybe die

1083
01:18:09,400 --> 01:18:15,520
in that wilderness because their investors are like, got to be a blockchain.

1084
01:18:15,520 --> 01:18:18,360
So it cuts both ways, doesn't it?

1085
01:18:18,360 --> 01:18:26,680
Yeah, I mean, my vision for why Juno is great is effectively my imagining of what a chain

1086
01:18:26,680 --> 01:18:32,680
should do or what an app should do is it should use Juno as or secret or in a generalized

1087
01:18:32,680 --> 01:18:37,760
smart contract platform, create their platform, create their community, create their base.

1088
01:18:37,760 --> 01:18:41,960
And if they find that they need to expand out, then they can go do that.

1089
01:18:41,960 --> 01:18:47,680
And my vision of what Juno should be is effectively that sandbox for lack of a better term to create

1090
01:18:47,680 --> 01:18:50,640
that community, especially when it already has such a strong community.

1091
01:18:50,640 --> 01:18:55,320
And so if there ends up being that desire where you find some reason to bounce out, that's

1092
01:18:55,320 --> 01:18:56,800
when you do it.

1093
01:18:56,800 --> 01:19:02,160
I think it's kind of a, the question is like a short term to long term goals to me.

1094
01:19:02,160 --> 01:19:07,040
I don't think that necessarily all new chains are short term oriented.

1095
01:19:07,040 --> 01:19:09,040
I certainly don't want to make that claim.

1096
01:19:09,040 --> 01:19:12,960
But I think that if you're going for like a really long term project, build somewhere

1097
01:19:12,960 --> 01:19:17,840
that already exists and expand if you need to versus if you need cash now, you would

1098
01:19:17,840 --> 01:19:22,320
just create your own chain and then dump on retail effectively.

1099
01:19:22,320 --> 01:19:26,280
Yeah.

1100
01:19:26,280 --> 01:19:35,480
So moving on to the interchange security bit, because that seems related to this 72 thing.

1101
01:19:35,480 --> 01:19:38,680
So the background to interchange security and why we're bringing up on the show is because

1102
01:19:38,680 --> 01:19:46,120
I said, I went on Twitter and just went, okay, don't really understand why we need

1103
01:19:46,120 --> 01:19:52,320
interchange security because it's not really hard to get a validator set.

1104
01:19:52,320 --> 01:19:58,160
And if your chain doesn't work out, that's too bad.

1105
01:19:58,160 --> 01:20:05,240
But like finding validators, admittedly potentially varying qualities is not actually a problem

1106
01:20:05,240 --> 01:20:09,200
that has existed for some time.

1107
01:20:09,200 --> 01:20:14,560
And so I was like, what is the actual use case?

1108
01:20:14,560 --> 01:20:17,720
What is the problem that ICS is solving?

1109
01:20:17,720 --> 01:20:22,160
And I guess the thing that I guess I wasn't really considering as part of this is from

1110
01:20:22,160 --> 01:20:26,600
somebody who's more used to working in a smart contract environment.

1111
01:20:26,600 --> 01:20:33,880
I'm used to having what Shorthy just described, which is an on ramp for an idea into a community.

1112
01:20:33,880 --> 01:20:40,840
But if you're building stuff at chain level, at SDK level, then maybe ICS is a better fit

1113
01:20:40,840 --> 01:20:41,840
for that.

1114
01:20:41,840 --> 01:20:46,720
And Twitter was very quick to tell me I was wrong in about 15 different ways.

1115
01:20:46,720 --> 01:20:52,960
And so I'm going to read out some of the responses that were very good on why we need

1116
01:20:52,960 --> 01:20:54,440
interchange security.

1117
01:20:54,440 --> 01:20:58,640
So I'm going to just do some call outs here.

1118
01:20:58,640 --> 01:21:01,280
PhonicG made some really, really good points.

1119
01:21:01,280 --> 01:21:07,120
I can't put them all in full because it would take me ages to read out a thread around sort

1120
01:21:07,120 --> 01:21:10,560
of market cap and stuff like that.

1121
01:21:10,560 --> 01:21:18,480
Well, XBT also said having a small market cap is dangerous when starting out.

1122
01:21:18,480 --> 01:21:26,360
And then Simon Water, friend of the show from Confio from Cosmosm, said that he essentially

1123
01:21:26,360 --> 01:21:29,600
backed up that this was on a longer thread.

1124
01:21:29,600 --> 01:21:32,000
Saying a small market cap means you can buy out the chain.

1125
01:21:32,000 --> 01:21:36,800
It's an important attack factor to consider in a bear market or for new projects.

1126
01:21:36,800 --> 01:21:42,080
And I think, again, especially tail end of last year, a lot of the chains that were launching,

1127
01:21:42,080 --> 01:21:44,880
hua hua, right?

1128
01:21:44,880 --> 01:21:46,000
Huge market cap.

1129
01:21:46,000 --> 01:21:49,200
So you look at that and you go, where does interchange security fit into this when a

1130
01:21:49,200 --> 01:21:55,120
literal meme chain can have a multi-hundred million dollar market cap?

1131
01:21:55,120 --> 01:21:57,000
It becomes more obvious in a bear market.

1132
01:21:57,000 --> 01:22:00,520
So thank you, Simon, for pointing that out.

1133
01:22:00,520 --> 01:22:09,880
Eric Opp said there's pros and cons to ICS and chains should decide and added a hot take

1134
01:22:09,880 --> 01:22:11,440
on the end, right?

1135
01:22:11,440 --> 01:22:13,880
And this one I like.

1136
01:22:13,880 --> 01:22:18,720
Interchange security is Atom's last chance to stay relevant after gravity decks and gravity

1137
01:22:18,720 --> 01:22:20,200
bridge failure.

1138
01:22:20,200 --> 01:22:25,200
If ICS isn't a hit, then it's over.

1139
01:22:25,200 --> 01:22:27,200
Spicy hot take.

1140
01:22:27,200 --> 01:22:30,920
So we continue on there, the Freya.

1141
01:22:30,920 --> 01:22:38,680
I've got a comment just about the market caps and buying out a chain.

1142
01:22:38,680 --> 01:22:46,200
I think that it probably would come cost prohibitive even for a shit chain because once you start

1143
01:22:46,200 --> 01:22:53,440
to buy all the tokens specifically with how a decks functions, you are going to get cost

1144
01:22:53,440 --> 01:22:55,880
prohibitive pretty quick.

1145
01:22:55,880 --> 01:23:00,800
You can't just keep buying the tokens without the market cap rising significantly.

1146
01:23:00,800 --> 01:23:02,800
But that's why they halted Terra.

1147
01:23:02,800 --> 01:23:06,840
I mean, it was happening.

1148
01:23:06,840 --> 01:23:07,960
Terra worked differently though.

1149
01:23:07,960 --> 01:23:16,760
It was like a mechanism where you cash in the UST and it was a new look.

1150
01:23:16,760 --> 01:23:19,000
That's right now.

1151
01:23:19,000 --> 01:23:24,560
Terra Classic, you can't register a new validator.

1152
01:23:24,560 --> 01:23:28,480
You can't delegate new tokens to a validator.

1153
01:23:28,480 --> 01:23:34,840
You can redelegate, but otherwise that was all a hard fork that went into the code because

1154
01:23:34,840 --> 01:23:40,800
once it devalued, it was possible to take over.

1155
01:23:40,800 --> 01:23:48,080
So yeah, but it devalued to basically nothing.

1156
01:23:48,080 --> 01:23:54,520
And the distribution of tokens is all over the place because of the way that that mint

1157
01:23:54,520 --> 01:23:57,360
printing module was working when you were cashing in UST.

1158
01:23:57,360 --> 01:24:05,200
But I think for regular chains, most of them aren't even on a sex, but even on a CAX, once

1159
01:24:05,200 --> 01:24:09,080
you start to try and buy up a meaningful amount of tokens, it's just going to push the price

1160
01:24:09,080 --> 01:24:13,480
sky high unless you do it over a really, really, really long time, which I think then people

1161
01:24:13,480 --> 01:24:18,560
would start to notice before you got anywhere near a big enough amount to actually do some

1162
01:24:18,560 --> 01:24:19,840
damage to a chain.

1163
01:24:19,840 --> 01:24:25,520
I wonder if you, if right, it's okay, like if I, okay, let's imagine I have loads of

1164
01:24:25,520 --> 01:24:28,880
money and I'm a bad guy, I'm a real bad dude.

1165
01:24:28,880 --> 01:24:35,440
I would imagine the way to go about it would be to try and over the counter purchase from

1166
01:24:35,440 --> 01:24:43,520
founders of a chain, but go to the niche individually and just gamble that they didn't tell each

1167
01:24:43,520 --> 01:24:47,680
other that they were doing it out of either embarrassment or sneakiness.

1168
01:24:47,680 --> 01:24:53,600
And then just see if you can get 33% because a lot of chains do have that much vested in

1169
01:24:53,600 --> 01:24:54,720
the founding team.

1170
01:24:54,720 --> 01:24:59,480
You know, so that might be possible, but imagine this right.

1171
01:24:59,480 --> 01:25:00,480
What's the point?

1172
01:25:00,480 --> 01:25:03,240
I'm imagining I've got my imagination hat on.

1173
01:25:03,240 --> 01:25:09,280
So if you go and get all this effort right to get all this token and spend all this money

1174
01:25:09,280 --> 01:25:13,560
to get all this token, why are you going to maliciously attack a chain that you own all

1175
01:25:13,560 --> 01:25:14,920
the token for?

1176
01:25:14,920 --> 01:25:19,400
If you drain the chain, you can't possibly fucking sell it without killing the price

1177
01:25:19,400 --> 01:25:20,400
of it anyway.

1178
01:25:20,400 --> 01:25:22,400
It just doesn't make any sense.

1179
01:25:22,400 --> 01:25:23,720
But I think that's the point though, right?

1180
01:25:23,720 --> 01:25:27,640
So like, D, DYDX is coming to Cosmos, right?

1181
01:25:27,640 --> 01:25:32,040
They're probably going to be, as soon as they spin up $100 million project.

1182
01:25:32,040 --> 01:25:34,320
But there's already other DeFi projects.

1183
01:25:34,320 --> 01:25:36,480
Let's say they're worth 1 million.

1184
01:25:36,480 --> 01:25:39,680
They might just buy all the tokens just to literally kill the project.

1185
01:25:39,680 --> 01:25:43,360
I think that's the very concern rather than some guy deciding that he likes the project

1186
01:25:43,360 --> 01:25:45,000
enough to kill it.

1187
01:25:45,000 --> 01:25:51,360
It's a computing entity that says, this is a really great product and they could take

1188
01:25:51,360 --> 01:25:52,720
our share.

1189
01:25:52,720 --> 01:25:54,400
We can't have that.

1190
01:25:54,400 --> 01:26:01,760
I submit to you that if a project's worth a million dollars, it's worth killing.

1191
01:26:01,760 --> 01:26:03,760
I mean, in crypto, yeah.

1192
01:26:03,760 --> 01:26:08,120
In normal startup land, I would feel pretty proud to start a million dollar company.

1193
01:26:08,120 --> 01:26:10,360
I'd be like, this is kind of nice.

1194
01:26:10,360 --> 01:26:15,520
Yeah, million dollar company mate.

1195
01:26:15,520 --> 01:26:20,480
Maybe before the crash.

1196
01:26:20,480 --> 01:26:24,120
So I just want to pull out a few comments we've had in the chat.

1197
01:26:24,120 --> 01:26:28,640
So Putmar says, interchain security comes with a sacrifice of self sovereignty, which

1198
01:26:28,640 --> 01:26:30,360
try and save very quickly.

1199
01:26:30,360 --> 01:26:31,360
That's quite tricky.

1200
01:26:31,360 --> 01:26:34,120
A lot of S's in that one.

1201
01:26:34,120 --> 01:26:38,440
Gravity Bridge is a prime candidate for interchain security because its token does not provide

1202
01:26:38,440 --> 01:26:42,880
sufficient economic security to the collateral it holds.

1203
01:26:42,880 --> 01:26:49,280
Juno.Love says, Fortis was worth 35 billion.

1204
01:26:49,280 --> 01:26:50,280
My brain just halted.

1205
01:26:50,280 --> 01:26:51,280
Is that a billion?

1206
01:26:51,280 --> 01:26:52,280
Yeah, it's a.

1207
01:26:52,280 --> 01:26:53,280
Yeah, on paper.

1208
01:26:53,280 --> 01:26:55,120
By the way, do you like my imagination hat?

1209
01:26:55,120 --> 01:26:58,880
This is my imagination hat.

1210
01:26:58,880 --> 01:27:01,080
And it looks very special.

1211
01:27:01,080 --> 01:27:03,800
Yeah, it's very sensational.

1212
01:27:03,800 --> 01:27:08,720
And then the variety of people in the chat have basically been advocating different DGEN

1213
01:27:08,720 --> 01:27:10,520
strategies for attacking a chain.

1214
01:27:10,520 --> 01:27:15,280
So we've obviously started something with that line of questioning.

1215
01:27:15,280 --> 01:27:22,640
So a final couple of points I had from other people, but read out in my voice so that the

1216
01:27:22,640 --> 01:27:27,320
clever points will attach themselves in your mind to me.

1217
01:27:27,320 --> 01:27:36,440
But Trill and Juno News said that ICS shouldn't be viewed as an end in itself.

1218
01:27:36,440 --> 01:27:41,600
You should have the set as interim when you go when you move to an app chain and then

1219
01:27:41,600 --> 01:27:44,680
you can get different new validators.

1220
01:27:44,680 --> 01:27:50,280
So I think what they're referring to there is that there's a second version of ICS coming

1221
01:27:50,280 --> 01:28:00,360
later that allows you to layer some validates from one set and have new validators.

1222
01:28:00,360 --> 01:28:05,440
So that's an interesting development as and when it comes.

1223
01:28:05,440 --> 01:28:12,760
And then the final point here, this is a good one, is from H. Devalence, who's one of the

1224
01:28:12,760 --> 01:28:17,160
founders of Penumbra.

1225
01:28:17,160 --> 01:28:22,120
And one of those people who's very, very clever and whenever you use crypto libraries in Rust,

1226
01:28:22,120 --> 01:28:26,520
you see their GitHub handle come up all the time to remind you that you are just a small

1227
01:28:26,520 --> 01:28:29,600
brain grug developer.

1228
01:28:29,600 --> 01:28:34,800
But they said app specific chains can have their own security and be fully self-reinforced

1229
01:28:34,800 --> 01:28:42,000
and pay the cost of that, or they can be secured by the hub or an ICS chain.

1230
01:28:42,000 --> 01:28:43,960
And this is the really powerful part.

1231
01:28:43,960 --> 01:28:49,640
They can choose somewhere right in the middle and they can change their choice at any time.

1232
01:28:49,640 --> 01:28:52,760
And that is a reference again to layering.

1233
01:28:52,760 --> 01:29:01,160
So yeah, those are all of the tepids to hot takes that I thought were worth mentioning

1234
01:29:01,160 --> 01:29:05,520
on the ICS thing.

1235
01:29:05,520 --> 01:29:09,600
I'm kind of going to go and say that I think I stand corrected on ICS.

1236
01:29:09,600 --> 01:29:11,160
I think there is a use case for it.

1237
01:29:11,160 --> 01:29:16,800
I think I just didn't see it initially because I was blinked by Cosmosm and by only building

1238
01:29:16,800 --> 01:29:19,280
on top of smart contract chains.

1239
01:29:19,280 --> 01:29:25,720
Well, yeah, it's like horizontal scaling, right?

1240
01:29:25,720 --> 01:29:30,240
You can only scale so big on Cosmosm before it's bloated and can't do shit.

1241
01:29:30,240 --> 01:29:37,320
Well, I mean, the other side of the conversation is calling Interchain Security worthless,

1242
01:29:37,320 --> 01:29:39,480
calling Polkadot worthless?

1243
01:29:39,480 --> 01:29:47,720
Yes.

1244
01:29:47,720 --> 01:29:51,680
That was the deep desire of the fray when he said that comment originally is that he

1245
01:29:51,680 --> 01:30:09,360
was throwing shade at Polkadot indirectly.