1
00:00:00,000 --> 00:00:03,320
Like the fray is still in a basement.

2
00:00:03,720 --> 00:00:09,360
So, you know, probably not going to be showing up any time soon.

3
00:00:09,560 --> 00:00:10,400
You just want to roll it.

4
00:00:14,040 --> 00:00:19,720
Welcome to Game of Nodes, a weekly podcast from independent value entities.

5
00:00:23,000 --> 00:00:27,880
Hello, loyal fan and welcome to Game of Nodes.

6
00:00:27,880 --> 00:00:35,000
This is our 111th episode, Critical Fail, which, you know, these

7
00:00:35,680 --> 00:00:42,160
these titles just get came up with without any consultation with anyone and generally make no sense whatsoever.

8
00:00:42,320 --> 00:00:52,080
Except it seems like this week they may make sense with what we're probably going to drone on about for the next at least 30 minutes, I'm assuming.

9
00:00:52,080 --> 00:01:03,040
One of our guys, the fray, decided that he wanted to, you know, jam in a basement instead of come and chill on the show.

10
00:01:03,560 --> 00:01:07,640
So, I hope you're enjoying your music.

11
00:01:07,640 --> 00:01:08,800
It was watching.

12
00:01:09,240 --> 00:01:11,760
Yeah, I was watching a jam, not playing it.

13
00:01:12,320 --> 00:01:13,400
Oh, wait.

14
00:01:13,400 --> 00:01:16,640
No, I yeah, he's actually at a venue though, right?

15
00:01:16,680 --> 00:01:16,960
Yeah.

16
00:01:16,960 --> 00:01:17,520
Yeah.

17
00:01:17,520 --> 00:01:20,160
He's not just in his basement watching it on TV.

18
00:01:20,160 --> 00:01:21,800
That would be something.

19
00:01:21,800 --> 00:01:22,800
Oh, I'm fucking you.

20
00:01:22,800 --> 00:01:24,600
That would be a new love for skipping a God.

21
00:01:24,600 --> 00:01:27,040
Sorry, boys, I'm not staring.

22
00:01:27,040 --> 00:01:27,560
I'm not staring.

23
00:01:27,560 --> 00:01:28,080
I'm not staring.

24
00:01:28,080 --> 00:01:29,080
I need to watch some TV.

25
00:01:29,080 --> 00:01:30,720
It's like two flights of stairs.

26
00:01:30,720 --> 00:01:31,720
I gotta go up.

27
00:01:32,920 --> 00:01:37,840
And Rahm is already in the comments, hosing us for what?

28
00:01:38,240 --> 00:01:39,240
Sounds like the last episode.

29
00:01:39,240 --> 00:01:41,640
I think that every, remember, it says it sounds like the last episode.

30
00:01:41,640 --> 00:01:42,640
I think that every week.

31
00:01:44,120 --> 00:01:45,120
We circled the drain.

32
00:01:45,440 --> 00:01:47,360
Why would it sound like the last episode ever?

33
00:01:47,360 --> 00:01:49,120
Is it because of the intro?

34
00:01:49,120 --> 00:01:50,920
The intro was, I don't know.

35
00:01:50,920 --> 00:01:52,920
He's a downer.

36
00:01:53,600 --> 00:01:58,520
I hope it's the Lord of the Rings reference because Bilbo on his 111th birthday,

37
00:01:58,920 --> 00:02:00,520
it's his last birthday.

38
00:02:00,520 --> 00:02:02,640
And so this is our 111th episode.

39
00:02:03,040 --> 00:02:04,800
Could be the last episode.

40
00:02:04,800 --> 00:02:06,920
That is some fucking deep reference.

41
00:02:09,680 --> 00:02:10,680
Holy shit.

42
00:02:11,400 --> 00:02:13,200
Dude, that's a man.

43
00:02:13,880 --> 00:02:15,480
So I haven't looked at the news.

44
00:02:15,480 --> 00:02:17,120
Do we have any news this week?

45
00:02:17,120 --> 00:02:17,880
We do.

46
00:02:17,880 --> 00:02:18,720
Yeah, there's a couple things.

47
00:02:18,720 --> 00:02:19,720
I don't know.

48
00:02:19,720 --> 00:02:21,440
Is Rahm contributing to the news?

49
00:02:21,640 --> 00:02:22,640
I don't know.

50
00:02:22,640 --> 00:02:24,440
There's like one link and like two notes.

51
00:02:24,760 --> 00:02:28,640
So maybe he broke a collarbone or a clavicle or some shit.

52
00:02:30,160 --> 00:02:33,200
Oh, well, there's episode 111.

53
00:02:33,200 --> 00:02:34,080
That's the one we're on.

54
00:02:34,080 --> 00:02:37,320
Yeah, there's a, all right, maybe it's time to start with that though.

55
00:02:37,320 --> 00:02:38,440
I don't want to start with that.

56
00:02:38,440 --> 00:02:40,440
No, let's talk about it.

57
00:02:40,440 --> 00:02:42,440
So it's all we're having.

58
00:02:42,440 --> 00:02:44,520
Hey, we're already at the 17 viewers.

59
00:02:44,520 --> 00:02:45,040
So you look at that.

60
00:02:45,040 --> 00:02:46,840
Just fucking amazing.

61
00:02:46,840 --> 00:02:47,240
Yeah.

62
00:02:47,240 --> 00:02:50,640
I mean, I reckon they were always in fire hover over this.

63
00:02:51,040 --> 00:02:56,240
Yeah, a lot of those are bullshit views from X Twitter.

64
00:02:56,640 --> 00:02:57,120
Yeah.

65
00:02:57,120 --> 00:03:02,480
So OK, well, at least actually all of us are having some.

66
00:03:02,480 --> 00:03:05,040
Well, I'm not having problems, but both of you two are having problems

67
00:03:05,040 --> 00:03:09,040
with various node related things this week.

68
00:03:09,320 --> 00:03:13,680
Actually, yeah, well, there's problems later on.

69
00:03:13,680 --> 00:03:19,720
I want to talk a little bit about the magical fix of injective.

70
00:03:20,120 --> 00:03:25,320
And and you guys were talking previously and probably wanted to continue

71
00:03:25,320 --> 00:03:27,880
your your musings about your nodes.

72
00:03:28,920 --> 00:03:33,520
I think she'll see is participating in a pretty high volume

73
00:03:33,520 --> 00:03:43,080
Xion testnet application, which is minting NFTs over IBC,

74
00:03:44,080 --> 00:03:48,800
which is a bit of a newish type of application

75
00:03:49,160 --> 00:03:53,200
and having some issues with with RPC nodes.

76
00:03:54,040 --> 00:03:57,680
And you serve her as a pretty high volume

77
00:03:58,360 --> 00:04:02,080
say deployment, which has also been having some issues lately.

78
00:04:02,080 --> 00:04:04,360
I think you guys wanted to talk a little bit like about that.

79
00:04:04,760 --> 00:04:09,600
And then, yeah, I wanted to talk, like I said before, about the injective stuff,

80
00:04:09,600 --> 00:04:11,800
which is quite interesting.

81
00:04:11,800 --> 00:04:15,280
And anyway, you guys can take it.

82
00:04:16,000 --> 00:04:18,120
Tell me what's going on with burnt.

83
00:04:18,120 --> 00:04:21,120
I don't know. I'm kind of excited about the injective improvements.

84
00:04:21,160 --> 00:04:21,960
I'm honest.

85
00:04:21,960 --> 00:04:24,880
Like, I want to talk about that first, she'll see.

86
00:04:24,880 --> 00:04:28,000
And then we'll sort of roll into your your Xion problems.

87
00:04:28,320 --> 00:04:31,280
Yeah. Yeah, let's do that, because I'm pretty stoked about the injective

88
00:04:31,280 --> 00:04:34,760
signing problem. So right now, for my signer, I have three nodes

89
00:04:34,760 --> 00:04:36,240
that I've been rotating between.

90
00:04:36,240 --> 00:04:39,320
And every single day, a new node resinks.

91
00:04:39,760 --> 00:04:43,480
And that's how I was able to keep up to, like, 87% uptime

92
00:04:43,480 --> 00:04:47,400
with three full machines, full nodes, resinking daily.

93
00:04:47,800 --> 00:04:49,320
That that's what I had to do.

94
00:04:49,320 --> 00:04:52,200
The fact that there was one little config change that had to happen

95
00:04:52,200 --> 00:04:56,880
and suddenly now, like 99% of time, I'm outraged, outraged, I say.

96
00:04:56,880 --> 00:05:03,480
So a little bit of background on the injective protocol.

97
00:05:03,480 --> 00:05:09,040
So as a validator on injective, it is quite frustrating trying to

98
00:05:09,480 --> 00:05:11,440
maintain a high uptime.

99
00:05:11,440 --> 00:05:16,600
There's a lot of, there's like a lot of IO on that application

100
00:05:16,600 --> 00:05:19,600
and a lot of nodes struggle to keep up.

101
00:05:20,840 --> 00:05:25,640
I'm assuming with the head and P2P to be able to

102
00:05:25,640 --> 00:05:29,720
sign all of the blocks and the remedies to date have been that

103
00:05:29,720 --> 00:05:33,360
you need to keep a relatively fresh database that is,

104
00:05:34,160 --> 00:05:37,520
you know, deleting your database and resinking from something

105
00:05:37,520 --> 00:05:44,360
that's either cousin pruned quite freshly or even a state sync.

106
00:05:44,360 --> 00:05:47,480
But I'm not sure if there is actually state sync issues on

107
00:05:48,200 --> 00:05:52,280
injective. So I'm pretty sure most of the snapshots that are

108
00:05:52,280 --> 00:05:56,200
coming through that are smaller made small by cousin pruned.

109
00:05:56,200 --> 00:05:59,720
Well, so that's mostly true, but not entirely true.

110
00:05:59,720 --> 00:06:03,040
There is one individual who has state sync working and that's high

111
00:06:03,040 --> 00:06:06,400
stake. Their snapshots are glorious.

112
00:06:06,400 --> 00:06:08,280
They're like seven gigabytes or something.

113
00:06:08,280 --> 00:06:12,360
And the way they fixed state sync quote unquote fixed is they

114
00:06:12,360 --> 00:06:14,000
turned off errors for state syncing.

115
00:06:14,000 --> 00:06:16,200
And so they modified the binary itself.

116
00:06:16,200 --> 00:06:17,000
Yeah.

117
00:06:17,000 --> 00:06:21,200
So it's just pushes through and that's what I was thinking of from

118
00:06:21,200 --> 00:06:24,320
every day. Every single day I download their latest snapshot and

119
00:06:24,320 --> 00:06:25,840
then run off of that.

120
00:06:25,840 --> 00:06:27,800
Shout out to high stake.

121
00:06:27,800 --> 00:06:29,240
They are glorious.

122
00:06:29,240 --> 00:06:35,680
So my validator nodes there have had pretty good uptime since we

123
00:06:35,680 --> 00:06:40,440
started comparatively speaking to like a lot of other operators.

124
00:06:40,440 --> 00:06:42,040
And I don't know why that is.

125
00:06:42,040 --> 00:06:48,960
I think a lot of it comes down to just luck in P2P and peering.

126
00:06:48,960 --> 00:06:53,160
But we seem to have been able to maintain probably between like

127
00:06:53,160 --> 00:06:56,680
503,000 misses per the window.

128
00:06:56,680 --> 00:07:03,080
A lot of other operators are sort of up around the 20 to 30,000

129
00:07:03,080 --> 00:07:08,680
range or probably at the 15 to 30,000 misses range in the window,

130
00:07:08,680 --> 00:07:12,320
which is what 100,000 blocks, Shilty, I think.

131
00:07:12,320 --> 00:07:13,640
Yeah, 100,000 blocks.

132
00:07:13,640 --> 00:07:15,280
100,000.

133
00:07:15,280 --> 00:07:21,880
So anyway, historically difficult to get a good signing rate.

134
00:07:21,880 --> 00:07:27,280
So keeping a fresh database, there are like certain settings that

135
00:07:27,280 --> 00:07:31,720
are recommended like peer limits and, geez, I can't remember what

136
00:07:31,720 --> 00:07:33,000
the other ones are offhand.

137
00:07:33,000 --> 00:07:40,240
But they typically need just blazing fast core speeds,

138
00:07:40,240 --> 00:07:41,640
like as fast as you can get.

139
00:07:41,640 --> 00:07:46,080
Bigger drives, fastest drives you can get your hands on.

140
00:07:46,080 --> 00:07:49,560
I don't think the RAM has particularly mattered too much in the past.

141
00:07:49,560 --> 00:07:54,040
And sort of RAID zeroing things, just getting as much IO and thread

142
00:07:54,040 --> 00:07:58,360
speed as possible on these machines to just sort of try and keep up.

143
00:07:58,360 --> 00:08:05,560
And Horcrux has helped with my nodes by having multiple distributed

144
00:08:05,560 --> 00:08:08,520
around the place and using Horcrux.

145
00:08:08,520 --> 00:08:17,400
So I think we run three dedicated servers for Injective as well with Horcrux.

146
00:08:17,400 --> 00:08:24,800
And the team at Injective Labs last week, we started testing some settings.

147
00:08:24,800 --> 00:08:27,880
Do you have those settings offhand, Shilty?

148
00:08:27,880 --> 00:08:29,280
I can pull them up.

149
00:08:29,280 --> 00:08:31,280
Not offhand, no.

150
00:08:31,280 --> 00:08:35,880
I don't remember what sections they actually were from, but I will.

151
00:08:35,880 --> 00:08:37,160
Peer to peer, I think.

152
00:08:37,160 --> 00:08:39,160
It was in the peer to peer, was it?

153
00:08:39,160 --> 00:08:41,160
I'm fairly sure.

154
00:08:41,160 --> 00:08:47,160
Was it Oracle changes or binary, like configure app Toml type shit?

155
00:08:47,160 --> 00:08:50,160
It's in the config Toml.

156
00:08:50,160 --> 00:08:51,160
And what did they do?

157
00:08:51,160 --> 00:08:54,160
They changed some signing windows or something?

158
00:08:54,160 --> 00:08:57,160
I'll tell you what they are here in a second.

159
00:08:57,160 --> 00:09:00,160
Peer to peer latency, I think.

160
00:09:00,160 --> 00:09:05,160
If I remember correctly, there is a config for basically disconnecting a little early.

161
00:09:05,160 --> 00:09:10,160
So the flush throttle timeout and the peer gossip sleep duration, basically both of those

162
00:09:10,160 --> 00:09:11,160
are reduced.

163
00:09:11,160 --> 00:09:15,160
So the peer gossip sleep duration probably works as explained.

164
00:09:15,160 --> 00:09:20,160
It's like between every kind of talk, we're having it wait less.

165
00:09:20,160 --> 00:09:23,160
It seems like the throttle timeout.

166
00:09:23,160 --> 00:09:28,160
Yeah, so flush throttle timeout, peer gossip sleep duration.

167
00:09:28,160 --> 00:09:29,160
Yeah.

168
00:09:29,160 --> 00:09:30,160
Yeah.

169
00:09:30,160 --> 00:09:33,160
And well, man, it works miracles.

170
00:09:33,160 --> 00:09:39,160
It's, you know, we're down to like what less than 100 misses.

171
00:09:39,160 --> 00:09:42,160
Is this anything to do with pego and all that shit?

172
00:09:42,160 --> 00:09:43,160
Or is this purely signing?

173
00:09:43,160 --> 00:09:45,160
No, this is just purely.

174
00:09:45,160 --> 00:09:47,160
Who gives a shit?

175
00:09:47,160 --> 00:09:49,160
I do, right?

176
00:09:49,160 --> 00:09:51,160
It's fucking vanity.

177
00:09:51,160 --> 00:09:56,160
This reason we have distributed systems, like the fact that you guys are 90% or 95% doesn't

178
00:09:56,160 --> 00:09:58,160
make a fucking bit of difference in the overall.

179
00:09:58,160 --> 00:09:59,160
Well, or anything else.

180
00:09:59,160 --> 00:10:02,160
But a lot of people are getting down to the 70s and 60s.

181
00:10:02,160 --> 00:10:06,160
So like, Jen, Jen, I mean, we were one of them and that was with the audience.

182
00:10:06,160 --> 00:10:07,160
I know it's.

183
00:10:07,160 --> 00:10:08,160
Yeah.

184
00:10:08,160 --> 00:10:14,160
So like even doing a lot of our best practices, we were going really low and even with most

185
00:10:14,160 --> 00:10:19,160
one in Horcrux, just so that's heading with still and D Jen Horcrux, everything within

186
00:10:19,160 --> 00:10:21,160
10 milliseconds round trip.

187
00:10:21,160 --> 00:10:23,160
Yeah, it was brutal.

188
00:10:23,160 --> 00:10:30,160
And with injective, like a lot of your delegations are based on certain performance metrics.

189
00:10:30,160 --> 00:10:32,160
One of them.

190
00:10:32,160 --> 00:10:39,160
And not only that though, like some operators were getting down to like the wire to getting

191
00:10:39,160 --> 00:10:42,160
slashed because the slash window is 50%.

192
00:10:42,160 --> 00:10:49,160
So, you know, if you're already missing a lot and then you have an issue with your node

193
00:10:49,160 --> 00:10:54,160
that makes you miss for a little bit longer, 100% and then you kind of stuffed.

194
00:10:54,160 --> 00:10:57,160
So yeah, so it was.

195
00:10:57,160 --> 00:11:02,160
What you just proved is that even though everybody makes delegation decisions off of these bullshit

196
00:11:02,160 --> 00:11:06,160
metrics, that that bullshit metric was completely changed by one small config change.

197
00:11:06,160 --> 00:11:07,160
Just one small.

198
00:11:07,160 --> 00:11:09,160
So did it happen?

199
00:11:09,160 --> 00:11:10,160
Anything to do with?

200
00:11:10,160 --> 00:11:13,160
Did it have anything to do with Schultz's ability to deliver this?

201
00:11:13,160 --> 00:11:14,160
No.

202
00:11:14,160 --> 00:11:15,160
Did did his signing ability?

203
00:11:15,160 --> 00:11:21,160
So this whole fucking bullshit, like that you totally just explain the whole situation.

204
00:11:21,160 --> 00:11:22,160
Right.

205
00:11:22,160 --> 00:11:23,160
Like all these.

206
00:11:23,160 --> 00:11:27,160
Deligators making decision off this fucking metric and this fucking metric is bullshit.

207
00:11:27,160 --> 00:11:29,160
Yeah, I mean, that's fair.

208
00:11:29,160 --> 00:11:30,160
Right.

209
00:11:30,160 --> 00:11:34,160
So I said it was like, like I made this one config change on the binary that the team made

210
00:11:34,160 --> 00:11:38,160
and now suddenly we're all at 95% and like, oh, look at, we're all able to sign.

211
00:11:38,160 --> 00:11:42,160
But is it, is it, does that, does the past reflect then Schultz's ability to delivery,

212
00:11:42,160 --> 00:11:47,160
to deliver nodes and RPC or to deliver validators in a secure manner at scale?

213
00:11:47,160 --> 00:11:49,160
And does that make him a worse validator?

214
00:11:49,160 --> 00:11:51,160
It's just fucking dumb, right?

215
00:11:51,160 --> 00:11:57,160
So I would say that like this was a, you know, a collective issue that needed to be solved

216
00:11:57,160 --> 00:12:03,160
because, you know, the more blocks you miss, you know, the more proposals you miss.

217
00:12:03,160 --> 00:12:07,160
And if you miss proposals that affects the chain and its ability to put through.

218
00:12:07,160 --> 00:12:09,160
I'm not saying it didn't need to be solved.

219
00:12:09,160 --> 00:12:14,160
I'm just saying there's so many fucking variables that just boiling it down to like some sort

220
00:12:14,160 --> 00:12:16,160
of signing thing is fucking dumb.

221
00:12:16,160 --> 00:12:20,160
Like it's the whole purpose of the whole purpose of the situation and the whole purpose of being

222
00:12:20,160 --> 00:12:24,160
able to do this is to be able to have a distributed structure that allows for failure.

223
00:12:24,160 --> 00:12:31,160
And in this situation, you had validators being measured against a metric that was totally out of their control.

224
00:12:31,160 --> 00:12:33,160
That's one config file change.

225
00:12:33,160 --> 00:12:36,160
And then suddenly like everybody's metrics changes, right?

226
00:12:36,160 --> 00:12:42,160
So the, well, what I said about the delegations was, I'd say it was a half truth.

227
00:12:42,160 --> 00:12:49,160
It is, it is a metric, but it was, was not enforced because of the blanket problem.

228
00:12:49,160 --> 00:12:51,160
Slashing window would have been forced, right?

229
00:12:51,160 --> 00:12:52,160
Yeah, slashing would have.

230
00:12:52,160 --> 00:12:59,160
But yeah, I mean, this is also, this is also a pretty specific, um, injective problem.

231
00:12:59,160 --> 00:13:04,160
Like not a lot of other chains have this absolutely prolific signing problem.

232
00:13:04,160 --> 00:13:09,160
And I've noticed that a lot of chains have gotten better since IAVL V1 as well.

233
00:13:09,160 --> 00:13:17,160
So, um, the, uh, DYDX was having a bit of a signing problem.

234
00:13:17,160 --> 00:13:22,160
Um, it probably wasn't as bad, but their window is only 8,000 blocks as well.

235
00:13:22,160 --> 00:13:26,160
So there were people who were sort of missing 1,500 blocks in there.

236
00:13:26,160 --> 00:13:33,160
And I noticed as soon as they upgraded, IAVL V1, it cleared up a lot of those as well.

237
00:13:33,160 --> 00:13:39,160
And man, the difference, it was kind of similar to when, um, you know, say DB was implemented,

238
00:13:39,160 --> 00:13:43,160
which I think is an implementation of IAVL V1.

239
00:13:43,160 --> 00:13:50,160
And like the abuse on your drives is night and day.

240
00:13:50,160 --> 00:13:51,160
I think so. Yeah.

241
00:13:51,160 --> 00:13:57,160
I think they took the upstream and sort of fast tracked it and made some other changes.

242
00:13:57,160 --> 00:14:05,160
Um, I think I'm not 100% sure, but the, the, um, the difference is night and day, uh,

243
00:14:05,160 --> 00:14:10,160
on all of the chains that I've seen that have changed IAVL V1.

244
00:14:10,160 --> 00:14:15,160
Um, the abuse on the drives just goes down astronomical, astronomical.

245
00:14:15,160 --> 00:14:20,160
Like the, I've got some bad drives around on some nodes, like, you know, they,

246
00:14:20,160 --> 00:14:22,160
they must be consumer drives and something.

247
00:14:22,160 --> 00:14:27,160
And the, um, the where levels on those were just astronomical, like 30% a month.

248
00:14:27,160 --> 00:14:33,160
And as soon as they switched over to IAVL, like basically the where level flat lined.

249
00:14:33,160 --> 00:14:38,160
So the amount of like reads and writes going on to those drives dramatically changed

250
00:14:38,160 --> 00:14:41,160
between the two software versions.

251
00:14:41,160 --> 00:14:43,160
I'm not sure if, um, you saw the same thing.

252
00:14:43,160 --> 00:14:45,160
She'll see with your nodes.

253
00:14:45,160 --> 00:14:48,160
I got rid of those nodes with those shitty drives, by the way,

254
00:14:48,160 --> 00:14:54,160
because they also had shitty networks and shitty uptime and cost the most out of all of my fucking nodes.

255
00:14:54,160 --> 00:14:56,160
Yeah. Yeah.

256
00:14:56,160 --> 00:14:57,160
Well, that's good.

257
00:14:57,160 --> 00:14:58,160
So the team in it.

258
00:14:58,160 --> 00:15:04,160
So did, did somebody spearhead that or that was at, was that the injective team that made that discovery?

259
00:15:04,160 --> 00:15:06,160
Or is that somebody else?

260
00:15:06,160 --> 00:15:11,160
The injective labs came up with the, the configuration change,

261
00:15:11,160 --> 00:15:19,160
but I think they have been messing with things for quite some time, um, to try and find like the smoking gun.

262
00:15:19,160 --> 00:15:20,160
Cool.

263
00:15:20,160 --> 00:15:24,160
And it appears as if they have found it.

264
00:15:24,160 --> 00:15:25,160
Yeah.

265
00:15:25,160 --> 00:15:26,160
I guess.

266
00:15:26,160 --> 00:15:27,160
Yeah.

267
00:15:27,160 --> 00:15:32,160
I mean, that, I was like, when they, when they asked me to test, I thought, oh, you know,

268
00:15:32,160 --> 00:15:37,160
it might be like a 20% improvement or something, you know, it's just like someone turned the lights on.

269
00:15:37,160 --> 00:15:38,160
Yeah.

270
00:15:38,160 --> 00:15:40,160
It was kind of like, that was like, say DB, right?

271
00:15:40,160 --> 00:15:42,160
Like that was totally different.

272
00:15:42,160 --> 00:15:43,160
Yeah.

273
00:15:43,160 --> 00:15:45,160
It's just completely, completely, completely.

274
00:15:45,160 --> 00:15:52,160
I think like what hardware we choose, like prunings, we had no pruning and like just the state sync structure and like all this shit to,

275
00:15:52,160 --> 00:15:55,160
to get those nodes to do what we want to, and then all kind of just disappeared.

276
00:15:55,160 --> 00:15:56,160
Right.

277
00:15:56,160 --> 00:15:58,160
Now we're doing different issues, but whatever.

278
00:15:58,160 --> 00:15:59,160
Yeah.

279
00:15:59,160 --> 00:16:01,160
Different issues, but similar outcome.

280
00:16:01,160 --> 00:16:08,160
And then just some small change or in say instances, probably quite a big change and a lot of engineering.

281
00:16:08,160 --> 00:16:09,160
Yeah.

282
00:16:09,160 --> 00:16:10,160
Yeah.

283
00:16:10,160 --> 00:16:14,160
But you know, a change and then just makes things so much, so much better.

284
00:16:14,160 --> 00:16:19,160
But I guess that's, you know, that's the iterative nature of these things, right?

285
00:16:19,160 --> 00:16:20,160
Yeah.

286
00:16:20,160 --> 00:16:23,160
You, you have a problem, you work on it, you fix it, you tackle the next one.

287
00:16:23,160 --> 00:16:24,160
Yeah.

288
00:16:24,160 --> 00:16:29,160
I think I think the challenge with, and part of the part that gives me stress with that is that, that, that

289
00:16:29,160 --> 00:16:34,160
realization where, hey, this is all in motion and all that kind of stuff is, is totally true.

290
00:16:34,160 --> 00:16:35,160
Right.

291
00:16:35,160 --> 00:16:37,160
Like it doesn't, it's not perfect.

292
00:16:37,160 --> 00:16:39,160
We find all these different issues.

293
00:16:39,160 --> 00:16:45,160
There's a so many moving parts and like so many things that are changing and you find a new avenues.

294
00:16:45,160 --> 00:16:47,160
Like I'm sure we're going to talk about with this part thing, right?

295
00:16:47,160 --> 00:16:53,160
You're finding new avenues, but at the same time, there's an expectation like this is totally going to work.

296
00:16:53,160 --> 00:16:59,160
And like, like, why can't we support this and why does, why is this issue happening and we're in main net or something

297
00:16:59,160 --> 00:17:00,160
else similar to that.

298
00:17:00,160 --> 00:17:07,160
And I think that idea that this is, there's an incredible amount of untested software that, that new issues are

299
00:17:07,160 --> 00:17:14,160
discovered every day as a note op kind of comes down to the note op, not necessarily the team and say, maybe

300
00:17:14,160 --> 00:17:18,160
not 100% sometimes or from a provider perspective.

301
00:17:18,160 --> 00:17:28,160
And, and it's tough to be able to, it's tough to be able to guarantee a specific SLA associated to a service when the

302
00:17:28,160 --> 00:17:35,160
underlying software to support that service may not give you at all the tools you need to be able to deliver that.

303
00:17:35,160 --> 00:17:36,160
You know what I mean?

304
00:17:36,160 --> 00:17:41,160
Which is, which could be RPC, it could be a validator service like we're talking about the injective could be anything.

305
00:17:41,160 --> 00:17:46,160
But it's tough to be able to be in that position where you're trying to troubleshoot the best you can, but underlying

306
00:17:46,160 --> 00:17:49,160
or something just incredibly broken, right?

307
00:17:49,160 --> 00:17:52,160
Or just needs to be, that needs to be pushed forward.

308
00:17:52,160 --> 00:17:53,160
Yeah.

309
00:17:53,160 --> 00:18:02,160
And I, but I think like, in terms of RPC specifically, like, I think the teams that you're providing it to have probably the

310
00:18:02,160 --> 00:18:09,160
best idea of the underlying issues in the chain that they're working on ends.

311
00:18:09,160 --> 00:18:15,160
Like, I've never had problems with, you know, we have certain SLA conditions and no one has ever tried to

312
00:18:15,160 --> 00:18:21,160
enforce one of those conditions based on a problem with the chain software.

313
00:18:21,160 --> 00:18:23,160
It's more like a clause of negligence.

314
00:18:23,160 --> 00:18:31,160
If you just allow your nodes to continue to have issues when there's no real underlying problem.

315
00:18:31,160 --> 00:18:41,160
But we, we have to, in our contracts, write out issues of the base software because we have absolutely no control over that whatsoever.

316
00:18:41,160 --> 00:18:42,160
Yeah, that's changed.

317
00:18:42,160 --> 00:18:49,160
Well, but I think that that falls back to what Eric, you, you said we were saying before in that, yes, that's true for other entities,

318
00:18:49,160 --> 00:18:55,160
but slashing is an example of a contract that can't always be totally upheld, right?

319
00:18:55,160 --> 00:18:56,160
Like Pega.

320
00:18:56,160 --> 00:18:57,160
Yeah.

321
00:18:57,160 --> 00:19:03,160
That's in effect a blockchain enforced contract that is very difficult to truly adhere to.

322
00:19:03,160 --> 00:19:04,160
Yeah.

323
00:19:04,160 --> 00:19:06,160
So I will say one thing about Pega.

324
00:19:06,160 --> 00:19:14,160
It is, well, it was more so a huge burden of like it was a pain to, to deal with Pega, right?

325
00:19:14,160 --> 00:19:24,160
In that, you know, there were software problems and there were any number of things that made it difficult to deal with.

326
00:19:24,160 --> 00:19:30,160
And the slashing was astronomical, especially if you had a lot of delegations that were not, you know,

327
00:19:30,160 --> 00:19:38,160
people validate as we're getting slashed while their delegators were getting slashed, you know, 100,000 US dollars in one instance.

328
00:19:38,160 --> 00:19:42,160
And you only have two hours to recognize and fix a problem.

329
00:19:42,160 --> 00:19:46,160
We'll had two hours to recognize and fix the problem, right?

330
00:19:46,160 --> 00:19:54,160
And, but in saying that, like it was possible, if you had a lot of people who were not able to do that,

331
00:19:54,160 --> 00:20:07,160
but in saying that, like it was possible, if you had, you know, on call people and you had fucking like just systems of annoyance to keep you up to date with it.

332
00:20:07,160 --> 00:20:16,160
So, you know, but, but it was also very easy to get slashed if the conditions were right.

333
00:20:16,160 --> 00:20:17,160
Yeah.

334
00:20:17,160 --> 00:20:19,160
But we've said that software is a piece of shit, right?

335
00:20:19,160 --> 00:20:20,160
Well, yeah.

336
00:20:20,160 --> 00:20:23,160
It seems to have been improved a lot recently.

337
00:20:23,160 --> 00:20:24,160
Okay.

338
00:20:24,160 --> 00:20:25,160
I mean, you're in that cartel.

339
00:20:25,160 --> 00:20:26,160
So I can understand.

340
00:20:26,160 --> 00:20:27,160
I'm not in that cartel.

341
00:20:27,160 --> 00:20:28,160
So I can say whatever I want.

342
00:20:28,160 --> 00:20:32,160
But I, from what I've heard, from what I've heard, it's been a real fucking bitch.

343
00:20:32,160 --> 00:20:34,160
And it's not the greatest piece of software.

344
00:20:34,160 --> 00:20:38,160
And then number two, you're held to a standard associated to that software.

345
00:20:38,160 --> 00:20:44,160
And there's a huge amount of learning curve to understand the intricacies of how to run that successfully.

346
00:20:44,160 --> 00:20:45,160
Right.

347
00:20:45,160 --> 00:20:46,160
And you guys learn that through testnet, right?

348
00:20:46,160 --> 00:20:51,160
And other types of situations like you, you learn how this, this thing fucking works, right?

349
00:20:51,160 --> 00:20:52,160
Well, no.

350
00:20:52,160 --> 00:20:56,160
That's, that's kind of the fun part of Pego as well is that the testnet is closed.

351
00:20:56,160 --> 00:21:02,160
So if you're going to join in on mainnet, most people who joined mainnet immediately get slashed on Pego.

352
00:21:02,160 --> 00:21:03,160
Right.

353
00:21:03,160 --> 00:21:04,160
That's my point.

354
00:21:04,160 --> 00:21:08,160
The point is the learning curve is so high in this thing.

355
00:21:08,160 --> 00:21:11,160
And it's not like the most like, it's not a great piece.

356
00:21:11,160 --> 00:21:13,160
Can we agree that it's not a great piece of software?

357
00:21:13,160 --> 00:21:15,160
Like it's not the best thing that's ever been written, right?

358
00:21:15,160 --> 00:21:16,160
It's somewhere in between.

359
00:21:16,160 --> 00:21:21,160
There, there are conditions where it just files to operate.

360
00:21:21,160 --> 00:21:27,160
So that, that would probably cost the most current answer I've ever heard of my life.

361
00:21:27,160 --> 00:21:30,160
It's like, it's definitely not the best, but it's, yeah, it's better.

362
00:21:30,160 --> 00:21:31,160
Okay.

363
00:21:31,160 --> 00:21:32,160
Okay.

364
00:21:32,160 --> 00:21:33,160
It's better.

365
00:21:33,160 --> 00:21:34,160
Better.

366
00:21:34,160 --> 00:21:35,160
It's better.

367
00:21:35,160 --> 00:21:36,160
It's the other way.

368
00:21:36,160 --> 00:21:37,160
It's, you guys, I understand it's better.

369
00:21:37,160 --> 00:21:39,160
It's, it's, it is better.

370
00:21:39,160 --> 00:21:40,160
Right.

371
00:21:40,160 --> 00:21:41,160
We're building.

372
00:21:41,160 --> 00:21:42,160
Yeah.

373
00:21:42,160 --> 00:21:50,160
But, but again, you guys, you're being held to a standard on something that is difficult to understand and, and has challenges to it that

374
00:21:50,160 --> 00:21:52,160
require a high amount of learning curve.

375
00:21:52,160 --> 00:21:59,160
And like to some point, like if you come into Validator today and in Kujira and somebody puts on a million dollars to get in that set and does whatever else.

376
00:21:59,160 --> 00:22:03,160
And they get slashed in the first two because of the stupid fucking thing.

377
00:22:03,160 --> 00:22:14,160
It's not like, is that where, is that why we're here for, for, for literally for this software that is run by a hundred other fucking entities that if one is down, it doesn't make any difference.

378
00:22:14,160 --> 00:22:16,160
Like, what the fuck are we doing with that?

379
00:22:16,160 --> 00:22:17,160
Like, I don't understand.

380
00:22:17,160 --> 00:22:20,160
Like, is that really the need?

381
00:22:20,160 --> 00:22:26,160
My only slashing is from the fucking inject from the Kujira.

382
00:22:26,160 --> 00:22:27,160
Oracle.

383
00:22:27,160 --> 00:22:29,160
No, it's from Kujira.

384
00:22:29,160 --> 00:22:33,160
I'm very fucking traumatized by that experience.

385
00:22:33,160 --> 00:22:34,160
I know, right.

386
00:22:34,160 --> 00:22:45,160
But I'm just saying, like, like, I just don't, I don't understand these standards that are built around executing on substandard software for a reason that has to do with the

387
00:22:45,160 --> 00:22:52,160
for a reason that has no fiscal or health capable like need.

388
00:22:52,160 --> 00:22:54,160
Like the chain will still run those.

389
00:22:54,160 --> 00:22:58,160
Well, I guess I guess you have to have something right.

390
00:22:58,160 --> 00:23:01,160
You can't just say, well, we don't give a fuck about metrics at all.

391
00:23:01,160 --> 00:23:06,160
Like you have, it keeps the, it keeps the operators honest in some respect in that.

392
00:23:06,160 --> 00:23:07,160
At least the modest.

393
00:23:07,160 --> 00:23:09,160
I'm not saying black and white.

394
00:23:09,160 --> 00:23:12,160
Like there's, there's, there's levels of gray, right?

395
00:23:12,160 --> 00:23:20,160
But, but like to say that like, hey, you know, you have this two hour slashing window or you lose, you lose $200,000 because the software is a piece of shit.

396
00:23:20,160 --> 00:23:24,160
There's something between that and like not having any controls, right?

397
00:23:24,160 --> 00:23:33,160
Like, like, I don't understand where the, where the, like where we, where these teams lose their sense of like, understand, even like we've seen like slashing windows at 100 blocks and stuff like that.

398
00:23:33,160 --> 00:23:34,160
Like, what the fuck?

399
00:23:34,160 --> 00:23:38,160
Like, well, that's just because that was the default in the fucking SDK.

400
00:23:38,160 --> 00:23:39,160
I know, I know.

401
00:23:39,160 --> 00:23:40,160
That's different.

402
00:23:40,160 --> 00:23:59,160
But like, in that same situation, like, like, like, this is not, it's not a, like you're treating, like treating this, this ecosystem, like it's 100% figured out and like, oh yeah, we stole the global internet of servers that communicate each other and do hashes associated to blocks and

403
00:23:59,160 --> 00:24:06,160
like this incredibly, and IBC and like all these wasm contracts and calculations and all like this incredibly complicated thing.

404
00:24:06,160 --> 00:24:24,160
And, and if you look back, if you peel back the covers associated to these SLAs, right around, you said, you think you pull back the covers and you actually look at what's running underneath that and you go, holy shit, like there's, there's a lot here that, that has a long way to go to be clearly this like enterprise ready.

405
00:24:24,160 --> 00:24:28,160
I hate using that word, but some sort of like, we can put this SLA associate to it.

406
00:24:28,160 --> 00:24:33,160
And so like things like that, like slashing windows associated to Pego just, I just don't understand that stuff.

407
00:24:33,160 --> 00:24:36,160
It seems like it's, seems like it's way too tight.

408
00:24:36,160 --> 00:24:47,160
I don't know. I'm kind of on the fence, man. Like, I, some, some windows, some standards that you've been held to are quite, you know, strict, but that's the sandbox we choose to play in, right?

409
00:24:47,160 --> 00:24:54,160
Like if you know, no, no, no, it's not, it is, you don't, there's no one fucking holding a gun to your head and saying go validate on injective.

410
00:24:54,160 --> 00:25:03,160
Like that's the sandbox you choose to play in. If, if you want to play in that sandbox and for example, offer a, you know, a slash.

411
00:25:03,160 --> 00:25:09,160
No, well, you know, it's for, for the, for the what? It's a business, right? It's for the revenue.

412
00:25:09,160 --> 00:25:13,160
No, no, no, no, no, but like, but what is that SLA enforcing?

413
00:25:13,160 --> 00:25:20,160
Like that, what you just said, like, like you're, like if you want to hop into sandbox that for the challenge, but the challenge of what?

414
00:25:20,160 --> 00:25:31,160
I'm not saying for the challenge. I'm saying that's, it's a known, it's a known quantity. Like if you, if you want to play in that sandbox, you know the risks and you know the challenge.

415
00:25:31,160 --> 00:25:46,160
So go play in it or don't like there's no one holding a gun to your head. So, or you could, you know, you could go and drum up some support and validate the hub where you'll never miss a block ever because there's six seconds and don't do anything.

416
00:25:46,160 --> 00:25:56,160
So it's not that. It's just like, it just seems like the, some of the stuff is, is engineered around expectations that are unrealistic, right?

417
00:25:56,160 --> 00:26:10,160
For, for the, the level of quality and the level of complexity that's associated to it. And it's, it's literally built to be resilient. Like that's the whole point.

418
00:26:10,160 --> 00:26:18,160
But it's all, it's all beta software, right? But there, there are issues associated with like poor performing operators.

419
00:26:18,160 --> 00:26:24,160
And again, it's not black and white. Like I understand that. Like, like, but a hundred blocks, like two hours to get slashed.

420
00:26:24,160 --> 00:26:31,160
And that's obviously fucking ridiculous. That's just a lapse in people losing. Like if somebody's like coming in from outside, you could get slashed while you were taking a shit.

421
00:26:31,160 --> 00:26:39,160
Yeah, seriously, take it to our shits. Like, but the idea like if I, if I, if I'm, if I'm, that's a fucking different problem.

422
00:26:39,160 --> 00:26:53,160
But if I, if I'm entering the Seco system and I'm learning all this shit and I lose money because, because null, null had a very minor incident associated to Pego and took a two hour shit.

423
00:26:53,160 --> 00:27:00,160
And they get slash because of it. That's fucking stupid. Like it doesn't like, what are we, like, what is that? Like what?

424
00:27:00,160 --> 00:27:03,160
I just don't understand. Again, not black and white.

425
00:27:03,160 --> 00:27:09,160
If I was on call and the alarm went off while I was taking a shit, I would snap one off and I'd go and fix it.

426
00:27:09,160 --> 00:27:11,160
Just pinch it.

427
00:27:11,160 --> 00:27:13,160
I'd pinch it and fix it.

428
00:27:17,160 --> 00:27:23,160
And I've had to do that before. Sometimes I've taken a laptop to the shitter with me. So.

429
00:27:23,160 --> 00:27:25,160
Oh yeah, I've done that.

430
00:27:25,160 --> 00:27:27,160
Yeah. When you're on call, you're on call.

431
00:27:27,160 --> 00:27:29,160
Not like it's not on call.

432
00:27:29,160 --> 00:27:35,160
I'm not in preparation of something happening just like while something's happening. Yeah, for sure.

433
00:27:35,160 --> 00:27:39,160
I've had that. I mean, yeah, you gotta do what you gotta do.

434
00:27:39,160 --> 00:27:43,160
I've even thought about, you know, making a, getting my woodworking skills out.

435
00:27:43,160 --> 00:27:45,160
Get a little stand.

436
00:27:45,160 --> 00:27:47,160
Get a little stand.

437
00:27:47,160 --> 00:27:51,160
Just pull that thing up here.

438
00:27:51,160 --> 00:27:57,160
Pretty handy with, with some, you know, plywood and some pocket screws.

439
00:27:57,160 --> 00:28:01,160
Perfect. How you get, you get the, you put little steps on the bottom, get your feet up so your legs don't fall asleep and shit.

440
00:28:01,160 --> 00:28:03,160
Like you do something nice.

441
00:28:03,160 --> 00:28:09,160
Well, you know, it's like scientifically proven that you shit better when you've got your knees on it.

442
00:28:09,160 --> 00:28:17,160
I know. I know. That's the, do they have the, here in the States, they, they, there's something called the squatty potty.

443
00:28:17,160 --> 00:28:19,160
Do they have that? Have you heard of that thing?

444
00:28:19,160 --> 00:28:23,160
I have not heard of that thing.

445
00:28:23,160 --> 00:28:27,160
I don't even know. It was probably like, I don't know, it's gotta be 15 years ago or something, but I had a look at it.

446
00:28:27,160 --> 00:28:31,160
It's called the squatty potty. It's just a piece of plastic that sits around the toilet.

447
00:28:31,160 --> 00:28:37,160
There's nothing crazy about it, but it does the same thing where it like lifts your legs up and gets your knees basically at or above.

448
00:28:37,160 --> 00:28:43,160
Yeah. And then, and then I think like it was, I don't know where I heard about me was Howard Stern, maybe back in the day or something like that.

449
00:28:43,160 --> 00:28:47,160
It was a radio jock here in the US.

450
00:28:47,160 --> 00:28:55,160
It's been around for 30 years or something or more. And he talked about it a little bit and then, and then I'm going to have to, I'm going to have to stop you there.

451
00:28:55,160 --> 00:29:09,160
Cause there is a website called squatty potty Australia and, and they are well, I guess this is a testament to the product in that their website says we'll be back soon.

452
00:29:09,160 --> 00:29:16,160
Sorry for the inconvenience. I guess they are not on call because they are probably enjoying taking their shit for four hours.

453
00:29:16,160 --> 00:29:20,160
They might be. They might be. Yeah. Sorry. Continue your Howard. No, that was it. That was it.

454
00:29:20,160 --> 00:29:32,160
So anyway, you could build that right into that. Maybe, maybe that's a product idea. Squatty potty that combined with like a laptop stand, have a little like spot for, I don't know, for, for the dev up elite.

455
00:29:32,160 --> 00:29:40,160
With a little UPS in there.

456
00:29:40,160 --> 00:29:54,160
A little coffee cup holder. What else do you need in there? Yeah, I can sell them to the Google H. Yeah, you get like a little separate like nine inch screen with like some Grafana or like Xabax dashboards in there that automatically scroll through just the problems dashboards up.

457
00:29:54,160 --> 00:30:08,160
Just two, two 20 inch monitors, a little slot for your like a docking slot for MacBook Pros only. Cause yeah, and it's always updated to be only compatible with the latest model, a little UPS battery.

458
00:30:08,160 --> 00:30:24,160
5G backup. Yeah. One of those little white little, you know, extenders for the 5G. A little Starlink backup. Yeah.

459
00:30:24,160 --> 00:30:35,160
And you could sell like a little camp aspect of that too. So if you need to take it on the road, you can just kind of bring that thing out there and dig a hole or something if you need to dig a hole. Set it up.

460
00:30:35,160 --> 00:30:43,160
Exactly. Sunshade for you. So what's been going on with your, your say shit, man.

461
00:30:43,160 --> 00:30:51,160
So we've been dealing with, this is on Arctic one, which is the say EVM B2 chain, which has been going really well actually.

462
00:30:51,160 --> 00:30:53,160
And then an area.

463
00:30:53,160 --> 00:31:08,160
Yeah. Well, well that, that, so that that EVM stuff just moved to Atlantic two that that last upgrade that 501 that was all the EVM stuff coming into Atlantic two, which is why it jumped from like B3 to B5, I guess is why, why that happened.

464
00:31:08,160 --> 00:31:22,160
And so on Arctic one, we have three nodes that are sitting behind a couple balancers and and we've been having this issue where basically, which we saw before, which was that nodes, not necessarily at the same time, but they would eventually

465
00:31:22,160 --> 00:31:34,160
keep growing in RAM, like they would be fine, like, like two or three gigs and they would eventually get to like, at some point, I put a cap on it, but at some point they would run the RAM to the point where the box would stop responding.

466
00:31:34,160 --> 00:31:36,160
Like it would completely, it wouldn't.

467
00:31:36,160 --> 00:31:37,160
It wouldn't.

468
00:31:37,160 --> 00:31:38,160
It wouldn't.

469
00:31:38,160 --> 00:31:43,160
It would lock the box where I couldn't do anything like it would like 100% right.

470
00:31:43,160 --> 00:31:45,160
Yeah, it wouldn't do anything.

471
00:31:45,160 --> 00:32:03,160
And so I ended up putting some system decontrols in there, which I need to do actually, I need to do that overall, which, which is just just put a memory cap on every single one and I think I'm going to do that in Antswell where I'll take basically, maybe a look at the total amount of RAM in the box and just cut it to say if it takes half of this and kill it, right,

472
00:32:03,160 --> 00:32:07,160
is something like that because usually my box is like all the stuff should have run is probably 128 gigs.

473
00:32:07,160 --> 00:32:12,160
So if you hit 64 gigs, something's fucking wrong on a cosmos change for sure.

474
00:32:12,160 --> 00:32:17,160
And you can't like you can't is there can you use a percentage in that?

475
00:32:17,160 --> 00:32:18,160
Yeah, sure.

476
00:32:18,160 --> 00:32:20,160
Instead of actually having used like a hard number.

477
00:32:20,160 --> 00:32:21,160
Yeah, yeah.

478
00:32:21,160 --> 00:32:30,160
So in Antswell, I would just take I would take the number because that's an it's an Ansible thing that comes in through through get fax and that was just divided by two and just say if it hits that kill it.

479
00:32:30,160 --> 00:32:31,160
So I'm going to do that.

480
00:32:31,160 --> 00:32:41,160
I don't get fax though when well, you know, with my it's why I have get that sound of I do whatever I need because I because I do because right now also what's really fucking this is totally wrong really.

481
00:32:41,160 --> 00:32:42,160
But I usually run Dev 12.

482
00:32:42,160 --> 00:32:43,160
I run Debbie and everywhere.

483
00:32:43,160 --> 00:32:52,160
And then I've been using so many different providers and different things and like Debbie and like not everybody supports and some people just use a bun to and I'm not a huge a bun to fan, but whatever.

484
00:32:52,160 --> 00:32:53,160
What do you have?

485
00:32:53,160 --> 00:33:00,160
I've noticed that you you run Dev 12 like over bun to what's your what's your grub and you bunting it.

486
00:33:00,160 --> 00:33:01,160
I'm over 50 and.

487
00:33:01,160 --> 00:33:16,160
I was born I don't like it pierced pierced no, I it's it's all Debbie and based and I've always I grew up with Debbie and and that was been my thing.

488
00:33:16,160 --> 00:33:18,160
I ran like I mean early days in Linux.

489
00:33:18,160 --> 00:33:24,160
I ran like the whole I went through the whole jet to fucking thing and then I ran all stuff within red hat.

490
00:33:24,160 --> 00:33:25,160
I read about a ton of red hats.

491
00:33:25,160 --> 00:33:27,160
I know I know the young structure really well.

492
00:33:27,160 --> 00:33:32,160
I had an arch period which I ran arch for a long time, but I've always had Debbie.

493
00:33:32,160 --> 00:33:33,160
I've always had Debbie.

494
00:33:33,160 --> 00:33:35,160
So I don't like that.

495
00:33:35,160 --> 00:33:40,160
I know that so I use 22 I for Ubuntu.

496
00:33:40,160 --> 00:33:43,160
It's the same thing, but for some reason.

497
00:33:43,160 --> 00:33:48,160
Yeah, well I is is Dev 12 ship with 6.1.

498
00:33:48,160 --> 00:33:49,160
Does it?

499
00:33:49,160 --> 00:33:53,160
It does the chip is six, but though though all my bunch of stuff I pushed a six anyway.

500
00:33:53,160 --> 00:33:55,160
I use whatever we do package.

501
00:33:55,160 --> 00:34:00,160
What's that package for it basically it's I'll tell you what it is.

502
00:34:00,160 --> 00:34:08,160
Oh, there's like a package that puts all the okay because well I it puts it Linux kind of so much better dude.

503
00:34:08,160 --> 00:34:11,160
It's like it's just I still use kernel six.

504
00:34:11,160 --> 00:34:13,160
I just build it every time.

505
00:34:13,160 --> 00:34:18,160
So I install I'm installed Ubuntu 20 to 04 and then I just upgrade the kernel.

506
00:34:18,160 --> 00:34:23,160
What do you do like like manually?

507
00:34:23,160 --> 00:34:24,160
What do I do?

508
00:34:24,160 --> 00:34:26,160
Well, no, there's a package you can install.

509
00:34:26,160 --> 00:34:28,160
Oh, yeah.

510
00:34:28,160 --> 00:34:32,160
Yeah, sometimes sometimes I build it.

511
00:34:32,160 --> 00:34:41,160
This one is what the fuck is this called Linux generic hwe dash and Ansible distribution version.

512
00:34:41,160 --> 00:34:45,160
So that'd be Linux generic hwe dash 20 to 04.

513
00:34:45,160 --> 00:34:46,160
I think is one of this.

514
00:34:46,160 --> 00:34:51,160
That's what that package is called, but that basically brings in the hwe version of that kernel for Ubuntu.

515
00:34:51,160 --> 00:34:53,160
What the fuck are we talking about this anyway?

516
00:34:53,160 --> 00:34:54,160
Why are we talking about Ubuntu?

517
00:34:54,160 --> 00:34:56,160
How are we in this situation?

518
00:34:56,160 --> 00:34:58,160
You said you're a dev 12 boy.

519
00:34:58,160 --> 00:35:00,160
Yeah, but what does that do with fucking say?

520
00:35:00,160 --> 00:35:03,160
Where do we how do we get down this path?

521
00:35:03,160 --> 00:35:09,160
Back to back to say anyway, you had a problem anyway.

522
00:35:09,160 --> 00:35:10,160
Anyway, what happened man?

523
00:35:10,160 --> 00:35:11,160
Where am I?

524
00:35:11,160 --> 00:35:12,160
This is the story of my life.

525
00:35:12,160 --> 00:35:16,160
Like we started like here and then we ended up like down the road.

526
00:35:16,160 --> 00:35:22,160
Anyway, back to say so we're so we do we have a few nodes that are running on say for artic.

527
00:35:22,160 --> 00:35:26,160
For artic one, which is the EVM type of structure.

528
00:35:26,160 --> 00:35:40,160
And what's been happening is that not at the same time, but over a period of time, we're starting to see that the RAM usage basically shoots up like two or three gigs and it'll shoot up to like 1012 and then it'll continually grow.

529
00:35:40,160 --> 00:35:46,160
Oh, you're talking about RAM controls and then it'll continually grow and then I put in the lamp controls and basically at 40 or 50 gigs.

530
00:35:46,160 --> 00:36:01,160
It starts to slow it and then it'll eventually oh and but before that point once it gets to 4050 gigs what ends up happening is that the stop the note stops responding it stops to it can't keep up with the block RPC stuff shit shut off all this kind of shit right.

531
00:36:01,160 --> 00:36:14,160
And so we're trying to figure out what the fuck is causing this because it's clearly something related to we get a lot of volume on our artic one like we have a significant number of RPC requests.

532
00:36:14,160 --> 00:36:19,160
The say team, their nodes, they haven't seen this but they don't get the volume that we do.

533
00:36:19,160 --> 00:36:23,160
We probably get, I don't even know how much volume they get, but we get we get a decent amount of volume on our artic one.

534
00:36:23,160 --> 00:36:39,160
And so, and so they, I gave them I gave them access to a box and then they've been running a couple of different modifier binaries they've been doing a lot of like core dumps and like other types of peeper off type stuff and like trying to identify what's going on.

535
00:36:39,160 --> 00:36:49,160
And then we had the identified an issue which might still be an issue but it was a separate issue that that on the Web socket on EVM side basically it wasn't like giving up.

536
00:36:49,160 --> 00:37:01,160
I feel exactly how they call it but it wasn't it was creating a socket for every connection that wasn't reusing sockets and some sort of thing where it might have been chewing up ram and it was like doing whatever.

537
00:37:01,160 --> 00:37:17,160
So we did a patch on that and then that was so we spent, excuse me, but spent a bunch of times a couple days ago doing a patch on that and then just kind of going through like what what the issues are and like it only happens every like three or four hours so it's really it's really

538
00:37:17,160 --> 00:37:22,160
difficult to you know, troubleshoot when everything takes three hours between kind of runs.

539
00:37:22,160 --> 00:37:31,160
And so we get through all that other than else and then yesterday or this morning actually we started getting we started getting all of these.

540
00:37:31,160 --> 00:37:35,160
When that was happening we'd restate stinking node and then we were getting app hashes.

541
00:37:35,160 --> 00:37:43,160
And so it like app hash, like after a point and then one time like all my notes app hash or they would all app hash at the same time or similar to the same time.

542
00:37:43,160 --> 00:37:52,160
We're like, what the hell is this now. And so then we got a call is we're going to call for like three or four hours today trying to figure out just focus on the app hash because without the app hash you couldn't solve the other issue.

543
00:37:52,160 --> 00:38:01,160
And we started to realize that the networks were so pretty small and so we're pulling snapshots from like the same couple nodes and one of those nodes is theirs.

544
00:38:01,160 --> 00:38:07,160
And for some reason we determined when I get a snapshot from them it doesn't have all the wasm files.

545
00:38:07,160 --> 00:38:19,160
So there might be 70 wasm files and I get 60 and the other 10 don't have the dot wasm on the name they're just the they're just the like the hash without a dot wasm.

546
00:38:19,160 --> 00:38:25,160
But the snapshot only gives me the dot wasm so I was missing those and then obviously it had passed.

547
00:38:25,160 --> 00:38:30,160
So we're trying to figure out a way to so we couldn't figure out why that thing was named that way and we try to figure that out.

548
00:38:30,160 --> 00:38:32,160
We took us a long time to figure that out.

549
00:38:32,160 --> 00:38:37,160
And then we got to the number one like make sure we like understanding all that and I couldn't get another stint sig node.

550
00:38:37,160 --> 00:38:39,160
It was fucking weird dude.

551
00:38:39,160 --> 00:38:43,160
So so you were sorry you were getting snapshots or you're getting state sync.

552
00:38:43,160 --> 00:38:54,160
I'm sorry they say snapshots stacing we're doing state sinks and then what state sync it wouldn't give me all the wasm because the state sync process would ignore files that weren't dot wasm files.

553
00:38:54,160 --> 00:39:05,160
So the wasm files were in there but those nodes weren't were an app hashing so it would actually run against that wasm file even though it didn't have the dot wasm name on it just had it just had a hash name on it.

554
00:39:05,160 --> 00:39:10,160
And then out but those would never get delivered in the state sync so I would have left of the cop I would have less of those.

555
00:39:10,160 --> 00:39:21,160
So every time I was trying to troubleshoot this RAM issue I would state sync and then then the contract would call node app hashing so it was like all this shit at one time trying to figure out like.

556
00:39:21,160 --> 00:39:32,160
It took us a while to even figure out there was your two separate issues number one because you look at it you say oh this like the fact that the RAM goes up or it apaches that could be the same thing it might be just a different instantiation of that thing right.

557
00:39:32,160 --> 00:39:40,160
And so so that the apache issue we traced back down to a specific say no that I was calling from an RPC perspective.

558
00:39:40,160 --> 00:39:48,160
But it was actually a center I was calling a different say no for RPC data but it was pulling snapshots from one say server that had snapshots.

559
00:39:48,160 --> 00:39:56,160
And then for some reason it was always pulling from that same server and I couldn't get rid of the address book like all this fucking shit dude.

560
00:39:56,160 --> 00:40:09,160
And so and then we figured out that this was the issue so they're going back to confio on the question around that because I'm not sure if that's related to the upgrade that they just went through or something else but it's really weird that that wasm was in there.

561
00:40:09,160 --> 00:40:15,160
So anyway so we figured that out I think that might be solved because we know what the source is and then.

562
00:40:15,160 --> 00:40:32,160
Yeah that kind of sounds like they changed convention at some point and then it does right and then change the state sync at some point to only draw in things that conform to the new convention and then there's just like chaff sitting in there from previous stuff which is.

563
00:40:32,160 --> 00:40:36,160
But but also I looked at that node that node they stay synced early this morning.

564
00:40:36,160 --> 00:40:46,160
And their state sync script I'll send you to copy the script like they wrote dude they wrote a huge pay in new state trip so what they do is which I thought was actually pretty cool.

565
00:40:46,160 --> 00:41:04,160
They saved the private key out they saved the node key out they completely delete the whole say directory 100% they dump it all they re in it with the moniker name they save the moniker name they re in it with the moniker name they move the keys back in and then they run the state sync against a different node.

566
00:41:04,160 --> 00:41:17,160
Why it's totally new can pave I don't know but but to that point I was like oh maybe you guys are like that's what I thought I'm like well maybe you guys are not deleting the wasm directory or you like the cash is still there or something else in that they send me the script was like holy shit like.

567
00:41:17,160 --> 00:41:22,160
I guess it's so that maybe it's just so they are they're always on the latest config.

568
00:41:22,160 --> 00:41:34,160
You know that's what I thought too and and because we do run into that issue because like say moves fast like with say DB or with this with the what's the other thing that just happened.

569
00:41:34,160 --> 00:41:37,160
The.

570
00:41:37,160 --> 00:41:47,160
The concurrency thing that that that concurrency thing where they run transactions concurrency maybe maybe that maybe you guys can see that but that's an Arctic parallelization.

571
00:41:47,160 --> 00:41:50,160
Thank you yeah that was parallel.

572
00:41:50,160 --> 00:41:57,160
Power is it only that's only be able to think right yeah so that's a new config section.

573
00:41:57,160 --> 00:42:10,160
So what I've been doing is like I basically been when I do those things I basically go back and run ansible to delete that whole node which is basically the same thing and then re in it and then reapply right a bunch of shit.

574
00:42:10,160 --> 00:42:17,160
And state thinking I mean that seems like a little bit much for state thinking because I think I would just manually do it but but yeah.

575
00:42:17,160 --> 00:42:26,160
Yeah like I can kind of see but it you know you don't only you don't need to do the new can pave I would think if you had like an upgrade that you wanted to test or something but.

576
00:42:26,160 --> 00:42:32,160
Or like yeah config changes I just thought like because that because.

577
00:42:32,160 --> 00:42:36,160
It was I thought it was interesting because it's cool I think it's totally fine.

578
00:42:36,160 --> 00:42:43,160
I thought it was interesting because we were talking about just the idea of having old shit in there at the same time like they completely.

579
00:42:43,160 --> 00:42:49,160
Like oh yeah there's clearly no old shit in there like it's completely wiping the whole thing out so on on.

580
00:42:49,160 --> 00:42:59,160
You know they might do it because of there's an issue with say where you can't actually use the tendermint unsafe reset or the comment BFT unsafe reset whatever it's called now.

581
00:42:59,160 --> 00:43:17,160
So you know my state sync script actually it'll just delete the wasm and data files and then just create a the problem is is the creation like it'll work the unsafe reset but it won't.

582
00:43:17,160 --> 00:43:28,160
I don't think you can then start a node from it or it doesn't work but if you just delete the data and wasm directories you have to then create the state file and now otherwise the node won't stay.

583
00:43:28,160 --> 00:43:39,160
Which is a bit of a nuance to say so yeah well I mean my yeah maybe that was there fix for that they just like fuck it will nuke the whole thing and start again.

584
00:43:39,160 --> 00:43:53,160
Yeah well I mean my because I don't know if I'm state syncing RPC node or a validator or whatever else right whether it's a signing validator if it's just a node so I always might my approach on because that reset was going away so I just moved the key out.

585
00:43:53,160 --> 00:44:05,160
I'm sorry I moved the state out I delete the folder I recreate the folder I move the key back and I delete the wasm directory if it's outside of data and then it just states in from there right that's the it's just a safe process.

586
00:44:05,160 --> 00:44:19,160
There are some interesting things with the various chains that some of them you need to actually get the wasm directory when you do like when you restore from a snapshot say some you need to get the wasm directory.

587
00:44:19,160 --> 00:44:31,160
How does he don't I don't know so that's a good question actually so I never do it I just get the data directory and it's so sometimes I have problems with that sometimes I don't.

588
00:44:31,160 --> 00:44:40,160
So you should check that because like polka I know I think I think she'll see the same thing I do the same thing if there's if the wasm is outside of.

589
00:44:40,160 --> 00:44:55,160
So my process on taking six snapshots is that I would delete the caches out of those number one is what I would do is do anything that's a cash folder both and whether it's whether that wasm director is in or outside of data that cash folder gets deleted because that's architecture specific and that can fuck things up.

590
00:44:55,160 --> 00:45:03,160
And then that gets added to the tar so if I tar it that would be the data and wasm or it'd be just the data if the wasm is inside.

591
00:45:03,160 --> 00:45:11,160
And then so when I when I when people untar that it would automatically be in there because you're right like if those contracts are not there.

592
00:45:11,160 --> 00:45:21,160
If you add a new contract and the chain is running the obviously the binary will download it right like it has a copy it'll create that wasm file because that contract came on chain and that and the file saved.

593
00:45:21,160 --> 00:45:27,160
But if it's not there which is what I ran today and it's instantiated it will clearly app hash.

594
00:45:27,160 --> 00:45:29,160
Interesting.

595
00:45:29,160 --> 00:45:36,160
So and state sync the one the question that we did ask today is I think maybe maybe she'll say you know this answer.

596
00:45:36,160 --> 00:45:46,160
So state sync when I get the chunks right so say there's 50 chunks associated to a snapshot and we state sync that down and those 50 chunks come in.

597
00:45:46,160 --> 00:45:48,160
Does the wasm structure.

598
00:45:48,160 --> 00:45:54,160
Is it part of those chunks number one is it comes separately and number and number two.

599
00:45:54,160 --> 00:45:57,160
Does it have to come from the same source as a snapshot.

600
00:45:57,160 --> 00:46:00,160
What do you mean does that come from the same source source of the snapshot.

601
00:46:00,160 --> 00:46:14,160
Well I'm picking like that thing it searches around for snapshots right and it finds a peer that has the snapshot at a specific height right and then there's so many chunks associated that snapshot so it says.

602
00:46:14,160 --> 00:46:18,160
Okay I'm searching around I found this peer at block height 100.

603
00:46:18,160 --> 00:46:21,160
Here's the peer ID right CC Web all.

604
00:46:21,160 --> 00:46:29,160
And then it says there's 56 chunks and it'll grab those 56 chunks right it'll fetch those 56 chunks from that peer.

605
00:46:29,160 --> 00:46:38,160
Is the wasm structure in that in those chunks or is that just the data and if not where the fuck does the bottom files come from.

606
00:46:38,160 --> 00:46:50,160
Yeah it's part of those chunks but it's also never dependent there are a couple of numbers I think tears one of them where you do have to restore the wasm directory manually like by via snapshot you can state sync the data but the wasm doesn't come with.

607
00:46:50,160 --> 00:47:04,160
Another important thing to know is that for state syncing you don't stay sync from a single peer you use kind of like so you have those initial state sync peers right those work is for lack of better term like a light client to then connect other peers and then it

608
00:47:04,160 --> 00:47:21,160
like goes through them but the snapshots and stuff themselves come from all different peers.

609
00:47:21,160 --> 00:47:35,160
So that's kind of the point right is that you'll find a peer with the accepted height and it'll find that and then if it finds a lower height from a different peer it'll take that as well.

610
00:47:35,160 --> 00:47:41,160
That's kind of why it's important that it comes from different peers because most of them probably like if you're going 15,000 blocks back.

611
00:47:41,160 --> 00:47:56,160
Most are gonna say 15,000 blocks of state syncing snapshots right so it can go to find to one of the one or two peers that have that and then it can get the rest of the snapshots from other ones that's why you can sometimes find like state syncing doesn't go linearly.

612
00:47:56,160 --> 00:47:58,160
I find that yeah go ahead.

613
00:47:58,160 --> 00:48:11,160
Okay, no I think I'm understanding that the snapshots are coming from different peers but once I select this once the node selects a snapshot okay we have a specific height and there's 57 chunks in that snapshot.

614
00:48:11,160 --> 00:48:14,160
Is it assumed that all 57 come from that same peer?

615
00:48:14,160 --> 00:48:16,160
No.

616
00:48:16,160 --> 00:48:31,160
So it might be that there are other peers that have the same snapshot at the same height because it has to be obviously the same block height right and each individual node operator can set any sort of interval on their own like I could set 666 on the fucking interval and it'll create a snapshot every 666 right.

617
00:48:31,160 --> 00:48:35,160
Then it has to be exactly on that height because it's a snapshot at a height.

618
00:48:35,160 --> 00:48:37,160
Yep.

619
00:48:37,160 --> 00:48:42,160
So it might find another node that has a snapshot at that exact same height with that same hash.

620
00:48:42,160 --> 00:48:54,160
I think you're sticking down the exact same height too much though so it's it's reducing that height as a method for building the chunk like tree.

621
00:48:54,160 --> 00:49:11,160
But if people have snapshots like let's say every 2000 it can be like if someone has every 2000 every 3000 they're going to take more or less the same have all spacing the same chunks right like there's gonna be 1000 apart where they're gonna be different but most of the chunks within that same block are going to be the same you might just be

622
00:49:11,160 --> 00:49:17,160
the same as the whole and so because they're missing a couple they're not going to take them from them obviously they'll go to a different pier.

623
00:49:17,160 --> 00:49:19,160
Does that make sense.

624
00:49:19,160 --> 00:49:34,160
It does but but the but the snapshot is not the whole chain history it's just the hash of that chain at that point in time right like it's not like grabbing all the history associated like 57 chunks doesn't make up the whole fucking history right so.

625
00:49:34,160 --> 00:49:44,160
So I guess it's okay maybe it's more complicated and thought but back to the wasm thing I don't get this is nobody's gonna give a shit about this.

626
00:49:44,160 --> 00:49:48,160
Who wears like we need like who would who would be able to help us out with this.

627
00:49:48,160 --> 00:49:54,160
A soft or soft and soft I know he did quit wasting my time with this question this basic question.

628
00:49:54,160 --> 00:49:55,160
Yeah.

629
00:49:55,160 --> 00:49:56,160
Yeah.

630
00:49:56,160 --> 00:50:00,160
So anyway the and the wasm piece and maybe Simon to would be great.

631
00:50:00,160 --> 00:50:09,160
But the wasm piece was really interesting because it was weird that we had a node that was a state so the node was absolutely still in sync and would run contracts because.

632
00:50:09,160 --> 00:50:20,160
It had the contract right in the wasm directory was there and had each of those wasm files but there was a mix of not that was just be a hash or whatever that is like some sort of ID.

633
00:50:20,160 --> 00:50:36,160
And that ID with that wasm and when I pulled a state when I pulled a snapshot from that from that peer because it showed the peer ID on the snapshot I was pulling it wouldn't pull it would only pull the dot wasm files which means I was getting.

634
00:50:36,160 --> 00:50:46,160
50 contracts even though that node had 60 contracts and so when it would call one of those 10 I would have hash it's crazy so.

635
00:50:46,160 --> 00:50:49,160
What why why why why was it named different okay.

636
00:50:49,160 --> 00:51:01,160
I don't know that that's a question so that we went back to confido to ask like why was this and that knows the state sync today so like why was that node had different level like it had some had wasm on and some did not.

637
00:51:01,160 --> 00:51:03,160
And.

638
00:51:03,160 --> 00:51:10,160
And it looked like it was older contracts that was and so we created a new one and that one obviously had that wasm and my node had it right as soon as I state synced and all that kind of stuff.

639
00:51:10,160 --> 00:51:20,160
But it was really weird so it took it just took a while to figure that out and figure out why why nodes were ap hashing on something that you know just writing a simple state sync on a network.

640
00:51:20,160 --> 00:51:27,160
So anyway, it was like it's just been and that's sort of stressful because there's a lot I mean there's a lot of hype around.

641
00:51:27,160 --> 00:51:41,160
Like anything else right it's a new chain right or this this new iteration of this chain and like there's a lot of teams that are building on it and like I don't want to run nodes that are fucking ap hashing and you know are down and we're getting no server to handle this request type of situations because.

642
00:51:41,160 --> 00:51:53,160
Because of issues that are not outside of my control but like somewhat out of time and control and and so like I'm a I almost I almost pulled no script and you're a lay oh script.

643
00:51:53,160 --> 00:51:57,160
I was gonna watch dog of fortune.

644
00:51:57,160 --> 00:52:04,160
I was literally I was typing it up I had it I had it up in in code because what I was gonna do is I was gonna watch the block height and if it's tall for like.

645
00:52:04,160 --> 00:52:12,160
Eight seconds I was gonna fucking state sync the node I was just gonna delete it all move the script restate second and just get a second like get some rest because.

646
00:52:12,160 --> 00:52:27,160
Even overnight like it would it both all three nodes would would hash out or not hash out either hash that had hash out what's happening more randomly but they would you know fail and memory or something else like I can't I don't live like this right I don't have these issues.

647
00:52:27,160 --> 00:52:38,160
Well that's what I'm doing with science watch dogs for the win just put a watch dog turn off monitoring fucking nuke and pave everything yeah I don't give a shit yeah.

648
00:52:38,160 --> 00:52:42,160
Yeah tell us what's going on with Zion I want to hear about that.

649
00:52:42,160 --> 00:52:44,160
So for Zion.

650
00:52:44,160 --> 00:52:49,160
There are so we are running a.

651
00:52:49,160 --> 00:52:59,160
Load balanced endpoint for the team as well as like the main Zion team and they're also kind of like subletting it to other teams right sure.

652
00:52:59,160 --> 00:53:09,160
And so it's taking a lot of a lot of heat which that's not a big deal.

653
00:53:09,160 --> 00:53:14,160
We haven't figured what it is yet but I think there's a query coming in that when it does come in it just apaches the node and.

654
00:53:14,160 --> 00:53:37,160
Is it it's probably concurrency if like if I'm has it in a guess I'd say it's concurrency because that's every time I've had this same shit go on another change even say in the early days it was always concurrency and it'd be a contract call which comes in through the g rpc does some concurrency bullshit and the apaches the node anyway that's my two.

655
00:53:37,160 --> 00:53:48,160
What do you mean by concurrency tries to do like two transactions in wasm at the same time which are related and there's like a concurrency issue.

656
00:53:48,160 --> 00:53:53,160
Oh many like it's shoving them both in the same block and once dependent on the other type of idea.

657
00:53:53,160 --> 00:54:03,160
Yeah I think there's like two VMs running that using the same shit and then they just you know they're related they use the doing that are related.

658
00:54:03,160 --> 00:54:10,160
I believe that's what it is and then something just puts up concurrency issue and that patches and then it's fucked and you have to restart so.

659
00:54:10,160 --> 00:54:15,160
So then what what what did you do to resolve that so the way it's been.

660
00:54:15,160 --> 00:54:19,160
Complain and complain to the team until I fixed it.

661
00:54:19,160 --> 00:54:21,160
So that you like it now.

662
00:54:21,160 --> 00:54:23,160
Yeah, back to our point.

663
00:54:23,160 --> 00:54:25,160
Yeah, exactly.

664
00:54:25,160 --> 00:54:34,160
But yeah the way we've been resolving it right now is I've got one note off the side that takes a new snapshot every hour because stacing doesn't work on Zion so.

665
00:54:34,160 --> 00:54:35,160
Why are you.

666
00:54:35,160 --> 00:54:37,160
I'm clear.

667
00:54:37,160 --> 00:54:38,160
I think they may be like is it.

668
00:54:38,160 --> 00:54:42,160
Is it errors off bro.

669
00:54:42,160 --> 00:54:44,160
Does it does it.

670
00:54:44,160 --> 00:54:49,160
Does it apache on after a state center it never grabs stamps or doesn't whatever.

671
00:54:49,160 --> 00:54:57,160
So get snaps it fails if I remember correctly there's an IVL store error that happens to the very end.

672
00:54:57,160 --> 00:54:58,160
Got it.

673
00:54:58,160 --> 00:55:11,160
So good and thoroughly nuked something snapshots very regularly as soon as an apache is detected and it skins every few minutes stops the node restores a snapshot and kicks it back going.

674
00:55:11,160 --> 00:55:12,160
Fucking fucking.

675
00:55:12,160 --> 00:55:18,160
I'm gonna grab a little and bro.

676
00:55:18,160 --> 00:55:20,160
What a glamorous life this is.

677
00:55:20,160 --> 00:55:22,160
I'm glad we're sharing those everybody.

678
00:55:22,160 --> 00:55:27,160
This is what it is folks it's fucking you think it's like a 100% oil machine.

679
00:55:27,160 --> 00:55:31,160
Meanwhile, fucking watchdog scripts that new can pay.

680
00:55:31,160 --> 00:55:38,160
It's a never ending saga of no sleep and last resort typing at bash.

681
00:55:38,160 --> 00:55:41,160
So you're so that those nodes only apache.

682
00:55:41,160 --> 00:55:44,160
Those are the nodes that are actually receiving the transactions.

683
00:55:44,160 --> 00:55:45,160
Right.

684
00:55:45,160 --> 00:55:47,160
Yeah, so that's a different app hashing.

685
00:55:47,160 --> 00:55:48,160
Consensus issue.

686
00:55:48,160 --> 00:55:49,160
Yep.

687
00:55:49,160 --> 00:55:50,160
Correct.

688
00:55:50,160 --> 00:55:52,160
I've gone through and looked at caddy logs.

689
00:55:52,160 --> 00:55:56,160
I've looked at all proxy logs and there's nothing that stands out.

690
00:55:56,160 --> 00:56:02,160
They also I've been piping the logs to the team and they also haven't seen anything that stands out.

691
00:56:02,160 --> 00:56:03,160
Everything looks fine.

692
00:56:03,160 --> 00:56:05,160
So it's like, go ahead.

693
00:56:05,160 --> 00:56:08,160
So is it how many nodes are in the bouncer?

694
00:56:08,160 --> 00:56:10,160
10 10.

695
00:56:10,160 --> 00:56:11,160
10.

696
00:56:11,160 --> 00:56:12,160
That's a lot.

697
00:56:12,160 --> 00:56:13,160
Yeah.

698
00:56:13,160 --> 00:56:15,160
Fucking 10 at one endpoint.

699
00:56:15,160 --> 00:56:16,160
Yes.

700
00:56:16,160 --> 00:56:18,160
Are you doing like geo one endpoint?

701
00:56:18,160 --> 00:56:20,160
Holy shit.

702
00:56:20,160 --> 00:56:22,160
So 10.

703
00:56:22,160 --> 00:56:23,160
Yeah.

704
00:56:23,160 --> 00:56:24,160
Are they all same region?

705
00:56:24,160 --> 00:56:27,160
How many fucking how many fucking transactions are you seeing?

706
00:56:27,160 --> 00:56:32,160
Like how much how much RPC requests are you getting?

707
00:56:32,160 --> 00:56:34,160
60 to 100 requests per second.

708
00:56:34,160 --> 00:56:36,160
Oh, that does not match.

709
00:56:36,160 --> 00:56:40,160
So what's what's what's your balance strategy?

710
00:56:40,160 --> 00:56:41,160
Round rolling.

711
00:56:41,160 --> 00:56:42,160
Pretty simple.

712
00:56:42,160 --> 00:56:45,160
And then if one fails and it just retries to the next note.

713
00:56:45,160 --> 00:56:53,160
So is it just the node that is so what what actually have app hashes all of them?

714
00:56:53,160 --> 00:56:54,160
No.

715
00:56:54,160 --> 00:57:02,160
So the one that so typically what happens is one lap hash and then it'll app hash the next one in the line because it'll retry.

716
00:57:02,160 --> 00:57:13,160
Can you can you balance based off of source IP so that in that situation to Mel's point, like, is it because they're I'm not sure I'm not sure why the first note would have has to be honest.

717
00:57:13,160 --> 00:57:19,160
But but is it because those two transactions need to go to the same node?

718
00:57:19,160 --> 00:57:20,160
Wouldn't think so.

719
00:57:20,160 --> 00:57:22,160
I wouldn't think they'd be going to the same node.

720
00:57:22,160 --> 00:57:26,160
If it's currency issue, you'd want them both in the same you want them to both in the same mempool.

721
00:57:26,160 --> 00:57:27,160
Right.

722
00:57:27,160 --> 00:57:34,160
If if like they're probably batching transactions though, if they need to be done together, if they need to go on the same block that'd be batching them.

723
00:57:34,160 --> 00:57:49,160
Like on the VM side and the not so much on the tendermen side to be honest, but on the VM side, I make sure that based on source IP, they're going to the same node because on the VM side, I know there is a there is a tremendous amount of there.

724
00:57:49,160 --> 00:57:51,160
It's a little bit more complicated.

725
00:57:51,160 --> 00:58:00,160
I don't really know it well enough, but I've seen this and also on other nodes like associate to geth and some other types of things even the Ethereum side, even though that that that's a little bit less of an issue because the block times are so longer.

726
00:58:00,160 --> 00:58:05,160
But on other nodes that are that are geth based that have a much shorter block time like I was a good example of that.

727
00:58:05,160 --> 00:58:07,160
Like I make sure that those notes are going the same one.

728
00:58:07,160 --> 00:58:10,160
So so you might want to change the balance to not be around Robin.

729
00:58:10,160 --> 00:58:13,160
You might want to say at least as a test.

730
00:58:13,160 --> 00:58:15,160
Well, you can round Robin.

731
00:58:15,160 --> 00:58:18,160
You can round Robin, but just have.

732
00:58:18,160 --> 00:58:21,160
What is it called affinity IP affinity?

733
00:58:21,160 --> 00:58:26,160
So that I mean, yeah, it'll round Robin to new users.

734
00:58:26,160 --> 00:58:31,160
But once it assigns like a server to a user, it'll stick to that until it becomes unavailable.

735
00:58:31,160 --> 00:58:36,160
Like HAProxy, I would I would balance based off of the wall.

736
00:58:36,160 --> 00:58:37,160
Are you using HAPho?

737
00:58:37,160 --> 00:58:39,160
Shilty using catty.

738
00:58:39,160 --> 00:58:41,160
No, no, but catty catty can do that.

739
00:58:41,160 --> 00:58:44,160
Yeah, yeah, I would look at this.

740
00:58:44,160 --> 00:58:45,160
But is it behind Cloudflare?

741
00:58:45,160 --> 00:58:46,160
No.

742
00:58:46,160 --> 00:58:47,160
Yeah.

743
00:58:47,160 --> 00:58:50,160
So you have to look at you're using Cloudflare and you've got multiple proxies.

744
00:58:50,160 --> 00:58:54,160
You'll have to put IP affinity in Cloudflare.

745
00:58:54,160 --> 00:59:00,160
And IP affinity in HAProxy or catty.

746
00:59:00,160 --> 00:59:01,160
Yeah.

747
00:59:01,160 --> 00:59:10,160
In catty, you need to look at the header called connecting dot dash IP because the IP is actually the Cloudflare IP.

748
00:59:10,160 --> 00:59:13,160
So you have to look at the header for connecting IP.

749
00:59:13,160 --> 00:59:19,160
And then that'll pick off the header, which is the client IP to make sure the client IP gets routed to the same node.

750
00:59:19,160 --> 00:59:21,160
And then yes, right.

751
00:59:21,160 --> 00:59:24,160
And forward IP, but yeah, same difference.

752
00:59:24,160 --> 00:59:25,160
No, not forward IP.

753
00:59:25,160 --> 00:59:26,160
You want to look at the header.

754
00:59:26,160 --> 00:59:31,160
There's a header called connecting IP because that none I even in catty.

755
00:59:31,160 --> 00:59:42,160
No, that's if it's proxy through something like Cloudflare, I'm pretty sure in catty, you need the forward IP, which is like before it's the previous proxy.

756
00:59:42,160 --> 00:59:48,160
Otherwise, if you use the if you use the source IP, it will just give you the proxy source IP will give you the proxy.

757
00:59:48,160 --> 00:59:51,160
But Cloudflare sends a specific header with the client IP.

758
00:59:51,160 --> 00:59:54,160
And that's called connecting capital C dash IP.

759
00:59:54,160 --> 00:59:57,160
That is the Cloudflare connecting IP.

760
00:59:57,160 --> 00:59:59,160
That means that's the IP that can ask the request.

761
00:59:59,160 --> 01:00:02,160
That's what you want to that's what you want to balance off of.

762
01:00:02,160 --> 01:00:04,160
We can still tell everybody.

763
01:00:04,160 --> 01:00:05,160
I can look at that.

764
01:00:05,160 --> 01:00:06,160
Yeah.

765
01:00:06,160 --> 01:00:12,160
And then and then the nulls point in Cloudflare, you want to make sure that you have affinity to the just means that like, hey, send it to the same balancer.

766
01:00:12,160 --> 01:00:13,160
Yeah.

767
01:00:13,160 --> 01:00:16,160
To be the right, the right term, 10 to the same origin.

768
01:00:16,160 --> 01:00:21,160
And then the origin server would then but if you only have one origin, you have one balancer multiple balancers.

769
01:00:21,160 --> 01:00:23,160
Oh, sorry, multiple balancers.

770
01:00:23,160 --> 01:00:24,160
Okay.

771
01:00:24,160 --> 01:00:25,160
All right.

772
01:00:25,160 --> 01:00:32,160
So then if any of their well, one, so hang on one, one balancer in Cloudflare, but that goes to multiple origins, right?

773
01:00:32,160 --> 01:00:34,160
Like you've got multiple caddies running.

774
01:00:34,160 --> 01:00:35,160
Yeah.

775
01:00:35,160 --> 01:00:36,160
Yeah.

776
01:00:36,160 --> 01:00:39,160
So you need session affinity to the, the caddy that it originally sends it to.

777
01:00:39,160 --> 01:00:40,160
Yeah.

778
01:00:40,160 --> 01:00:41,160
Okay.

779
01:00:41,160 --> 01:00:42,160
That's a cool idea.

780
01:00:42,160 --> 01:00:43,160
I will do that.

781
01:00:43,160 --> 01:00:44,160
I don't know.

782
01:00:44,160 --> 01:00:45,160
I don't know.

783
01:00:45,160 --> 01:00:50,160
I think it's like this idea me sir, because we had this problem like years ago, right?

784
01:00:50,160 --> 01:00:55,160
And that's when we came up with that fucking affinity thing.

785
01:00:55,160 --> 01:00:59,160
I've had this problem on say ends osmosis.

786
01:00:59,160 --> 01:01:00,160
Not so much.

787
01:01:00,160 --> 01:01:01,160
Yeah.

788
01:01:01,160 --> 01:01:02,160
I haven't had so much on the tenement side.

789
01:01:02,160 --> 01:01:06,160
I've had a little more new VM side, but, but it's not bad.

790
01:01:06,160 --> 01:01:13,160
I mean, I mean, I don't, I don't really enjoy that too much with like, I mean, we have a lot of clients that have like fixed IPs.

791
01:01:13,160 --> 01:01:15,160
We get a lot of traffic from the same IP.

792
01:01:15,160 --> 01:01:19,160
I don't know if I want just, you know, whatever.

793
01:01:19,160 --> 01:01:20,160
It doesn't really matter.

794
01:01:20,160 --> 01:01:21,160
I have to be honest.

795
01:01:21,160 --> 01:01:29,160
And then, and then HAProxy, which is like a, like, like a child to me, but that HAProxy,

796
01:01:29,160 --> 01:01:33,160
then, then I do balancing based off of like a ton of different shit.

797
01:01:33,160 --> 01:01:36,160
And then I love the failover aspect of that and the controls in there.

798
01:01:36,160 --> 01:01:42,160
But that might, I, so it's a long shot, but in those types of situations, if you are getting a shiller

799
01:01:42,160 --> 01:01:46,160
transactions in a very short time span, that is a good example.

800
01:01:46,160 --> 01:01:53,160
The other thing you could do is just expose one port on one node and, and have them try to just one node.

801
01:01:53,160 --> 01:01:55,160
That's another way of getting around this.

802
01:01:55,160 --> 01:02:02,160
Like you have this giant fucking ten node, whatever the fuck you're doing, but just expose one thing and say, hey, try to get into here.

803
01:02:02,160 --> 01:02:08,160
And if it fails there, like you just have one, one node with a, you know, 57 port open, it fails there.

804
01:02:08,160 --> 01:02:09,160
Then, you know, this isn't the issue.

805
01:02:09,160 --> 01:02:13,160
That's a good point, except the fact that there are a bunch of different teams involved that are hitting it.

806
01:02:13,160 --> 01:02:17,160
And so I have to go to each team saying, Hey, only use this and see what happens.

807
01:02:17,160 --> 01:02:18,160
I think that's where the issue is coming from.

808
01:02:18,160 --> 01:02:28,160
Cause I think that if I were to, if I were to point fingers, this wasn't an issue until no most came up and it's happened basically since then.

809
01:02:28,160 --> 01:02:38,160
And so if I had to guess, it was the no most team, nothing to do here in Fisher or something, just that whatever's happening with their utilization has exacerbated the issue.

810
01:02:38,160 --> 01:02:40,160
They are doing lots of crazy shit though.

811
01:02:40,160 --> 01:02:45,160
We think it looks 10,000 fucking IBC channels.

812
01:02:45,160 --> 01:02:46,160
Yeah.

813
01:02:46,160 --> 01:02:51,160
Well, that's expected though, because ICA is a new channel per user.

814
01:02:51,160 --> 01:02:53,160
So that's, that's totally normal.

815
01:02:53,160 --> 01:02:54,160
What?

816
01:02:54,160 --> 01:02:56,160
I was waiting for it.

817
01:02:56,160 --> 01:02:57,160
Look at what?

818
01:02:57,160 --> 01:02:58,160
Yeah.

819
01:02:58,160 --> 01:03:02,160
And they got just an insane amount of channels and what is your goal?

820
01:03:02,160 --> 01:03:03,160
Wait, wait, wait, wait.

821
01:03:03,160 --> 01:03:04,160
Per what, what kind of user?

822
01:03:04,160 --> 01:03:06,160
Like me, like a person?

823
01:03:06,160 --> 01:03:14,160
Yeah, so for every ICA account, yeah, like if I create an ICA account, an account over ICA, what's the entertain account?

824
01:03:14,160 --> 01:03:17,160
Yeah, then you'll get your own channel for that.

825
01:03:17,160 --> 01:03:21,160
If I do, if I do one transaction, that channel is created and then it's there forever.

826
01:03:21,160 --> 01:03:22,160
Yeah, basically.

827
01:03:22,160 --> 01:03:23,160
What the fuck?

828
01:03:23,160 --> 01:03:27,160
That's, that's an overhead.

829
01:03:27,160 --> 01:03:28,160
The strange love know about this.

830
01:03:28,160 --> 01:03:31,160
Like, you know, is it built for that?

831
01:03:31,160 --> 01:03:32,160
It's not really built for that.

832
01:03:32,160 --> 01:03:34,160
Is it kind of it?

833
01:03:34,160 --> 01:03:36,160
It handles it fine.

834
01:03:36,160 --> 01:03:38,160
Like you basically just do a wild card for channels.

835
01:03:38,160 --> 01:03:41,160
The problem is that, I mean, they need to stay open, right?

836
01:03:41,160 --> 01:03:48,160
And so the more I see channels you have, you're pulling now for more and more channels and keeping more and more channels alive.

837
01:03:48,160 --> 01:03:50,160
It's kind of like a recursive hellhole.

838
01:03:50,160 --> 01:03:56,160
Could you imagine the, the bloody chain?

839
01:03:56,160 --> 01:03:57,160
What do they call it?

840
01:03:57,160 --> 01:04:01,160
Chain the poor registries for, for that chain.

841
01:04:01,160 --> 01:04:06,160
The channel is just 400 pages long.

842
01:04:06,160 --> 01:04:07,160
Yeah.

843
01:04:07,160 --> 01:04:09,160
The poor mid-skinned, IBC channel team.

844
01:04:09,160 --> 01:04:10,160
Oh God.

845
01:04:10,160 --> 01:04:11,160
Oh my God.

846
01:04:11,160 --> 01:04:18,160
But, but once that channel is open, doesn't expire?

847
01:04:18,160 --> 01:04:19,160
Yeah.

848
01:04:19,160 --> 01:04:21,160
So that's why we had to keep pulling for it.

849
01:04:21,160 --> 01:04:25,160
So in theory it should, but no, wait, wait, wait, wait, wait, wait, because it has to,

850
01:04:25,160 --> 01:04:26,160
has to expire.

851
01:04:26,160 --> 01:04:30,160
It was a two thirds of the unbonding period, right?

852
01:04:30,160 --> 01:04:33,160
So does that mean you have to run?

853
01:04:33,160 --> 01:04:35,160
Wait, I don't understand this at all.

854
01:04:35,160 --> 01:04:39,160
Does that mean you have to run a transaction through every one of those channels, every

855
01:04:39,160 --> 01:04:41,160
two thirds of the unbonding period?

856
01:04:41,160 --> 01:04:45,160
So I'm not sure how that works for ICA specifically.

857
01:04:45,160 --> 01:04:48,160
It might have acquired that or they might have done it differently.

858
01:04:48,160 --> 01:04:50,160
I would guess it to have it differently.

859
01:04:50,160 --> 01:04:51,160
You're right.

860
01:04:51,160 --> 01:04:52,160
It's not true.

861
01:04:52,160 --> 01:04:53,160
IBC, right?

862
01:04:53,160 --> 01:04:54,160
It's not a regular IBC channel.

863
01:04:54,160 --> 01:04:55,160
It's truly an IBA.

864
01:04:55,160 --> 01:04:56,160
It's definitely different, right?

865
01:04:56,160 --> 01:04:57,160
Yeah.

866
01:04:57,160 --> 01:04:58,160
Yeah.

867
01:04:58,160 --> 01:04:59,160
Okay.

868
01:04:59,160 --> 01:05:00,160
All right.

869
01:05:00,160 --> 01:05:03,160
I was thinking in my head, it was like an IBC channel per user.

870
01:05:03,160 --> 01:05:05,160
I was like, what the hell is going on?

871
01:05:05,160 --> 01:05:07,160
Well, I mean, it is, it is a channel per user.

872
01:05:07,160 --> 01:05:11,160
I just don't know if it has the same rules around channels closing.

873
01:05:11,160 --> 01:05:12,160
Yeah.

874
01:05:12,160 --> 01:05:13,160
Yeah.

875
01:05:13,160 --> 01:05:17,160
And I think strictly speaking, if I were to guess, I would say that it does.

876
01:05:17,160 --> 01:05:23,160
But for IBC, you can just do an update channel transaction, which are very simple, but it

877
01:05:23,160 --> 01:05:25,160
would still, you know, increase state.

878
01:05:25,160 --> 01:05:26,160
Right.

879
01:05:26,160 --> 01:05:27,160
Right.

880
01:05:27,160 --> 01:05:28,160
Okay.

881
01:05:28,160 --> 01:05:31,160
So, this is like a technical fucking episode, man.

882
01:05:31,160 --> 01:05:32,160
We're like deep in it.

883
01:05:32,160 --> 01:05:33,160
Phrasing on here.

884
01:05:33,160 --> 01:05:35,160
Maybe we, is it?

885
01:05:35,160 --> 01:05:37,160
Is that what gets our viewers?

886
01:05:37,160 --> 01:05:39,160
Hey, there you go.

887
01:05:39,160 --> 01:05:42,160
Sup, rockaboy.

888
01:05:42,160 --> 01:05:44,160
Look at you.

889
01:05:44,160 --> 01:05:45,160
I like this.

890
01:05:45,160 --> 01:05:48,160
Let me, let me get, let me fix things here.

891
01:05:48,160 --> 01:05:49,160
Yeah.

892
01:05:49,160 --> 01:05:50,160
Different this way.

893
01:05:50,160 --> 01:05:55,160
And what's with this big ass couch?

894
01:05:55,160 --> 01:05:58,160
I can't use the computer upstairs because everybody's already asleep.

895
01:05:58,160 --> 01:06:00,160
Where's the fucking mess, man?

896
01:06:00,160 --> 01:06:01,160
Where's the mess?

897
01:06:01,160 --> 01:06:02,160
Where's the mess?

898
01:06:02,160 --> 01:06:03,160
There's no racks of shit.

899
01:06:03,160 --> 01:06:09,160
This is like, I'm taking a chin view, not the, not the, this is like a whole new fray.

900
01:06:09,160 --> 01:06:10,160
Yeah.

901
01:06:10,160 --> 01:06:11,160
I mean, I'm not on my desk.

902
01:06:11,160 --> 01:06:12,160
I'm not on my desk.

903
01:06:12,160 --> 01:06:13,160
So, obviously that's not so.

904
01:06:13,160 --> 01:06:14,160
There you go.

905
01:06:14,160 --> 01:06:16,160
Maybe turn half the lights off and shine a flashlight.

906
01:06:16,160 --> 01:06:22,160
Oh man, you look like a completely different person like straight on than the chin view.

907
01:06:22,160 --> 01:06:24,160
So funny.

908
01:06:24,160 --> 01:06:28,160
How was your, how was your prog rock situation over there?

909
01:06:28,160 --> 01:06:30,160
Give us a little.

910
01:06:30,160 --> 01:06:37,160
I would say that the band prognosis were substantially better than their band name.

911
01:06:37,160 --> 01:06:45,160
It was kind of technical, modern British, I would say British progressive tech metal

912
01:06:45,160 --> 01:06:52,160
in that kind of, it like kind of post taken post Tesseract kind of vibe, like a little bit of atmosphere,

913
01:06:52,160 --> 01:06:57,160
but still like lots of quite heavy poly rhythms, but not like quite as Gonzo in your face as

914
01:06:57,160 --> 01:07:02,160
the American stuff, like the periphery animals as leaders type style.

915
01:07:02,160 --> 01:07:05,160
There was one particular riff that was very animals as leaders.

916
01:07:05,160 --> 01:07:09,160
It was very like modal over like a poly rhythm thing.

917
01:07:09,160 --> 01:07:15,160
And it wasn't like, you know, Scandinavian heavy, like, you know, in the sugars and whatnot.

918
01:07:15,160 --> 01:07:18,160
I didn't have that kind of dissonance to it.

919
01:07:18,160 --> 01:07:24,160
They know it probably short set, but it was, yeah, it was interesting with good, much better than their band name.

920
01:07:24,160 --> 01:07:32,160
I without my glasses and I think the other guitarist was wearing a silk shirt, not a kimono as I had initially thought.

921
01:07:32,160 --> 01:07:40,160
But I think they have, they have, they have things ahead of them in the UK tech metal scene.

922
01:07:40,160 --> 01:07:42,160
If they change their band name, that would be my review.

923
01:07:42,160 --> 01:07:45,160
Such a fine line between silk shirt and kimono.

924
01:07:45,160 --> 01:07:53,160
I basically I left the venue and I realized, I realized actually I look really tired because I am.

925
01:07:53,160 --> 01:07:59,160
And I saw one of the bands smoking outside and I just went, I'm sorry, I'm probably literally never going to see you again.

926
01:07:59,160 --> 01:08:06,160
So I've got to tell you, you're like an eight year young tech metal band and you need to change your name.

927
01:08:06,160 --> 01:08:08,160
Is that what you're saying?

928
01:08:08,160 --> 01:08:09,160
Yeah.

929
01:08:09,160 --> 01:08:16,160
I was like, the name is maybe the name word. What do I know about anything, but I think you need to change your name.

930
01:08:16,160 --> 01:08:20,160
Peace. And then I literally disappeared into the night, smoke bombed away.

931
01:08:20,160 --> 01:08:23,160
You actually said that to them.

932
01:08:23,160 --> 01:08:26,160
Good feedback, man. You got to get good to them.

933
01:08:26,160 --> 01:08:27,160
I do see.

934
01:08:27,160 --> 01:08:34,160
They were talking to like somebody else, I don't know, or a friend and that person just look at broken every rule of British.

935
01:08:34,160 --> 01:08:40,160
That person looked mortified that I was being that honest is we don't do that.

936
01:08:40,160 --> 01:08:43,160
Your name sucks. You need more 17 eight.

937
01:08:43,160 --> 01:08:45,160
Peace.

938
01:08:45,160 --> 01:08:51,160
See ya. Where's the tube?

939
01:08:51,160 --> 01:08:54,160
They do have albums on Spotify.

940
01:08:54,160 --> 01:08:59,160
They have one with a huge eyeball on it. So that's always like a red flag, but you know, so is the name.

941
01:08:59,160 --> 01:09:03,160
So, oh, there's prognosis radio. Somebody actually built a radio station around it.

942
01:09:03,160 --> 01:09:05,160
I'm all over this.

943
01:09:05,160 --> 01:09:07,160
Yeah, pretty good.

944
01:09:07,160 --> 01:09:10,160
Say that was what I was doing.

945
01:09:10,160 --> 01:09:11,160
Nice.

946
01:09:11,160 --> 01:09:14,160
We've been talking. We've been actually in pretty deep technical talk to be honest.

947
01:09:14,160 --> 01:09:15,160
Been deep in it.

948
01:09:15,160 --> 01:09:20,160
Yeah, we've been in it. We talked about wasm shit. We talked about

949
01:09:20,160 --> 01:09:27,160
Schultz working on weird concurrency issues on g rp. g rp. See was a g rp.

950
01:09:27,160 --> 01:09:28,160
API.

951
01:09:28,160 --> 01:09:32,160
I was able to disable g rp. See just to kind of reduce that.

952
01:09:32,160 --> 01:09:38,160
We talked about maybe some load balancer structure, which I'm sure he's working on right now and ignoring the rest of this podcast.

953
01:09:38,160 --> 01:09:41,160
We talked about

954
01:09:41,160 --> 01:09:51,160
He's got something that might be promising that will eventually disappoint him like the rest of the shit.

955
01:09:51,160 --> 01:09:55,160
We talked about injective pego and shitting on that a little bit.

956
01:09:55,160 --> 01:09:58,160
We missed a lot. It was a good show.

957
01:09:58,160 --> 01:10:04,160
Yeah, but there were more riffs where I was and as a bonus, the basement did smell like piss.

958
01:10:04,160 --> 01:10:06,160
Did it?

959
01:10:06,160 --> 01:10:08,160
Like all good.

960
01:10:08,160 --> 01:10:10,160
Very authentic.

961
01:10:10,160 --> 01:10:12,160
Was it a real basement?

962
01:10:12,160 --> 01:10:13,160
Post.

963
01:10:13,160 --> 01:10:21,160
Because the hip thing is right. It used to be that you could smoke indoors and that was obviously really bad for all of our health when we were all like dying.

964
01:10:21,160 --> 01:10:27,160
And that was bad, especially if you worked in the venue. Obviously you were like dying and that was super bad.

965
01:10:27,160 --> 01:10:29,160
And I do recognize all of that.

966
01:10:29,160 --> 01:10:35,160
But the venue smelled like cigarettes and not like piss and now they smell like piss.

967
01:10:35,160 --> 01:10:41,160
Are you suggesting that the smell of piss is worse than cigarettes?

968
01:10:41,160 --> 01:10:44,160
Like is that where you're coming at me with that?

969
01:10:44,160 --> 01:10:53,160
I think the smell of piss and especially gig venues because it's actually kind of piss and strong bow, but it's like strong where it's been on the floor for a while.

970
01:10:53,160 --> 01:10:55,160
Yeah, it's not like new piss. It's old piss.

971
01:10:55,160 --> 01:10:57,160
There's a difference.

972
01:10:57,160 --> 01:11:06,160
No, the piss is just a background miasma, but the strong bow is like fermenting again.

973
01:11:06,160 --> 01:11:08,160
Yeah.

974
01:11:08,160 --> 01:11:19,160
Strong bow is a cider. Not your cider, hard cider, like actual cider, like the meaning of the word.

975
01:11:19,160 --> 01:11:21,160
Not apple juice.

976
01:11:21,160 --> 01:11:26,160
So funny story about smoking in piss.

977
01:11:26,160 --> 01:11:32,160
In Australia, there used to be basically every bar you'd get.

978
01:11:32,160 --> 01:11:36,160
Bars have changed a lot in like the last 10 years in Australia.

979
01:11:36,160 --> 01:11:38,160
It used to be every bar you'd go into.

980
01:11:38,160 --> 01:11:45,160
Well, it's probably been almost, it'll be 10 to 15 years ago when they banned smoking in pubs, right?

981
01:11:45,160 --> 01:11:48,160
And dance venues and that type of stuff.

982
01:11:48,160 --> 01:12:01,160
So when I first went to college, you could like be on the dance floor burning the fuck out of everyone with your cigarette and come home with like your dreadlocks smelling like fucking smoke and then your bed and your mrs.

983
01:12:01,160 --> 01:12:03,160
Hold on.

984
01:12:03,160 --> 01:12:06,160
Smoking ban, notwithstanding, you had dreadlocks?

985
01:12:06,160 --> 01:12:08,160
Yeah, man. The banging is fucking dreadlocks.

986
01:12:08,160 --> 01:12:09,160
You had white with dreadlocks.

987
01:12:09,160 --> 01:12:10,160
Oh my God.

988
01:12:10,160 --> 01:12:11,160
That's bullshit.

989
01:12:11,160 --> 01:12:12,160
I think I need to see that.

990
01:12:12,160 --> 01:12:13,160
You're fucking, you're bullshit.

991
01:12:13,160 --> 01:12:15,160
I can, I can find some evidence.

992
01:12:15,160 --> 01:12:18,160
I can find some evidence, but not with the band because that was in the period.

993
01:12:18,160 --> 01:12:26,160
No, it would have been, I know it would have been a skate metal band like like corn, something like that, like a new metal band.

994
01:12:26,160 --> 01:12:29,160
I don't know if I should be giving you the evidence.

995
01:12:29,160 --> 01:12:31,160
Not right now, but literally, yes.

996
01:12:31,160 --> 01:12:34,160
I don't even know if I should give it to you privately.

997
01:12:34,160 --> 01:12:38,160
I feel like it's something you all like screenshot immediately.

998
01:12:38,160 --> 01:12:40,160
Did you?

999
01:12:40,160 --> 01:12:42,160
Anyway, so.

1000
01:12:42,160 --> 01:12:45,160
The block media is what will happen.

1001
01:12:45,160 --> 01:12:48,160
Yes, so NFT.

1002
01:12:48,160 --> 01:12:56,160
So the anyway, so back, back, back, back when you could like smoke in the pubs.

1003
01:12:56,160 --> 01:13:02,160
There used to be at every bar that'd be like an astray that ran around the perimeter of the base of the bar.

1004
01:13:02,160 --> 01:13:04,160
Did you have those over there?

1005
01:13:04,160 --> 01:13:05,160
I'm sorry.

1006
01:13:05,160 --> 01:13:12,160
There would be like a dished masonry at the base of the bar.

1007
01:13:12,160 --> 01:13:15,160
So, you know, in Australia, you have big, long bars, right?

1008
01:13:15,160 --> 01:13:20,160
Big corner like the old school bars, pubs, the old school pubs.

1009
01:13:20,160 --> 01:13:31,160
The bar in the pub would have like this dished masonry bowl that just went right around the perimeter of the, the bar.

1010
01:13:31,160 --> 01:13:32,160
Okay.

1011
01:13:32,160 --> 01:13:40,160
And people would like stand there and drink their schooners, having their cigarettes and then piss in the astray.

1012
01:13:40,160 --> 01:13:47,160
So that because they could do it in a crowd, like if there was just shitloads of people there, you could get away with it.

1013
01:13:47,160 --> 01:13:48,160
I don't understand.

1014
01:13:48,160 --> 01:13:55,160
Like, what the guy, you just stand at the bar and then they'll just piss against the bar wall and it would go down into the astray.

1015
01:13:55,160 --> 01:13:56,160
Oh, I see.

1016
01:13:56,160 --> 01:13:58,160
So that was at the base.

1017
01:13:58,160 --> 01:13:59,160
Yeah.

1018
01:13:59,160 --> 01:14:05,160
Yeah, bunch of fucking, fucking, I'm out of rage.

1019
01:14:05,160 --> 01:14:07,160
This is why you all got deported, man.

1020
01:14:07,160 --> 01:14:14,160
You'd wonder why the one guy had been standing at the bar drinking schooners for fucking seven hours and not moved.

1021
01:14:14,160 --> 01:14:16,160
Like how, what was he doing?

1022
01:14:16,160 --> 01:14:18,160
I was going in the astray.

1023
01:14:18,160 --> 01:14:21,160
It's a whole new world to what's been waking right, doesn't it?

1024
01:14:21,160 --> 01:14:23,160
I have no idea.

1025
01:14:23,160 --> 01:14:25,160
We'll give you a laugh for a moment.

1026
01:14:25,160 --> 01:14:30,160
Did it drain or would it just like as the night went on, get deeper and deeper and deeper?

1027
01:14:30,160 --> 01:14:34,160
Yeah, just like, just get deeper and smell like this.

1028
01:14:34,160 --> 01:14:36,160
Just get deeper.

1029
01:14:36,160 --> 01:14:38,160
God.

1030
01:14:38,160 --> 01:14:42,160
And then probably never get good, probably just evaporate the next day, you know?

1031
01:14:42,160 --> 01:14:44,160
Yeah, I'm sure that's exactly what happened.

1032
01:14:44,160 --> 01:14:46,160
That's pretty interesting.

1033
01:14:46,160 --> 01:14:48,160
Just evaporated.

1034
01:14:48,160 --> 01:14:50,160
Oh my goodness.

1035
01:14:50,160 --> 01:14:52,160
What the hell are we talking about?

1036
01:14:52,160 --> 01:14:55,160
We have a minus.

1037
01:14:55,160 --> 01:14:58,160
I can't believe I probably spent some hours going on.

1038
01:14:58,160 --> 01:15:00,160
I was like, hey, news.

1039
01:15:00,160 --> 01:15:02,160
You want to do news? The one line of news?

1040
01:15:02,160 --> 01:15:04,160
Or fuck it.

1041
01:15:04,160 --> 01:15:08,160
We've avoided news already today, like say, we're just,

1042
01:15:08,160 --> 01:15:10,160
we're on episode 113 news.

1043
01:15:10,160 --> 01:15:13,160
There's literally no episode 113 really on like,

1044
01:15:13,160 --> 01:15:15,160
but it will have news.

1045
01:15:15,160 --> 01:15:17,160
It will have news.

1046
01:15:17,160 --> 01:15:19,160
Yeah.

1047
01:15:19,160 --> 01:15:25,160
And that news is that CZ got, got sentenced finally.

1048
01:15:25,160 --> 01:15:30,160
So, so CZ from Binance pleaded guilty.

1049
01:15:30,160 --> 01:15:32,160
Oh, let me find that.

1050
01:15:32,160 --> 01:15:34,160
Let me find that.

1051
01:15:34,160 --> 01:15:36,160
Let me find that better piece of news or source.

1052
01:15:36,160 --> 01:15:38,160
There's actually a pretty funny quote in this news.

1053
01:15:38,160 --> 01:15:41,160
I'll put it up with a link to there's a, there's an article on CNBC,

1054
01:15:41,160 --> 01:15:43,160
which is not the greatest source, but whatever.

1055
01:15:43,160 --> 01:15:45,160
I thought it was pretty hilarious.

1056
01:15:45,160 --> 01:15:46,160
But CZ got sentenced.

1057
01:15:46,160 --> 01:15:49,160
He was really guilty to money laundering here in the U.S.

1058
01:15:49,160 --> 01:15:52,160
and a couple of other things.

1059
01:15:52,160 --> 01:15:54,160
And so he got sentenced.

1060
01:15:54,160 --> 01:15:57,160
The prosecutor was looking for three years.

1061
01:15:57,160 --> 01:16:05,160
They ordered Binance to pay 4.3 billion and Zao agreed to pay 50 million.

1062
01:16:05,160 --> 01:16:11,160
Kind of a gap between those two items.

1063
01:16:11,160 --> 01:16:13,160
They wanted, they wanted him to be in jail for three years.

1064
01:16:13,160 --> 01:16:15,160
He got four months,

1065
01:16:15,160 --> 01:16:19,160
which really means that here in the U.S. four months probably means you're out in 60 days.

1066
01:16:19,160 --> 01:16:20,160
Maybe.

1067
01:16:20,160 --> 01:16:21,160
And it's like, it's not even worthwhile.

1068
01:16:21,160 --> 01:16:22,160
Like you're, you're done.

1069
01:16:22,160 --> 01:16:23,160
Right.

1070
01:16:23,160 --> 01:16:28,160
Um, he did have to agree to step down as part of his, of his agreement,

1071
01:16:28,160 --> 01:16:33,160
but he owns like fucking what, like for, he owns like 90% of Binance, right?

1072
01:16:33,160 --> 01:16:37,160
So like there's nothing, he can't be CEO anymore.

1073
01:16:37,160 --> 01:16:39,160
Um, yeah, it's more about that though, isn't it?

1074
01:16:39,160 --> 01:16:42,160
It's more about the administering the slap on the wrist.

1075
01:16:42,160 --> 01:16:45,160
And it's the psychology of saying we are in charge.

1076
01:16:45,160 --> 01:16:47,160
We are in control, right?

1077
01:16:47,160 --> 01:16:48,160
Oh yeah.

1078
01:16:48,160 --> 01:16:52,160
So yeah, and like they're, obviously they have a new CEO and, and all that kind of structure

1079
01:16:52,160 --> 01:16:56,160
associated to saying that they're going to build more robust, robust compliance structure

1080
01:16:56,160 --> 01:16:57,160
and all that kind of stuff.

1081
01:16:57,160 --> 01:17:02,160
And like, which, and like, I'm not defending CZ like, like it was, it was definitely Wild West,

1082
01:17:02,160 --> 01:17:03,160
right?

1083
01:17:03,160 --> 01:17:05,160
And there was a ton of shit within there.

1084
01:17:05,160 --> 01:17:08,160
He's also not operating out of the U.S., but whatever.

1085
01:17:08,160 --> 01:17:13,160
Um, there, there was a, there, we find this great quote that came from the prosecutor.

1086
01:17:13,160 --> 01:17:18,160
I think this must have been before the, um, the actual decision.

1087
01:17:18,160 --> 01:17:26,160
Um, he was just saying that, that's, prosecutors said that Zao violated U.S. laws on unprecedented

1088
01:17:26,160 --> 01:17:27,160
scale.

1089
01:17:27,160 --> 01:17:31,160
Zao bet that he would not get caught and that if he did, the consequences would not be as

1090
01:17:31,160 --> 01:17:35,160
serious as the crime the member random said, but Zao was caught.

1091
01:17:35,160 --> 01:17:40,160
And now the court will decide what price Zao will pay for his crimes.

1092
01:17:40,160 --> 01:17:47,160
And that price is a fucking Malibu beach house and four months and he keeps four billion

1093
01:17:47,160 --> 01:17:48,160
dollars.

1094
01:17:48,160 --> 01:17:49,160
Like it's totally worth it.

1095
01:17:49,160 --> 01:17:50,160
Like I would do it for four billion.

1096
01:17:50,160 --> 01:17:51,160
Like it's fucking 60 days.

1097
01:17:51,160 --> 01:17:52,160
Like no problem.

1098
01:17:52,160 --> 01:17:54,160
I'm in how hard are you going to be 60 days?

1099
01:17:54,160 --> 01:17:57,160
What's the damage to people's lives?

1100
01:17:57,160 --> 01:18:02,160
There's actually, this, this article goes around a lot because I mean, I mean, look at an SBF

1101
01:18:02,160 --> 01:18:08,160
like Sam caught 25 years and CZ here is at four months.

1102
01:18:08,160 --> 01:18:13,760
And I think this, this article does point at, I mean, as FTX in terms of causing harm

1103
01:18:13,760 --> 01:18:16,000
to investors is one thing.

1104
01:18:16,000 --> 01:18:19,160
And the idea around just like regulatory and compliance bullshit.

1105
01:18:19,160 --> 01:18:22,560
Um, honestly, juries don't give a fuck.

1106
01:18:22,560 --> 01:18:29,040
And so I think that is kind of the, the message here, which is, which is, Hey, individuals

1107
01:18:29,040 --> 01:18:32,280
were able to do what they wanted to be able to do and as a free market for them to be

1108
01:18:32,280 --> 01:18:34,840
able to do that.

1109
01:18:34,840 --> 01:18:38,560
And whether he's in compliance or not, this is a, I mean, obviously the dollars are huge

1110
01:18:38,560 --> 01:18:43,040
and they're, they're big dollars in here, but like the jail time is nothing like this

1111
01:18:43,040 --> 01:18:45,360
is, this is pretty significant.

1112
01:18:45,360 --> 01:18:47,480
Now he's still, I think he still faces some other shit.

1113
01:18:47,480 --> 01:18:51,760
If I remember, because this is only, this was only federal, right?

1114
01:18:51,760 --> 01:18:54,760
And he still has FTC and something else I think coming.

1115
01:18:54,760 --> 01:18:57,320
So it might be, it might be a bit more of this, but this kind of sits atone.

1116
01:18:57,320 --> 01:19:00,560
I thought, um, so anyway, interesting.

1117
01:19:00,560 --> 01:19:04,880
I mean, I think the, the, the interesting thing out of this is the, like it is what

1118
01:19:04,880 --> 01:19:05,880
you just said there, right?

1119
01:19:05,880 --> 01:19:10,560
Which is in a lot of other jurisdictions, the, the threshold for even doing anything

1120
01:19:10,560 --> 01:19:12,000
is to use a harm.

1121
01:19:12,000 --> 01:19:13,000
That's the thing.

1122
01:19:13,000 --> 01:19:14,000
Yeah.

1123
01:19:14,000 --> 01:19:16,080
Um, and that's kind of what this, this ruling reflects.

1124
01:19:16,080 --> 01:19:20,760
I think the, the more twisted bit of it is it's once again, kind of, you know, like

1125
01:19:20,760 --> 01:19:23,800
you said, they don't, they're not based in the U S like they happen to operate.

1126
01:19:23,800 --> 01:19:27,920
Yeah, a lot of their markets in the U S, but I think that's the bit that is, I think a

1127
01:19:27,920 --> 01:19:33,800
bit insidious because it's like, well, for any of this party to work in the long term,

1128
01:19:33,800 --> 01:19:39,080
um, it's the, it's going to be the question of who sets the rules, right?

1129
01:19:39,080 --> 01:19:44,720
Because it's a multilateral, we're in a multilateral world and we're building software systems that

1130
01:19:44,720 --> 01:19:48,680
can work in only a multilateral setting.

1131
01:19:48,680 --> 01:19:53,360
And the idea that the U S gets to call the shots, like, yeah, you have aircraft, aircraft

1132
01:19:53,360 --> 01:19:58,200
carriers, but, uh, like, I mean, well, it does mean you can't control data sensors.

1133
01:19:58,200 --> 01:20:02,240
So yes, you can't control the internet, but like, I don't know, it just feels, it just

1134
01:20:02,240 --> 01:20:07,240
feels kind of weird to my side for punk bones to be like, I don't think your aircraft carriers

1135
01:20:07,240 --> 01:20:10,280
mean that you get to control and like, I don't know.

1136
01:20:10,280 --> 01:20:11,280
Fuck it.

1137
01:20:11,280 --> 01:20:12,280
I don't know.

1138
01:20:12,280 --> 01:20:13,280
It's late here.

1139
01:20:13,280 --> 01:20:18,160
No, I, I, I mean, I think you've seen that Napoleon film like to bring about the films

1140
01:20:18,160 --> 01:20:19,160
because that's what we really want to talk about.

1141
01:20:19,160 --> 01:20:22,160
Have you seen the Napoleon film, the Ridley Scott one?

1142
01:20:22,160 --> 01:20:23,160
No, that's a new one.

1143
01:20:23,160 --> 01:20:24,160
It's shit.

1144
01:20:24,160 --> 01:20:25,160
Don't watch it.

1145
01:20:25,160 --> 01:20:26,160
Oh, okay.

1146
01:20:26,160 --> 01:20:27,160
Well, it's not shit.

1147
01:20:27,160 --> 01:20:28,720
No, it's, it's worth watching, but it's like fine out of 10.

1148
01:20:28,720 --> 01:20:32,360
It's like a bit of a fuck up and it's way too long.

1149
01:20:32,360 --> 01:20:37,440
But there is an amazingly bad line in it, like 10 out of 10 dog shit writing, how did this

1150
01:20:37,440 --> 01:20:40,480
get past the, the writing room?

1151
01:20:40,480 --> 01:20:42,440
Is that the, is that the Joaquin Phoenix?

1152
01:20:42,440 --> 01:20:43,440
Is it?

1153
01:20:43,440 --> 01:20:44,440
Yeah.

1154
01:20:44,440 --> 01:20:47,960
He literally just shouts at the British ambassador, you think you're so great because you have

1155
01:20:47,960 --> 01:20:48,960
boats.

1156
01:20:48,960 --> 01:20:51,680
That is the line they went with.

1157
01:20:51,680 --> 01:20:55,280
It got past the writers room, it got past the editors and they put it in the film.

1158
01:20:55,280 --> 01:20:59,680
They gave it to Joaquin and then and said, yes, this is the line.

1159
01:20:59,680 --> 01:21:01,080
This is the line.

1160
01:21:01,080 --> 01:21:02,080
This is the line.

1161
01:21:02,080 --> 01:21:03,080
Please read the line.

1162
01:21:03,080 --> 01:21:04,880
This is historically accurate.

1163
01:21:04,880 --> 01:21:05,880
Taps a piece of paper.

1164
01:21:05,880 --> 01:21:07,080
Please read Mr. Phoenix.

1165
01:21:07,080 --> 01:21:09,200
Please read the line.

1166
01:21:09,200 --> 01:21:11,280
And that's kind of what, that's kind of what I feel like.

1167
01:21:11,280 --> 01:21:16,000
You look at this, this ruling and you're like, you think you're so great America because

1168
01:21:16,000 --> 01:21:18,000
you have boats.

1169
01:21:18,000 --> 01:21:19,000
Yeah.

1170
01:21:19,000 --> 01:21:20,000
Yeah.

1171
01:21:20,000 --> 01:21:26,320
Well, then again though, they probably are using Binance US as their cause of jurisdiction,

1172
01:21:26,320 --> 01:21:27,320
right?

1173
01:21:27,320 --> 01:21:28,320
Which makes it totally bad.

1174
01:21:28,320 --> 01:21:29,320
And would it also explain more?

1175
01:21:29,320 --> 01:21:32,680
Although the story is to never incorporate shit in the US ever under any circumstances,

1176
01:21:32,680 --> 01:21:35,400
otherwise you would get four months in prison.

1177
01:21:35,400 --> 01:21:36,400
Exactly.

1178
01:21:36,400 --> 01:21:37,400
Yeah.

1179
01:21:37,400 --> 01:21:38,400
Four billion.

1180
01:21:38,400 --> 01:21:39,400
Four months in prison.

1181
01:21:39,400 --> 01:21:40,400
Four months.

1182
01:21:40,400 --> 01:21:41,400
Yeah.

1183
01:21:41,400 --> 01:21:42,400
That's how they get you.

1184
01:21:42,400 --> 01:21:43,400
And that's it.

1185
01:21:43,400 --> 01:21:44,400
That's the news.

1186
01:21:44,400 --> 01:21:45,400
I think we covered it all.

1187
01:21:45,400 --> 01:21:50,200
Also, and for the record, Robot Jocks is definitely better than Napoleon.

1188
01:21:50,200 --> 01:21:56,560
And in addition, my other half, having walked in only halfway through Robot Jocks.

1189
01:21:56,560 --> 01:21:59,120
This isn't, wait, wait, wait, wait, before we go, this isn't like Dark Side of the Moon.

1190
01:21:59,120 --> 01:22:02,760
The only view of Napoleon was that Robot Jocks had a clearer plot arc.

1191
01:22:02,760 --> 01:22:03,760
Just saying.

1192
01:22:03,760 --> 01:22:05,200
This isn't like Dark Side of the Moon Robot Jocks.

1193
01:22:05,200 --> 01:22:06,560
You're talking about like true Robot Jocks.

1194
01:22:06,560 --> 01:22:11,080
You know, like sit down and smoke this with me and then we're going to start over and

1195
01:22:11,080 --> 01:22:13,520
start with a fucking money or something.

1196
01:22:13,520 --> 01:22:15,000
I tell you what I will do.

1197
01:22:15,000 --> 01:22:19,040
I will try and Torrent Robot Jocks, sink it up to Dark Side of the Moon and then like

1198
01:22:19,040 --> 01:22:22,360
put it on YouTube and get and get immediately taken down.

1199
01:22:22,360 --> 01:22:25,320
Robot Jocks is a good fucking movie.

1200
01:22:25,320 --> 01:22:27,840
It is a clear, has a clear purpose.

1201
01:22:27,840 --> 01:22:28,840
It's fun.

1202
01:22:28,840 --> 01:22:33,880
You got a good, it's got a good like, you know, protagonist.

1203
01:22:33,880 --> 01:22:34,880
Like it's all right.

1204
01:22:34,880 --> 01:22:35,880
You know what I did watch this week?

1205
01:22:35,880 --> 01:22:37,120
I watched Arena.

1206
01:22:37,120 --> 01:22:39,200
That movie, were you talking about that?

1207
01:22:39,200 --> 01:22:40,200
Yeah.

1208
01:22:40,200 --> 01:22:42,200
I was recommended it when I went looking for Robot Jocks.

1209
01:22:42,200 --> 01:22:44,480
Somebody was like, make Arena.

1210
01:22:44,480 --> 01:22:47,720
I think I think I watched that in the 80s because I remember it because I remember that

1211
01:22:47,720 --> 01:22:52,200
dude, whatever his guy's name, but he was like battling this giant grasshopper.

1212
01:22:52,200 --> 01:22:56,720
He had these huge fucking legs and like, I mean, literally it's like it's just a dude

1213
01:22:56,720 --> 01:23:03,640
beating up actors and in rubber suits and and like the robot from from Lost in Space

1214
01:23:03,640 --> 01:23:04,720
and shit.

1215
01:23:04,720 --> 01:23:10,600
But that that actually that movie was, I mean, the quality of the of the action in that movie

1216
01:23:10,600 --> 01:23:13,520
and like the sets and everything else wasn't bad for the that time.

1217
01:23:13,520 --> 01:23:14,760
Like it was a pretty decent movie.

1218
01:23:14,760 --> 01:23:15,760
I liked it.

1219
01:23:15,760 --> 01:23:16,760
You should go watch it.

1220
01:23:16,760 --> 01:23:20,240
It's on my it's the whole things on YouTube as luck would have it.

1221
01:23:20,240 --> 01:23:21,760
So they give that one away for free.

1222
01:23:21,760 --> 01:23:23,440
Not even hard to track down.

1223
01:23:23,440 --> 01:23:25,440
It's a it's on my to-do list this weekend.

1224
01:23:25,440 --> 01:23:26,440
Do it.

1225
01:23:26,440 --> 01:23:27,440
Highly recommended.

1226
01:23:27,440 --> 01:23:28,440
Well, all right.

1227
01:23:28,440 --> 01:23:29,440
Maybe the highlight.

1228
01:23:29,440 --> 01:23:30,440
At least a little bit much.

1229
01:23:30,440 --> 01:23:31,440
But it's recommended.

1230
01:23:31,440 --> 01:23:32,940
It's it's okay.

1231
01:23:32,940 --> 01:23:48,400
Like you put on the put in the background while you're doing something else.