1
00:00:00,000 --> 00:00:04,920
Very good. Well played. I just got trolled, didn't I? Oh shit, we're live.

2
00:00:08,760 --> 00:00:15,160
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent validator teams.

3
00:00:19,200 --> 00:00:28,040
Okay, so hello, everybody. Welcome to Game of Nodes, a weekly podcast from on the cosmos

4
00:00:28,040 --> 00:00:35,080
from independent validator teams from a quite tired group of independent validators this week.

5
00:00:35,080 --> 00:00:43,040
And what are we talking about this week, fellas? So this week we're talking about the Juno network

6
00:00:43,040 --> 00:00:53,640
halt, sort of what went wrong, what the cause was for the halt, where we're at now, the current

7
00:00:53,640 --> 00:00:59,760
testing that's ongoing, and how we're going to get it restarted, I think, if we have a

8
00:00:59,760 --> 00:01:08,000
solution for that yet, which I think we have a couple of ideas. So, Yusuf, we had some other

9
00:01:08,000 --> 00:01:11,840
things we're going to talk about if we had time too. What were they? I think we had some

10
00:01:11,840 --> 00:01:17,200
questions that came in a couple of weeks ago. So I think we had some interest in kind of

11
00:01:17,200 --> 00:01:22,400
understanding how some of us as independent validators got into this business and a little bit

12
00:01:22,400 --> 00:01:28,880
around backgrounds and what it takes to be a core validator for some of the cosmos. And also,

13
00:01:28,880 --> 00:01:33,840
I think there's some questions just around the full time doing this, is this kind of a side

14
00:01:33,840 --> 00:01:39,120
thing and how do you make a living out of it and those types of things? As well as, I think we

15
00:01:39,120 --> 00:01:44,280
had a third topic that was getting into it was a little bit more around, is it lucrative or not?

16
00:01:44,280 --> 00:01:49,960
And how lucrative is it? What's the necessary investments is kind of a goal on it and those

17
00:01:49,960 --> 00:01:53,400
types of areas. So maybe if we have some time, we can get into some of those.

18
00:01:53,400 --> 00:01:54,520
Cool. Yeah, that would be cool.

19
00:01:54,520 --> 00:02:01,680
So I guess to start off with, the tired, the fray is probably going to be the best person

20
00:02:01,680 --> 00:02:05,000
to give us a leading introduction to the halt.

21
00:02:05,000 --> 00:02:08,880
I thought you were going to ask, was the dumbest thing we've seen in crypto this week?

22
00:02:08,880 --> 00:02:14,880
Oh, yeah. Well, yeah. Let's do the question. I think, well, I think, okay, go the fray.

23
00:02:14,880 --> 00:02:17,120
What's the dumbest thing you've seen in crypto this week?

24
00:02:17,120 --> 00:02:21,520
Well, the dumbest thing I've seen in crypto this week was me designed to take a day off

25
00:02:21,520 --> 00:02:24,480
on Tuesday, which actually was only yesterday, wasn't it?

26
00:02:24,480 --> 00:02:25,480
It was all yesterday.

27
00:02:25,480 --> 00:02:33,600
It's been the longest 36 hours. I've, yeah, I think I've, I think I've traveled 1300

28
00:02:33,600 --> 00:02:40,600
kilometers today while, while I'm telling to a phone, burn through 2.5 gig of mobile

29
00:02:40,600 --> 00:02:43,120
data or something like that.

30
00:02:43,120 --> 00:02:48,400
So you have been buzzing across the country in a, well, across the continent in a train

31
00:02:48,400 --> 00:02:55,880
going 300 kilometers an hour whilst troubleshooting and, you know, just generally trying to get

32
00:02:55,880 --> 00:02:56,880
stuff done.

33
00:02:56,880 --> 00:03:03,320
Yeah. A friend of mine said, don't be pissed off because of the cyberpunk future you got.

34
00:03:03,320 --> 00:03:08,400
Wasn't the one you wanted when I told him what was happening. He was like, when are

35
00:03:08,400 --> 00:03:11,840
you supposed to be traveling today? And I was like, yeah, I'm on a high speed train

36
00:03:11,840 --> 00:03:16,680
literally right now. And he was like, you can't be, you can't be sore because the cyberpunk

37
00:03:16,680 --> 00:03:21,600
future you got wasn't the one you wanted, man. This is exactly what you signed up for,

38
00:03:21,600 --> 00:03:22,600
you prick.

39
00:03:22,600 --> 00:03:23,600
So yeah,

40
00:03:23,600 --> 00:03:28,400
I think we basically signed up to never sleep again.

41
00:03:28,400 --> 00:03:30,240
So I'm beginning to tell that way. Yeah.

42
00:03:30,240 --> 00:03:35,720
Yeah. For every chain you add, you lose another hour and a half of your day.

43
00:03:35,720 --> 00:03:39,680
I'm kind of surprised we're all not gray yet. We're all like surprisingly ungray and

44
00:03:39,680 --> 00:03:42,760
surprisingly handsome. I mean, I'm just looking at Schultz actually.

45
00:03:42,760 --> 00:03:44,560
Yeah. Yeah. Exactly.

46
00:03:44,560 --> 00:03:49,200
I think Schultz is like had a haircut or something.

47
00:03:49,200 --> 00:03:51,160
You know, a different angle does a lot to me.

48
00:03:51,160 --> 00:03:53,760
Yeah. It's a different angle this week.

49
00:03:53,760 --> 00:03:59,040
Yeah. I'm in Puerto Rico right now. So straight on this time. I couldn't do it this time,

50
00:03:59,040 --> 00:04:00,040
I suppose.

51
00:04:00,040 --> 00:04:01,040
There you go.

52
00:04:01,040 --> 00:04:08,160
You're outside your studio. I'm outside my studio. Yusef is still on point. And the

53
00:04:08,160 --> 00:04:09,800
phrase back in his hobble.

54
00:04:09,800 --> 00:04:10,800
Yeah.

55
00:04:10,800 --> 00:04:17,640
Back in the hobble. Yeah. So dumbest thing in crypto this week has to be joking aside

56
00:04:17,640 --> 00:04:25,080
that we got hit by the exploit that we were patching six hours before we patched it. That

57
00:04:25,080 --> 00:04:33,120
is devastating. Like the second that I saw my pager, I was literally like, oh, like not

58
00:04:33,120 --> 00:04:34,120
this.

59
00:04:34,120 --> 00:04:38,400
It could not be that if, I mean, obviously it's good to know what it is, but I was like,

60
00:04:38,400 --> 00:04:42,760
if it's that ABCI thing, I'm going to be pissed.

61
00:04:42,760 --> 00:04:47,400
And really, we just randomly chose an upgrade block, right? It could have been, you give

62
00:04:47,400 --> 00:04:51,600
yourself four days because we've, you don't want to make it too close, right? We made

63
00:04:51,600 --> 00:04:52,600
it four days.

64
00:04:52,600 --> 00:04:54,960
Which is going to do it on Monday.

65
00:04:54,960 --> 00:05:00,560
Right on Monday, like first thing Monday morning. And it was like, oh no, we just, you know,

66
00:05:00,560 --> 00:05:04,640
we're going to be late. We were jumping this upgrade on them, relatively short notice. Let's

67
00:05:04,640 --> 00:05:09,160
just give everybody an extra day. What's the worst that could happen?

68
00:05:09,160 --> 00:05:11,960
You gave us three extra days. It was nice.

69
00:05:11,960 --> 00:05:15,160
What's the worst that could happen? Terrible.

70
00:05:15,160 --> 00:05:20,880
Okay. So it has to do with the exploit that was being patched next time, not the exploit

71
00:05:20,880 --> 00:05:23,440
that was patched previously. Correct?

72
00:05:23,440 --> 00:05:32,440
No. So the LupaKalia upgrade patched the exploit we got hit with. So six hours later, and would

73
00:05:32,440 --> 00:05:44,000
have been home and dry. So the WasmD, whatever it is, CWA 002 2022, which got released this

74
00:05:44,000 --> 00:05:51,960
morning, which is bad. It's a zero day in Cosmosm. We've already got a patch in for that

75
00:05:51,960 --> 00:05:57,560
today, in amongst all the other stuff. And that's a separate thing. So we didn't get

76
00:05:57,560 --> 00:06:04,800
hit by that. We got hit by something we already knew about, something that we had a patch

77
00:06:04,800 --> 00:06:15,640
already 29 or 30 days ago. But this thing you might have heard of, Prop 16, happened.

78
00:06:15,640 --> 00:06:20,800
And it meant that our testing and our original release date for that, which was mid-March,

79
00:06:20,800 --> 00:06:26,280
got pushed back because we all had to focus on dealing with the fire that was in front

80
00:06:26,280 --> 00:06:33,600
of us. And we couldn't, it just didn't seem like we could get the validator set or anybody

81
00:06:33,600 --> 00:06:36,720
to focus on getting that, like the final bit of testing signed off and get it out of the

82
00:06:36,720 --> 00:06:42,240
door. Now in hindsight, it's really obvious to say, you know, that was a mistake. We should

83
00:06:42,240 --> 00:06:47,040
have just prioritized the security release. And that is what, I mean, we're going to do

84
00:06:47,040 --> 00:06:52,320
a full root cause from all of this. We've already started pulling together some notes

85
00:06:52,320 --> 00:06:57,200
on like things that we already know, just, you know, we're going to have to be quite

86
00:06:57,200 --> 00:07:02,360
harsh on ourselves, I think, about this. And I think one of the big things that will come

87
00:07:02,360 --> 00:07:12,760
out of it is that in particular, the core devs were very blindsided by Prop 16. And the,

88
00:07:12,760 --> 00:07:17,000
I don't want to go over it too much. I know we talked about it on our episode on Prop 16,

89
00:07:17,000 --> 00:07:22,480
you know, if you don't know whether you legally have to leave a network, like walk away from

90
00:07:22,480 --> 00:07:28,440
it potentially, which, you know, we've all, a lot of us have, you know, had to go and

91
00:07:28,440 --> 00:07:33,280
get legal advice, even if that legal advice ended up being this is completely uncharted

92
00:07:33,280 --> 00:07:41,200
waters, it's up to you, take your risk profile as you see it, you know, in that context,

93
00:07:41,200 --> 00:07:45,520
your biggest priority as a dev sadly is actually not looking at the security upgrade you need

94
00:07:45,520 --> 00:07:52,280
to get out. It's like, you know, am I still working on this project? Like what is actually

95
00:07:52,280 --> 00:07:58,200
going on here with this level of uncertainty? So there's a big kind of continuity question,

96
00:07:58,200 --> 00:08:02,600
I think, that we were sort of blindsided by with Prop 16, because it's such an unusual

97
00:08:02,600 --> 00:08:08,480
thing to happen, like the fact we've got hit with a constitutional crisis effectively,

98
00:08:08,480 --> 00:08:16,440
and then a cyber attack. I mean, I don't think there's many other systems than a proof of

99
00:08:16,440 --> 00:08:21,680
state blockchain where that combination of things is even possible, you know.

100
00:08:21,680 --> 00:08:26,920
So it was a cyber attack. So did they get anything out of it? Did they get anything out of the

101
00:08:26,920 --> 00:08:33,080
fact that we have lost some sleep effectively? Like there was no funds that were stolen.

102
00:08:33,080 --> 00:08:38,440
There was no, just a point was made by presumably some developer because only someone that's

103
00:08:38,440 --> 00:08:43,320
accustomed to Kazim one would be able to achieve what was achieved. Is that right?

104
00:08:43,320 --> 00:08:50,840
All that happened, the fray right is that basically all of the nodes got confused on

105
00:08:50,840 --> 00:08:54,080
that block and just hold it because they didn't know what to do.

106
00:08:54,080 --> 00:09:01,560
Every single node effectively forked. So we create 125 forks of the chain, which is kind

107
00:09:01,560 --> 00:09:11,240
of cool in a twisted way. And then everybody had completely indeterminate app state. That

108
00:09:11,240 --> 00:09:18,760
was done by calling a low level ABCI query in Tenderman. Now, you obviously have to kind

109
00:09:18,760 --> 00:09:23,640
of, you have to know it's there, but if you knew it was there, it's not that hard to exploit

110
00:09:23,640 --> 00:09:29,960
it. So it's not quite like baby's first exploit, but it is like, it's like Fisher price, my

111
00:09:29,960 --> 00:09:36,960
first exploit or my first cause and my as an exploit or something like that. Like it is

112
00:09:36,960 --> 00:09:41,040
and it just happens so happens in this case, it was actually quite a devastating exploit.

113
00:09:41,040 --> 00:09:46,160
I shouldn't be laughing about it so soon, but honestly, I've had very little sleep,

114
00:09:46,160 --> 00:09:51,480
but I got to see the funny side, you know, but yeah. So what happened with this exploit

115
00:09:51,480 --> 00:10:00,160
is it was found pretty early doors in terror, I think, and they rushed out an emergency

116
00:10:00,160 --> 00:10:08,160
upgrade for it. For all the other chains affected, there was a tracking issue put up which had

117
00:10:08,160 --> 00:10:11,520
like no real specifics, but just said, oh, you know, there's some unit tests that need

118
00:10:11,520 --> 00:10:18,240
adding and the and Confio sort of just got it expedited and got it in. And the second

119
00:10:18,240 --> 00:10:23,080
thing they did, we then obviously put it into Juno's code base and we had it ready to go.

120
00:10:23,080 --> 00:10:28,760
So you kind of had to be paying attention a little bit to know that this was a vulnerability

121
00:10:28,760 --> 00:10:36,720
or you had to be poking around. However, like the, well, the thing is right, you can't speculate

122
00:10:36,720 --> 00:10:42,400
too much because obviously an attacker is going to want to outfox you using social stuff

123
00:10:42,400 --> 00:10:46,720
more than code and stuff, right? Because it's much, much easier to be like, to call the

124
00:10:46,720 --> 00:10:54,960
contract short sees killer contract and then make it look really slick and competent. So

125
00:10:54,960 --> 00:10:58,760
you're like, this is definitely the work of short see is really nicely written. And then

126
00:10:58,760 --> 00:11:01,440
like take down the chain. Everybody's like, well, this contract is obviously written by

127
00:11:01,440 --> 00:11:05,200
short seed because it's really slick and you know, there's lots of code comments and it's

128
00:11:05,200 --> 00:11:11,080
really well written. It just kind of it's quite handsome contract. So, you know, we,

129
00:11:11,080 --> 00:11:14,960
we nobody's going to like be like, Oh, no, no, right, this contract because we all know

130
00:11:14,960 --> 00:11:20,240
that null just passed as usurpers code, right? So you kind of, you know, there's no way

131
00:11:20,240 --> 00:11:25,320
null attack the chain. It was definitely short seed, but, but actually it was null, the attack

132
00:11:25,320 --> 00:11:30,960
the chain, you heard it here first. But, but that's what I mean. Like when you kind of

133
00:11:30,960 --> 00:11:35,160
look at like, you know, what can we infer from the attacker? What can we infer from the

134
00:11:35,160 --> 00:11:41,960
contract? It's cool that we bottomed it out and we know that it was an ABCI query. And

135
00:11:41,960 --> 00:11:52,560
you know, shout out to my brain is not broken is my brain is broken as a mo is his Twitter

136
00:11:52,560 --> 00:12:00,240
handle for working that out. He puts in a lot of hours to basically write a bunch of

137
00:12:00,240 --> 00:12:05,440
stuff in order that he could sort of test those transactions to just ensure that that

138
00:12:05,440 --> 00:12:09,960
was actually our kind of working theory was in fact correct.

139
00:12:09,960 --> 00:12:17,080
Oh, it's off. It's off figured out. Yeah. Yeah. And well, funny, funny thing is I'm

140
00:12:17,080 --> 00:12:22,520
repping secret network right now. So that's that work. Yeah, right. To the, to the fucking

141
00:12:22,520 --> 00:12:29,560
rescue. So that was rad. And then big shout out as well to like the sect out guys and

142
00:12:29,560 --> 00:12:33,680
everything who said our folks, I should say, who, you know, kind of came in and helped

143
00:12:33,680 --> 00:12:38,480
out with sort of like de-compliation against the bottom stuff. There were quite a lot of

144
00:12:38,480 --> 00:12:42,320
people working just on the five forensic side rather than the recovery side, which

145
00:12:42,320 --> 00:12:48,080
was quite, which was, which was really cool as well. Like that was a really interesting

146
00:12:48,080 --> 00:12:53,440
chat to be in. I think in general, like, I think we've probably all learned a lot in

147
00:12:53,440 --> 00:13:01,240
the last anybody who's not sort of, you know, basically Jack or Andrew or a self has learned

148
00:13:01,240 --> 00:13:09,320
a lot in the last 24 hours. Just because like it's, there's just had to be like, I mean,

149
00:13:09,320 --> 00:13:12,680
just in terms of like, not only are like kind of thinking of like, Oh, are we certain this

150
00:13:12,680 --> 00:13:16,960
is the, the attack vector? What are all the other things we can try? What are all the

151
00:13:16,960 --> 00:13:20,080
other things we can look at? That's like one side of it. And then the other side of

152
00:13:20,080 --> 00:13:26,680
it is like, what are all the different things we can try to kickstart the chain, right? Because

153
00:13:26,680 --> 00:13:33,160
it's, it's, it's 125 forks at the moment. So all options are on the table. But we obviously

154
00:13:33,160 --> 00:13:39,400
want to avoid a hard fork because of the IVC connections, right? Right. So it's been really,

155
00:13:39,400 --> 00:13:43,320
really interesting to see how those ideas have developed, but also like we've, we've

156
00:13:43,320 --> 00:13:48,560
had to kind of, yeah, it's been assembling the plane on the way down, but you know, we've

157
00:13:48,560 --> 00:13:57,520
got a few expert engineers in the fuselage, if you like. So the, which I'm not one, I

158
00:13:57,520 --> 00:14:02,320
should clarify, but there are, there are adults in the room who are, who are telling people

159
00:14:02,320 --> 00:14:07,600
like me what they can and can't touch as the plane is falling off the cliff. Well, yeah,

160
00:14:07,600 --> 00:14:13,600
I guess just read along and chip in with the things you can, right?

161
00:14:13,600 --> 00:14:23,360
So basically the methodology that they tested was the one where they replaced all of the

162
00:14:23,360 --> 00:14:28,560
validators. So they were just using the existing state, right? And they just replaced all of

163
00:14:28,560 --> 00:14:35,320
the validators with new keys and then just started it back up to see if starting with

164
00:14:35,320 --> 00:14:44,880
all of the same data state. Yeah. So, yeah. So like one block before the problem block

165
00:14:44,880 --> 00:14:50,960
and then yeah, like removing the Gov prop, remove, basically removing any of the things

166
00:14:50,960 --> 00:14:58,200
that would impede the chain from moving past where it was without any problems. So like

167
00:14:58,200 --> 00:15:04,360
removing the smart contract state, removing the smart contract itself, plus a bunch of

168
00:15:04,360 --> 00:15:07,000
other bits and pieces and then running it.

169
00:15:07,000 --> 00:15:14,240
I saw strange love testing rig. I saw in the, in the handler that it, there was, it needed

170
00:15:14,240 --> 00:15:21,560
to remove the, the Gov prop. But why was that exactly? The Gov prop?

171
00:15:21,560 --> 00:15:26,880
I think mainly, hang on, let me think. Oh, because we don't want to hit the, we don't

172
00:15:26,880 --> 00:15:31,000
want to change the, the upgrade height because we're not going to start, we're not going

173
00:15:31,000 --> 00:15:38,040
to have that binary. Right. Although, we're not, we're skipping that binary and basically

174
00:15:38,040 --> 00:15:43,760
we're going to a new binary now. Yeah. So we're actually wanting to like hit that upgrade

175
00:15:43,760 --> 00:15:50,720
height and then like downgrade to another, the older binary that we'd already done.

176
00:15:50,720 --> 00:15:54,920
So we were, we were, so this is a little bit of jargon for those watching the stream who

177
00:15:54,920 --> 00:16:02,000
may be on. So we, we use the thing called semantic versioning to tag versions of the

178
00:16:02,000 --> 00:16:08,520
software. Lupa Kailia is the code name for a binary, a piece of runnable code, which

179
00:16:08,520 --> 00:16:15,000
is 2.3.0. We're using this opportunity not only to get out something that is a little

180
00:16:15,000 --> 00:16:21,800
bit further along, actually we are 2.3.2 or something, it would be technically, but also

181
00:16:21,800 --> 00:16:28,400
roll in some additional, this new Cosmos and security fix that came out today. And it's

182
00:16:28,400 --> 00:16:35,400
going to be sort of like version 3.0.0. So we've moved a major version forward, I think

183
00:16:35,400 --> 00:16:39,320
primarily as well to indicate that we're actually going to, you know, move past this problem

184
00:16:39,320 --> 00:16:46,480
block in an unusual way, which is by incrementing the block, which, so I actually haven't been

185
00:16:46,480 --> 00:16:51,200
in the ecosystem long enough to, to have seen this firsthand, but I gather is the way that

186
00:16:51,200 --> 00:16:56,240
some of the early migrations were done in Cosmos. And I think that's why Jack and Co

187
00:16:56,240 --> 00:17:00,800
was sort of familiar with the, well, obviously very intimately familiar with this as a solution

188
00:17:00,800 --> 00:17:10,200
path. So Jack, Demi and Ethan Fry of Cosmos and fame with whom some people occasionally

189
00:17:10,200 --> 00:17:15,880
confuse my handle, although he is the original Fry in the Cosmos ecosystem and the first

190
00:17:15,880 --> 00:17:21,160
time my handle is ever conflicted with anybody's name. But those three was quite bullish on

191
00:17:21,160 --> 00:17:26,200
the, the incrementing one block and then moving forward approach. And it's for historical

192
00:17:26,200 --> 00:17:30,320
reasons, it's what it's, it's a way they used to do upgrades in the past. So it was

193
00:17:30,320 --> 00:17:36,360
known to work. We just have a much more complicated state now, what with IBC and smart contracts

194
00:17:36,360 --> 00:17:38,320
and yada, yada, yada.

195
00:17:38,320 --> 00:17:46,040
So we're going to have like, I think we're going to export the state. Does that going

196
00:17:46,040 --> 00:17:52,320
to necessitate a new Genesis then or same Genesis one block forward and then upgrade

197
00:17:52,320 --> 00:17:54,960
handle is going to run, we're going to have the new patch and then we'll continue on from

198
00:17:54,960 --> 00:17:57,920
there. Is that what's going to happen?

199
00:17:57,920 --> 00:18:05,360
Yeah, I think that's right. That sounds familiar. That sounds correct. Any mistakes are purely

200
00:18:05,360 --> 00:18:07,680
my own tiredness. That's the, that's the happy.

201
00:18:07,680 --> 00:18:08,680
Yeah.

202
00:18:08,680 --> 00:18:10,680
Just making sure I understand somewhat.

203
00:18:10,680 --> 00:18:16,800
Yeah, we're using, we're using the, I guess we're using the export. Yeah. Essentially

204
00:18:16,800 --> 00:18:21,640
the Genesis, aren't we? Because we're, we're, we're just going to move it forward one block

205
00:18:21,640 --> 00:18:26,520
as if we're, as if we're continuing, but we keep all that, we keep the chain integrity

206
00:18:26,520 --> 00:18:29,560
of the chain name and the all the state and everything.

207
00:18:29,560 --> 00:18:33,800
But the transactions in the, in the halted block would disappear, right?

208
00:18:33,800 --> 00:18:37,920
Yeah, I guess so. We're just keeping the block in the preview.

209
00:18:37,920 --> 00:18:42,200
Yeah, we keep the, we keep the ones from the last confirmed block, but anything, I guess

210
00:18:42,200 --> 00:18:43,840
we'll get rejected.

211
00:18:43,840 --> 00:18:46,760
And I would assume there'd be some ABC in there as well that might be sitting that might

212
00:18:46,760 --> 00:18:49,600
have to just be reattempted.

213
00:18:49,600 --> 00:18:51,000
Would it not time out?

214
00:18:51,000 --> 00:18:52,000
I don't know.

215
00:18:52,000 --> 00:18:57,800
So I'm not an ABC expert. So that I couldn't comment on, but I would be very surprised if

216
00:18:57,800 --> 00:19:01,800
it didn't time out because of, well, hang on.

217
00:19:01,800 --> 00:19:06,440
We figure it out given enough time because we kind of know, we can sort of puzzle out

218
00:19:06,440 --> 00:19:08,320
how this works.

219
00:19:08,320 --> 00:19:13,280
But presumably, it's a light client on each network, right?

220
00:19:13,280 --> 00:19:20,600
So if the, if the light client on the receiving network hasn't, hasn't actually confirmed the

221
00:19:20,600 --> 00:19:21,760
transaction, right?

222
00:19:21,760 --> 00:19:22,760
That's right.

223
00:19:22,760 --> 00:19:23,760
It will at some point fail it.

224
00:19:23,760 --> 00:19:24,760
Right.

225
00:19:24,760 --> 00:19:29,840
The problem at the moment, presumably is that it's just dialing, right?

226
00:19:29,840 --> 00:19:32,480
So or the confirmation never comes back, right?

227
00:19:32,480 --> 00:19:33,480
So it just fails.

228
00:19:33,480 --> 00:19:34,480
Is that right?

229
00:19:34,480 --> 00:19:35,480
Right.

230
00:19:35,480 --> 00:19:38,560
So it's kind of the Relayer Expert here, I think, right?

231
00:19:38,560 --> 00:19:39,960
The Resident Relayer Expert.

232
00:19:39,960 --> 00:19:40,960
Yeah.

233
00:19:40,960 --> 00:19:41,960
Yeah.

234
00:19:41,960 --> 00:19:45,640
So in theory, that's basically what's going to happen is any, any ABC transactions that

235
00:19:45,640 --> 00:19:50,360
were kind of the process of happening, whether it be inbound or outbound, they are, they

236
00:19:50,360 --> 00:19:54,680
might actually still go through just because like once you go past the block state, it'll

237
00:19:54,680 --> 00:19:59,520
then pass forward because as far as it's concerned, time hasn't changed.

238
00:19:59,520 --> 00:20:00,760
Or just time out.

239
00:20:00,760 --> 00:20:04,280
I don't expect anything that had happened or potentially happened during that block

240
00:20:04,280 --> 00:20:05,680
would cause issues.

241
00:20:05,680 --> 00:20:08,760
All funds should be continuing safe.

242
00:20:08,760 --> 00:20:09,760
Cool.

243
00:20:09,760 --> 00:20:12,000
So I guess you might have to.

244
00:20:12,000 --> 00:20:15,760
I think there's quite a lot of people asking on Telegram about how the IBC would work.

245
00:20:15,760 --> 00:20:20,080
And like, I think me and other people who don't understand IBC very well, we're going,

246
00:20:20,080 --> 00:20:24,920
well, it's going to be fine because obviously these transactions are atomic, but I don't

247
00:20:24,920 --> 00:20:26,080
know the exact details.

248
00:20:26,080 --> 00:20:27,680
But there you go.

249
00:20:27,680 --> 00:20:35,280
If anybody in the stream was wondering, Schultzie, I think has, has hopefully reassured, reassured

250
00:20:35,280 --> 00:20:37,920
everybody that it's going to be fine.

251
00:20:37,920 --> 00:20:42,280
So Schultzie, if, if, if the block never gets finalized on this end, you can just flush

252
00:20:42,280 --> 00:20:45,880
out all those transactions once we get it going again, right?

253
00:20:45,880 --> 00:20:46,880
Yeah.

254
00:20:46,880 --> 00:20:47,880
So that's basically the idea.

255
00:20:47,880 --> 00:20:52,720
Right now, obviously you can flush out transactions because that requires a, like a commit to flush

256
00:20:52,720 --> 00:20:53,720
them.

257
00:20:53,720 --> 00:20:58,480
And so since the chains are moving, you can't commit to, you can't flush them back.

258
00:20:58,480 --> 00:21:02,320
So right now, like if you look on this most, I'm sure it looks like it looks like there's

259
00:21:02,320 --> 00:21:07,400
a bunch of packets waiting because I guess in theory they are, but there's really to

260
00:21:07,400 --> 00:21:11,640
be done about that.

261
00:21:11,640 --> 00:21:18,640
So yeah, I mean, I guess that is, that is our concern of getting the chain going properly.

262
00:21:18,640 --> 00:21:22,440
And that's one of the things that we need to do more testing on before we move ahead

263
00:21:22,440 --> 00:21:29,280
with a, with a solution to, to put it out into production, right?

264
00:21:29,280 --> 00:21:35,760
So all that stuff has been tested at the moment, I believe the fray.

265
00:21:35,760 --> 00:21:36,760
Yeah.

266
00:21:36,760 --> 00:21:41,520
On the testing path, it's like get something that works, tested it with a bunch of different

267
00:21:41,520 --> 00:21:45,600
validated configurations, then start testing IBC.

268
00:21:45,600 --> 00:21:47,960
Once we're sure everything works, then push it out.

269
00:21:47,960 --> 00:21:52,360
The strange love folks are literally testing it as we speak.

270
00:21:52,360 --> 00:21:58,640
I think I did a shout out on Twitter to new Andrew or GNU Andrew, depending on how you

271
00:21:58,640 --> 00:22:04,320
pronounce new, I pronounced it new personally.

272
00:22:04,320 --> 00:22:11,560
But yeah, who's done just a load of incredible work on automating all of this testing so

273
00:22:11,560 --> 00:22:14,000
that we essentially don't have to do it by hand.

274
00:22:14,000 --> 00:22:20,080
And it's the reason we've been able to kind of react so quickly to this has been really,

275
00:22:20,080 --> 00:22:25,080
really heavily driven by, by the strange love team.

276
00:22:25,080 --> 00:22:28,280
And like I say, Andrew in particular.

277
00:22:28,280 --> 00:22:33,600
And then we, there were a couple of other strategies that were being sort of looked into an experiment

278
00:22:33,600 --> 00:22:40,240
with and that was driven very much by DEMI who had essentially a couple of other quite

279
00:22:40,240 --> 00:22:47,400
good ideas that would hopefully excuse me, allow us to move forward without resorting

280
00:22:47,400 --> 00:22:48,400
to a hard fork.

281
00:22:48,400 --> 00:22:53,040
But so ultimately, I think we're going to go with the incrementing strategy as we've

282
00:22:53,040 --> 00:22:54,240
sort of already talked about.

283
00:22:54,240 --> 00:22:57,920
But yeah, that's what they're doing at the moment is they're running a series of tests

284
00:22:57,920 --> 00:22:59,880
for sort of like edge cases.

285
00:22:59,880 --> 00:23:06,920
So now where they're up to is that we've been able to make blocks using that new strategy

286
00:23:06,920 --> 00:23:12,720
or Andrew has specifically, and we were all very excited to watch him make blocks with

287
00:23:12,720 --> 00:23:18,880
the, the genesis, which was based off of an export we sort of prepared from the, the final

288
00:23:18,880 --> 00:23:21,400
block.

289
00:23:21,400 --> 00:23:29,120
And now I think DEMI and Andrew are going through a variety of scenarios, like, you

290
00:23:29,120 --> 00:23:33,760
know, some validators haven't switched over or some validators are on a previous version

291
00:23:33,760 --> 00:23:40,000
of genesis or, you know, these kinds of things that there will be edge cases where some of

292
00:23:40,000 --> 00:23:45,600
the valset doesn't agree or people haven't quite configured it right or, you know, nothing

293
00:23:45,600 --> 00:23:51,320
malicious but potentially accidental Byzantine behavior or accidental configuration or whatever

294
00:23:51,320 --> 00:23:57,400
and that is something that, that we need to test, right?

295
00:23:57,400 --> 00:23:59,800
Because it's quite likely to happen.

296
00:23:59,800 --> 00:24:05,520
So one of the other interesting things that happened, I don't know, Noel was in this,

297
00:24:05,520 --> 00:24:11,240
I know, Shultzie, I think had a, a DEM from me in the middle of the night about this was

298
00:24:11,240 --> 00:24:14,400
quite early on in trying to diagnose the root cause.

299
00:24:14,400 --> 00:24:17,880
We were trying to work out whether or not.

300
00:24:17,880 --> 00:24:23,560
So way back when Manita first came out, we had a security vulnerability that came up

301
00:24:23,560 --> 00:24:30,200
very, very short notice and we ultimately decided to patch it by distributing a binary

302
00:24:30,200 --> 00:24:35,160
from the core team to the validator set that patched it rather than, because it wasn't

303
00:24:35,160 --> 00:24:40,600
consensus breaking in the way that we, if we were distributed it, it just ran it.

304
00:24:40,600 --> 00:24:42,680
It wouldn't be breaking consensus.

305
00:24:42,680 --> 00:24:46,440
So we were able to just get validators to upgrade.

306
00:24:46,440 --> 00:24:50,400
But we sort of realized that we didn't have any way of checking that every validator was

307
00:24:50,400 --> 00:24:57,640
on the same version and if we had more than 33% of validators not running the same binary,

308
00:24:57,640 --> 00:25:00,280
then we were still vulnerable to that security exploit.

309
00:25:00,280 --> 00:25:08,600
So on the first things we had to do, sort of overnight really last night was compile

310
00:25:08,600 --> 00:25:12,960
the spreadsheet of all of the valid, and again, DEMI was amazing kind of getting off the mark

311
00:25:12,960 --> 00:25:17,040
on this and starting to get responses.

312
00:25:17,040 --> 00:25:21,240
And I spent a lot of my evening and night nagging people over DEMs and trying to get

313
00:25:21,240 --> 00:25:26,040
in touch with people and tracking down validators who aren't in our Discord, but I'd seen them

314
00:25:26,040 --> 00:25:29,400
on a different Discord somewhere and going and finding them and going, hey, sorry, you're

315
00:25:29,400 --> 00:25:31,480
not on the junior Discord, but your node's down.

316
00:25:31,480 --> 00:25:32,480
Here's why.

317
00:25:32,480 --> 00:25:33,480
Can you fill in the spreadsheet?

318
00:25:33,480 --> 00:25:39,360
And what we were trying to establish was whether or not over 33% of voting power was on a binary

319
00:25:39,360 --> 00:25:43,360
that was vulnerable to a previously known exploit.

320
00:25:43,360 --> 00:25:49,040
So we could rule that one out because it was one of the potential consensus type breaking

321
00:25:49,040 --> 00:25:53,200
bugs that we might have been vulnerable to.

322
00:25:53,200 --> 00:26:00,160
So just so people understand why some people might not have been on that latest binary is

323
00:26:00,160 --> 00:26:07,480
because it wasn't, most people in all across the cosmos, when they're migrating a node

324
00:26:07,480 --> 00:26:14,680
and then they're building the binary stack back up in CosmoVisor, they're probably just

325
00:26:14,680 --> 00:26:18,280
building the versions from the source.

326
00:26:18,280 --> 00:26:23,160
And so in this case, there was a binary that was released that you couldn't build from

327
00:26:23,160 --> 00:26:32,560
source unless you built it a certain way to include that mucil C, the music binary.

328
00:26:32,560 --> 00:26:38,120
And so what I suspect has happened, and this would have happened to me as well if I didn't

329
00:26:38,120 --> 00:26:44,360
do it a different way, is that when people have migrated their nodes, they've probably

330
00:26:44,360 --> 00:26:50,440
taken their data folder with them and then just built the new binary from the source

331
00:26:50,440 --> 00:26:55,560
code as 2.1 and then they lose the patched binary.

332
00:26:55,560 --> 00:27:00,440
So if you're like me and didn't want to build it and just copied your CosmoVisor directory

333
00:27:00,440 --> 00:27:04,320
instead, you'd still have the patched binary.

334
00:27:04,320 --> 00:27:09,720
So it's probably, I suspect that that's why a lot of those validators weren't on that

335
00:27:09,720 --> 00:27:16,960
patched binary, probably from migrating and also new validators that had joined the set

336
00:27:16,960 --> 00:27:21,880
after that upgrade and then just built directly the 2.1 from source.

337
00:27:21,880 --> 00:27:25,600
So I don't think they can be fully blamed for it.

338
00:27:25,600 --> 00:27:30,920
There's also so much automation going on and anytime that you have a release that doesn't

339
00:27:30,920 --> 00:27:35,000
fit into those automation plans, I think things naturally fall through the cracks.

340
00:27:35,000 --> 00:27:40,720
So they might have Ansible or some other type of method that is looking either A, looking

341
00:27:40,720 --> 00:27:43,240
for new binaries or getting things that are ready.

342
00:27:43,240 --> 00:27:49,920
And I'm just going to go run the job and go pull down and check out the right branch

343
00:27:49,920 --> 00:27:53,520
and run it and go without actually maybe looking at the read me, right?

344
00:27:53,520 --> 00:27:56,600
Because everything is kind of being notified in Discord.

345
00:27:56,600 --> 00:27:59,680
And I'm really surprised that there's Juno validators that aren't in the Discord.

346
00:27:59,680 --> 00:28:00,680
Yeah.

347
00:28:00,680 --> 00:28:05,280
Like, well, some are super unknown and don't like talking to people, right?

348
00:28:05,280 --> 00:28:07,440
You have to talk to people.

349
00:28:07,440 --> 00:28:11,000
At least you have to get some announcements because if you're just watching GitHub, you're

350
00:28:11,000 --> 00:28:15,880
really not getting the full story on what's coming or what's going on and everything else,

351
00:28:15,880 --> 00:28:16,880
right?

352
00:28:16,880 --> 00:28:17,880
Yeah.

353
00:28:17,880 --> 00:28:18,880
I don't know.

354
00:28:18,880 --> 00:28:24,760
I guess there's a Telegram, I guess, and other modes like I guess not everyone uses

355
00:28:24,760 --> 00:28:25,760
Discord.

356
00:28:25,760 --> 00:28:29,680
Is there a Juno Telegram?

357
00:28:29,680 --> 00:28:33,160
I don't think there is a Dev Telegram.

358
00:28:33,160 --> 00:28:34,400
So Juno Telegram?

359
00:28:34,400 --> 00:28:39,440
Because there's like the Juno Telegram and then there's the Juno Price Telegram, but

360
00:28:39,440 --> 00:28:41,200
I don't think there's like a Juno Dev.

361
00:28:41,200 --> 00:28:43,000
Oh, like an announcements.

362
00:28:43,000 --> 00:28:48,080
I don't think there's like, there's like an upgrade is coming Juno announcements channel

363
00:28:48,080 --> 00:28:49,080
on Telegram.

364
00:28:49,080 --> 00:28:50,080
No.

365
00:28:50,080 --> 00:28:51,080
No, I don't think so.

366
00:28:51,080 --> 00:28:54,800
Could this halt have happened non-maliciously?

367
00:28:54,800 --> 00:28:59,360
Could it have happened just somebody's doing a contract that goes a specific way and it

368
00:28:59,360 --> 00:29:00,760
just, it just happened.

369
00:29:00,760 --> 00:29:01,760
Like, does this have to be malicious?

370
00:29:01,760 --> 00:29:02,760
Does it have to be an attacker?

371
00:29:02,760 --> 00:29:08,200
It could just be, hey, there's an X, there's something that somebody wrote something poorly.

372
00:29:08,200 --> 00:29:12,720
So it's possible, but unlikely.

373
00:29:12,720 --> 00:29:17,360
The reason that it looks more suspicious.

374
00:29:17,360 --> 00:29:26,600
Well, so the, if it was an attack, I think there was one of the obfuscation techniques

375
00:29:26,600 --> 00:29:32,840
was using essentially the Cosmos and default template, like the literal learners template

376
00:29:32,840 --> 00:29:35,080
to, as the basis for the contract.

377
00:29:35,080 --> 00:29:41,040
Because you look at it and you see like a chain of executes on it and you're like, some

378
00:29:41,040 --> 00:29:45,600
learner dev is just executing it a bunch of times and something crazy has happened.

379
00:29:45,600 --> 00:29:46,600
Right.

380
00:29:46,600 --> 00:29:51,320
But then you look at it and you go like, oh, no, they've, it's a counter basically.

381
00:29:51,320 --> 00:29:53,560
It just increments by one, right?

382
00:29:53,560 --> 00:29:57,920
And it'd been incremented like 55 times and you're like, nobody has the patience to submit

383
00:29:57,920 --> 00:30:01,920
55 transactions to a blockchain just because they're learning.

384
00:30:01,920 --> 00:30:07,680
Two, maybe three I buy, but we've all been learner devs and like you're just like, cool,

385
00:30:07,680 --> 00:30:09,240
it works right onto the next thing.

386
00:30:09,240 --> 00:30:11,800
You don't sit there and submit 50 transactions.

387
00:30:11,800 --> 00:30:12,800
That's just not a thing.

388
00:30:12,800 --> 00:30:14,200
A learner dev, whatever you do.

389
00:30:14,200 --> 00:30:17,920
You're too excited by the experience to do that.

390
00:30:17,920 --> 00:30:18,920
Yeah, it's like, it works.

391
00:30:18,920 --> 00:30:23,800
It was super suspicious.

392
00:30:23,800 --> 00:30:27,960
But the kind of effort to go like, oh, no, make it a learner template says one of two

393
00:30:27,960 --> 00:30:33,920
things, either they themselves are pretty new or they just want the simplest possible

394
00:30:33,920 --> 00:30:40,200
template they can use or three, it's a deliberate obfuscation technique to make it look like

395
00:30:40,200 --> 00:30:45,400
somebody who doesn't know what they're doing to kind of send us down the wrong path for

396
00:30:45,400 --> 00:30:46,680
a little while.

397
00:30:46,680 --> 00:30:50,320
So we're like, oh, what's going on here?

398
00:30:50,320 --> 00:30:55,760
But I think because we were aware of the ABCI thing, we kind of theorized it very early

399
00:30:55,760 --> 00:31:00,080
on like, like, I mean, like I say, it was pretty much instant.

400
00:31:00,080 --> 00:31:02,520
Like, is it the ABCI thing?

401
00:31:02,520 --> 00:31:07,200
And it was something that got vocalized like again, pretty much straight away.

402
00:31:07,200 --> 00:31:12,520
It just took a long time for somebody with a big enough brain to actually get all the

403
00:31:12,520 --> 00:31:16,120
way to actually proving it was that thing.

404
00:31:16,120 --> 00:31:18,320
Whereas the rest of us were kind of happy to go like, that's probably that.

405
00:31:18,320 --> 00:31:19,320
Let's move on.

406
00:31:19,320 --> 00:31:20,520
Like, right, right.

407
00:31:20,520 --> 00:31:26,880
Because what we were able to do or what particularly Andrew was able to do was actually essentially

408
00:31:26,880 --> 00:31:35,360
spin up a set of fake validators to show that if you had validators on 2.3, I loop

409
00:31:35,360 --> 00:31:38,200
Kali, they were not vulnerable.

410
00:31:38,200 --> 00:31:43,920
But because we had the the WASM, the compiled contract, the malicious contract, we could

411
00:31:43,920 --> 00:31:46,000
run it and see what happens.

412
00:31:46,000 --> 00:31:51,520
So we were able to basically see that current main net is vulnerable, which we've already

413
00:31:51,520 --> 00:31:53,480
seen, but we could replicate it.

414
00:31:53,480 --> 00:31:56,680
And then we could say, oh, well, loop Kali isn't vulnerable.

415
00:31:56,680 --> 00:32:02,760
So at that point, unless it's some completely third thing that we don't know about, it's

416
00:32:02,760 --> 00:32:05,400
likely something that we know about from the changelog.

417
00:32:05,400 --> 00:32:09,560
It's likely something that we know about from security disclosures.

418
00:32:09,560 --> 00:32:13,840
And we're relatively close to the confio teams and we talk to them about these sorts of things.

419
00:32:13,840 --> 00:32:19,360
And particularly the ABCI one was a very, like a known pretty bad thing that we were

420
00:32:19,360 --> 00:32:22,840
very, very quick in getting our fix in as a result.

421
00:32:22,840 --> 00:32:27,680
But obviously events sort of derailed us actually getting it out, which is why the second this

422
00:32:27,680 --> 00:32:28,680
happened.

423
00:32:28,680 --> 00:32:34,240
So have we just screwed it by not getting this upgrade out as early as we could have done?

424
00:32:34,240 --> 00:32:37,960
And that's that's going to be one of the things that comes out in root cause for this, you

425
00:32:37,960 --> 00:32:38,960
know, bluntly.

426
00:32:38,960 --> 00:32:45,080
So yeah, and probably maybe not even that much of a coincidence that happened six hours

427
00:32:45,080 --> 00:32:46,080
before this upgrade, right?

428
00:32:46,080 --> 00:32:49,000
We were talking about like we moved it when we started this, we're saying, oh, we moved

429
00:32:49,000 --> 00:32:55,240
it yesterday or something or Monday, but maybe they upload the contracts on Monday, I think

430
00:32:55,240 --> 00:33:00,680
and we're executing it overnight, just trying to get the same block.

431
00:33:00,680 --> 00:33:03,160
Yeah, with a couple of different transactions.

432
00:33:03,160 --> 00:33:09,160
And so it would have been that and that that mix would have been the right proposer.

433
00:33:09,160 --> 00:33:12,200
Is it proposer based or why wouldn't I think it actually matters?

434
00:33:12,200 --> 00:33:16,120
It just looked like they they had something in their mind, but actually it would have

435
00:33:16,120 --> 00:33:18,440
worked.

436
00:33:18,440 --> 00:33:25,000
If the ABCI query had worked in the right, yeah, if it if it works, it works.

437
00:33:25,000 --> 00:33:28,880
I think I don't think it proposed it was material, but yeah, they were looking for

438
00:33:28,880 --> 00:33:30,440
something and they found it.

439
00:33:30,440 --> 00:33:36,880
That was a question for 1132 after 36 hours.

440
00:33:36,880 --> 00:33:38,040
What time is it hit 12?

441
00:33:38,040 --> 00:33:39,040
It's 1230.

442
00:33:39,040 --> 00:33:40,040
Right.

443
00:33:40,040 --> 00:33:43,000
I thought we lost your connection there.

444
00:33:43,000 --> 00:33:46,320
Pause your whole brain pause on that one.

445
00:33:46,320 --> 00:33:53,960
Yeah, I've been up for I've I got a 530 this morning.

446
00:33:53,960 --> 00:33:57,000
I'm in UTC plus two and I'm in UTC plus one now.

447
00:33:57,000 --> 00:33:59,560
So can you do some quick square roots for us?

448
00:33:59,560 --> 00:34:00,560
Maybe?

449
00:34:00,560 --> 00:34:01,560
Yeah.

450
00:34:01,560 --> 00:34:02,560
I've been up for 20 hours.

451
00:34:02,560 --> 00:34:03,840
I guess I've been up for 20 hours.

452
00:34:03,840 --> 00:34:04,840
Maybe.

453
00:34:04,840 --> 00:34:05,840
I don't know.

454
00:34:05,840 --> 00:34:07,160
Something like that.

455
00:34:07,160 --> 00:34:14,600
So the fray people keep speculating that, you know, this is the whale revenge.

456
00:34:14,600 --> 00:34:17,760
So I think there's a lot of them.

457
00:34:17,760 --> 00:34:18,760
Are they?

458
00:34:18,760 --> 00:34:20,640
Hey, I think the whale is smart enough.

459
00:34:20,640 --> 00:34:21,640
Are they?

460
00:34:21,640 --> 00:34:22,640
Shots fired.

461
00:34:22,640 --> 00:34:29,680
The negotiating strategy so far does not does not seem that of a monster tactician.

462
00:34:29,680 --> 00:34:33,840
So I would say from like what we know that's highly unlikely that it's the whale.

463
00:34:33,840 --> 00:34:36,240
It doesn't really benefit anyone, right?

464
00:34:36,240 --> 00:34:38,240
Well, no, it's what?

465
00:34:38,240 --> 00:34:39,240
Yeah.

466
00:34:39,240 --> 00:34:40,240
What why?

467
00:34:40,240 --> 00:34:41,240
Why?

468
00:34:41,240 --> 00:34:42,240
Key bono, right?

469
00:34:42,240 --> 00:34:43,240
Who benefits?

470
00:34:43,240 --> 00:34:44,240
Yeah.

471
00:34:44,240 --> 00:34:49,440
Well, doesn't benefit because if it's found out to be them, their funds burn out a spike,

472
00:34:49,440 --> 00:34:50,440
right?

473
00:34:50,440 --> 00:34:56,040
If they successfully take down the chain for good, they also lose their funds.

474
00:34:56,040 --> 00:34:58,360
It's kind of seems like a lose-lose.

475
00:34:58,360 --> 00:35:00,880
There's no leverage to be gained from it, right?

476
00:35:00,880 --> 00:35:05,720
A better move would be to go, well, we know you've got, I should know, even then, even

477
00:35:05,720 --> 00:35:10,920
if they knew that we knew about a security fault that they knew about and we were days

478
00:35:10,920 --> 00:35:15,280
away from upgrading, there's still no real leverage.

479
00:35:15,280 --> 00:35:17,640
And they would know that we could recover it anyway.

480
00:35:17,640 --> 00:35:20,040
So what would be the point?

481
00:35:20,040 --> 00:35:26,920
I suppose some people will look at Evmos and go, I think Evmos is an example of, like,

482
00:35:26,920 --> 00:35:30,360
without throwing any shade, it just shows how hard it is to recover a chain when it

483
00:35:30,360 --> 00:35:32,080
gets into an indeterminate state.

484
00:35:32,080 --> 00:35:39,040
And the fact is taken 35 people to get us to a solution.

485
00:35:39,040 --> 00:35:44,560
And what strange love I've brought to the table to help us get to the testing point

486
00:35:44,560 --> 00:35:50,360
of knowing we can get this back online is cutting edge, is a cutting edge piece of test

487
00:35:50,360 --> 00:35:51,360
automation stuff.

488
00:35:51,360 --> 00:35:52,360
It is very cool.

489
00:35:52,360 --> 00:35:56,400
It is very, very clever.

490
00:35:56,400 --> 00:36:05,080
So I think if you didn't know that the devs on Juno were pretty stubborn and also had

491
00:36:05,080 --> 00:36:10,880
enough friends and favours to call in to bring the cavalry, you might think, oh, we can probably

492
00:36:10,880 --> 00:36:14,200
take them down for a couple of weeks.

493
00:36:14,200 --> 00:36:18,400
But again, who benefits from the network being down for a couple of weeks?

494
00:36:18,400 --> 00:36:20,880
Nothing's grown.

495
00:36:20,880 --> 00:36:22,360
Yeah.

496
00:36:22,360 --> 00:36:24,600
No one.

497
00:36:24,600 --> 00:36:31,600
A competitor or a board kid, really, are the two people I can think of.

498
00:36:31,600 --> 00:36:39,040
And is a competitor of Juno really that worried about us that they're going to attack us?

499
00:36:39,040 --> 00:36:41,040
I mean, maybe.

500
00:36:41,040 --> 00:36:43,280
But again, who?

501
00:36:43,280 --> 00:36:45,040
I don't think it would.

502
00:36:45,040 --> 00:36:52,600
Like what's, yeah, I mean, the more activity in the ecosystem, the more attention the ecosystem

503
00:36:52,600 --> 00:36:55,200
gets, then the better it's going to be for all of us.

504
00:36:55,200 --> 00:37:03,400
So why would you say, like, why would anyone want to take down another one of the IBC connected

505
00:37:03,400 --> 00:37:05,680
networks and it doesn't really make any sense?

506
00:37:05,680 --> 00:37:12,520
I mean, my guess would be like, if I had to just completely guess, I would say, you know,

507
00:37:12,520 --> 00:37:18,000
if you were a dev on terror and you had seen that patch, you could easily make the link

508
00:37:18,000 --> 00:37:23,000
to the ABCR, the ABCI queries.

509
00:37:23,000 --> 00:37:26,600
And then you could go, OK, if you knew what you were doing, you could then craft an exploit,

510
00:37:26,600 --> 00:37:27,600
right?

511
00:37:27,600 --> 00:37:33,880
So a board, a board young terror dev, not not as in from the terror organization, but somebody

512
00:37:33,880 --> 00:37:41,440
who knows Cosmosm because they were doing on terror and had seen their security notification.

513
00:37:41,440 --> 00:37:45,200
Maybe that's somebody who's they're just like, you know, they're doing, they're teaching

514
00:37:45,200 --> 00:37:48,480
themselves smart contracts and they're just like a bit bored.

515
00:37:48,480 --> 00:37:49,760
And they're like, oh, this will be funny.

516
00:37:49,760 --> 00:37:52,680
I wonder if I can take down Juno with this thing I've just learned.

517
00:37:52,680 --> 00:37:53,680
Right.

518
00:37:53,680 --> 00:37:57,480
Well, we've all been bored teenagers, man.

519
00:37:57,480 --> 00:37:59,880
We've all smashed windows on stupid shit.

520
00:37:59,880 --> 00:38:06,720
Like this is kind of just a multimillion dollar example of the same thing, isn't it?

521
00:38:06,720 --> 00:38:12,880
Just been a bit surprised then by appearing on the screen.

522
00:38:12,880 --> 00:38:14,640
Jack has joined us.

523
00:38:14,640 --> 00:38:17,160
And a wild Jack has appeared.

524
00:38:17,160 --> 00:38:22,640
You know, I think it just sort of underscores the importance of procedure and sort of more

525
00:38:22,640 --> 00:38:26,720
regular check ins about this stuff.

526
00:38:26,720 --> 00:38:31,600
You know, I think that the engineers who get into blockchain engineering are people who

527
00:38:31,600 --> 00:38:33,120
are kind of security focused.

528
00:38:33,120 --> 00:38:37,080
And I think especially when we're spending up chains and doing a lot of things, it is

529
00:38:37,080 --> 00:38:39,080
the first and only thing we think about.

530
00:38:39,080 --> 00:38:44,480
But I think that a lot of this stuff around sort of like regular network maintenance is

531
00:38:44,480 --> 00:38:45,480
not sexy.

532
00:38:45,480 --> 00:38:48,640
It's not quite as fun as launching new products.

533
00:38:48,640 --> 00:38:53,920
And it's really easy to forget that we just need to, you know, do the basic blocking and

534
00:38:53,920 --> 00:38:55,800
tackling.

535
00:38:55,800 --> 00:39:02,480
And yeah, you know, what I have seen is a lot of great basic blocking and tackling and

536
00:39:02,480 --> 00:39:05,080
tackling this incident, which is fantastic.

537
00:39:05,080 --> 00:39:09,360
And the number of folks that kind of came together to do that just huge, a bunch of

538
00:39:09,360 --> 00:39:11,760
people in this call, which by the way, thank you guys.

539
00:39:11,760 --> 00:39:16,120
Jack, are you talking about specifically around validators not running latest binaries and

540
00:39:16,120 --> 00:39:18,480
just kind of getting lax around those types of things?

541
00:39:18,480 --> 00:39:19,480
Is that what you're referencing?

542
00:39:19,480 --> 00:39:22,440
Well, I mean, that's definitely part of it.

543
00:39:22,440 --> 00:39:27,200
But I think from the core team perspective, too, you know, what about a security council

544
00:39:27,200 --> 00:39:30,560
that meets once a week and just does a dependency audit?

545
00:39:30,560 --> 00:39:31,560
Sure.

546
00:39:31,560 --> 00:39:37,480
And like, you know, and more regular security upgrades.

547
00:39:37,480 --> 00:39:42,560
I mean, this, I think we've effectively found that this was a target query issue where the

548
00:39:42,560 --> 00:39:47,520
attacker was querying the node ID out of the node and then hashing that.

549
00:39:47,520 --> 00:39:49,680
And that was the source of the non-determinism.

550
00:39:49,680 --> 00:39:52,960
Yeah, that was it, I think.

551
00:39:52,960 --> 00:39:58,040
It was literally the ABCI query that, yeah, it was the ABCI query exploit that meant that

552
00:39:58,040 --> 00:40:05,040
terror panic upgraded, you know, whatever it was for four and a half weeks ago.

553
00:40:05,040 --> 00:40:08,960
And then we got a security notification and we put our, we, you know, we patched, you

554
00:40:08,960 --> 00:40:15,320
know, 29 or 30 days ago, something like that on GitHub 31 days ago.

555
00:40:15,320 --> 00:40:18,360
So you know, for Jack's point as well about being sloppy, I mean, we already brought this

556
00:40:18,360 --> 00:40:19,880
out earlier.

557
00:40:19,880 --> 00:40:24,240
Like this is an easy broadside for me to fire because it mainly falls, I think, on my desk

558
00:40:24,240 --> 00:40:29,240
as somebody who works very closely with organizing test nets and validators and does a lot of

559
00:40:29,240 --> 00:40:36,200
the, shall we say janitorial, you know, stuff like just like bumping dependencies, moving

560
00:40:36,200 --> 00:40:37,200
stuff.

561
00:40:37,200 --> 00:40:40,520
Janitorial, yeah, it's like, you know, it's really boring stuff.

562
00:40:40,520 --> 00:40:44,920
Well, I mean, the thing is, you got like, we've got like the name service, we've got

563
00:40:44,920 --> 00:40:49,120
Ed's secret projects in the, in the, in the works, you know, and I obviously want to go

564
00:40:49,120 --> 00:40:54,000
work on smart contracts and work on the fun sexy stuff, but if we don't got a chain, we

565
00:40:54,000 --> 00:40:55,280
haven't got anything to deploy on, right?

566
00:40:55,280 --> 00:40:57,000
So you end up doing the janitorial stuff.

567
00:40:57,000 --> 00:41:03,920
But then yeah, I think we've gotten lax with like focusing on like Jack says, focusing

568
00:41:03,920 --> 00:41:06,800
on the new shiny stuff that we're working on.

569
00:41:06,800 --> 00:41:11,400
And maybe like needing to go like, okay, no, this is janitorial, but just because it's

570
00:41:11,400 --> 00:41:16,800
boring and it takes five minutes to bump dependencies, and then it takes like a week to get it tested,

571
00:41:16,800 --> 00:41:19,520
you know, blocks, isn't it?

572
00:41:19,520 --> 00:41:20,520
Yeah.

573
00:41:20,520 --> 00:41:24,800
What's that?

574
00:41:24,800 --> 00:41:28,320
Andrew, that's blocks, isn't it?

575
00:41:28,320 --> 00:41:35,360
So Jack's, Jack's paying attention to the, to one of the many test scenarios I think

576
00:41:35,360 --> 00:41:36,880
that's being run in the background.

577
00:41:36,880 --> 00:41:37,880
Yeah.

578
00:41:37,880 --> 00:41:40,800
So the height plus plus upgrade is now working.

579
00:41:40,800 --> 00:41:45,160
Is this the one with one of the validators still on the previous genesis?

580
00:41:45,160 --> 00:41:49,160
No, but we don't anticipate that one to be an issue.

581
00:41:49,160 --> 00:41:57,760
This is starting from 99 with an export from 90s of the normalized export from 97.

582
00:41:57,760 --> 00:41:59,760
We have blocks.

583
00:41:59,760 --> 00:42:05,640
This, this 512 gig Ram instance with 64 cores is getting a huge workout today.

584
00:42:05,640 --> 00:42:09,360
I don't think it'd be expected to be, to be running all day.

585
00:42:09,360 --> 00:42:11,840
You got to start licensing that out.

586
00:42:11,840 --> 00:42:12,840
Yeah.

587
00:42:12,840 --> 00:42:17,240
There's probably 20 other chains that would be potential customers for that, for that service.

588
00:42:17,240 --> 00:42:23,120
Well, you know, I mean, this is the, we have an ICF contract that's covering basically

589
00:42:23,120 --> 00:42:29,440
all of our developer time on the IBC testing framework.

590
00:42:29,440 --> 00:42:33,280
I think that the base of what we've built, we've been able to show off today, which has

591
00:42:33,280 --> 00:42:38,880
been really exciting, but this builds on a ton of work that we as a team have done for

592
00:42:38,880 --> 00:42:42,080
years, like the last year, year and a half.

593
00:42:42,080 --> 00:42:46,680
It started with the framework that we built to test IBC and bring that to market that

594
00:42:46,680 --> 00:42:48,720
is in the go relayer.

595
00:42:48,720 --> 00:42:53,360
That spins up to individual node chains and connects them with a relayer and runs sort

596
00:42:53,360 --> 00:42:55,640
of a standard set of acceptance tests.

597
00:42:55,640 --> 00:42:58,040
We've been using that to drive relayer development.

598
00:42:58,040 --> 00:43:01,120
We used it to drive IBC development.

599
00:43:01,120 --> 00:43:06,720
I took the lessons learned there and built out a couple of other tools, but the biggest

600
00:43:06,720 --> 00:43:09,600
one was the test framework for Horcrux.

601
00:43:09,600 --> 00:43:15,280
I sort of inherited this legacy piece of software from the Polychain team.

602
00:43:15,280 --> 00:43:18,880
It was really cool, but it was written by a C++ engineer who didn't really know how

603
00:43:18,880 --> 00:43:23,800
to write go and it was like all standard live, very hard to get through.

604
00:43:23,800 --> 00:43:28,920
I thought the best thing that I can do is just verify the functioning of this and ensure

605
00:43:28,920 --> 00:43:31,080
that it does exactly what it says.

606
00:43:31,080 --> 00:43:36,160
We built this framework where it spun up a whole chain, upgraded one of the validators

607
00:43:36,160 --> 00:43:40,880
to a three-century, three-core-carts node instance and ensured there was no slashing,

608
00:43:40,880 --> 00:43:45,720
that we were missing blocks, stuff like that.

609
00:43:45,720 --> 00:43:51,800
That process of spinning, that took me about six to eight months of development time to

610
00:43:51,800 --> 00:43:53,920
do through last year.

611
00:43:53,920 --> 00:43:56,720
I tried a couple of different Docker testing frameworks.

612
00:43:56,720 --> 00:43:58,880
I tried a couple of different ways of doing it.

613
00:43:58,880 --> 00:44:06,960
First I did it with fixtures, then we sort of did it all with the containers themselves.

614
00:44:06,960 --> 00:44:10,760
The big thing that we did was we switched to job containers.

615
00:44:10,760 --> 00:44:14,840
We're not using any of the underlying Go APIs really for any of this stuff.

616
00:44:14,840 --> 00:44:17,160
It just treats the containers as a black box.

617
00:44:17,160 --> 00:44:25,360
It's like whatever you expose as far as whether it's RPC endpoints or command-line tools,

618
00:44:25,360 --> 00:44:27,920
whatever you expose, that's what we're going to use against.

619
00:44:27,920 --> 00:44:31,560
It is like if you're integration testing the system, that's how you attack it as a

620
00:44:31,560 --> 00:44:32,560
client.

621
00:44:32,560 --> 00:44:36,360
Why this will do it that way?

622
00:44:36,360 --> 00:44:39,560
I've had a number of discussions with folks in IBC about doing this.

623
00:44:39,560 --> 00:44:46,320
The GORAC team wants to contribute to this, but it wasn't until Andrew finished up the

624
00:44:46,320 --> 00:44:48,360
Hork Rucksery factor after working on the test.

625
00:44:48,360 --> 00:44:49,600
He was like, this test suite is great.

626
00:44:49,600 --> 00:44:55,560
I was like, if you like that test suite, we've got this ridiculous one.

627
00:44:55,560 --> 00:45:00,200
It's the matrix of all the IBC-enabled chains and all of the relayer implementations that

628
00:45:00,200 --> 00:45:04,960
we need to make sure that they all work together so that we don't have splits in this network.

629
00:45:04,960 --> 00:45:09,520
It's like, oh, well, yeah, that's a huge job.

630
00:45:09,520 --> 00:45:12,360
Looking forward to doing a lot more with this.

631
00:45:12,360 --> 00:45:15,480
We have another engineer starting here in a couple of weeks who I'm looking forward to

632
00:45:15,480 --> 00:45:20,560
also putting on this and hoping to get some more contributions from elsewhere in the ecosystem.

633
00:45:20,560 --> 00:45:25,520
I think that as far as a developer tool, if we want people to be able to quickly and easily

634
00:45:25,520 --> 00:45:31,200
launch chains and test these complex cross-chain contract call integrations, you need something

635
00:45:31,200 --> 00:45:32,200
like this.

636
00:45:32,200 --> 00:45:34,840
I think it's an invaluable developer tool.

637
00:45:34,840 --> 00:45:38,520
I'm really glad we've gotten to flex it today.

638
00:45:38,520 --> 00:45:49,880
Is that machine that you're using, Jack, is that a local machine or is that cloud-based?

639
00:45:49,880 --> 00:45:50,960
Google compute.

640
00:45:50,960 --> 00:45:54,440
We have some fairly beefy limits.

641
00:45:54,440 --> 00:45:59,120
That's an understanding, I think, right?

642
00:45:59,120 --> 00:46:00,120
How much out of interest?

643
00:46:00,120 --> 00:46:05,200
Darryl asked how much does that cost in compute per second or something like that?

644
00:46:05,200 --> 00:46:07,400
It's, I think, 10,000 a month.

645
00:46:07,400 --> 00:46:09,760
5,000 to 10,000 a month, yeah.

646
00:46:09,760 --> 00:46:10,760
Yeah.

647
00:46:10,760 --> 00:46:11,760
Nice.

648
00:46:11,760 --> 00:46:14,920
I thought my R5s were pretty expensive, but that's a different level.

649
00:46:14,920 --> 00:46:15,920
Yeah.

650
00:46:15,920 --> 00:46:19,720
I think StrangeLive is doing on the order of 50,000 to 70,000 a month in cloud spend at

651
00:46:19,720 --> 00:46:20,720
this point.

652
00:46:20,720 --> 00:46:23,520
Sir, did you come to your house for that?

653
00:46:23,520 --> 00:46:26,520
Did you guys get hand-delivered invoices from?

654
00:46:26,520 --> 00:46:27,520
No.

655
00:46:27,520 --> 00:46:30,160
We barely get anyone to respond.

656
00:46:30,160 --> 00:46:35,480
Kelsey Hightower, the lead developer evangelist, just talks shit on all of our products.

657
00:46:35,480 --> 00:46:39,400
Google doesn't give a fuck about our business.

658
00:46:39,400 --> 00:46:46,320
What did he very publicly, like on Twitter the other day, say, like, hang on, here is

659
00:46:46,320 --> 00:46:50,760
like one use case for crypto that I agree with and everyone's like, fucking, Kelsey,

660
00:46:50,760 --> 00:46:51,760
Hightower's gone D-Gen.

661
00:46:51,760 --> 00:46:52,760
It's all over it.

662
00:46:52,760 --> 00:46:56,160
It's like, not yet.

663
00:46:56,160 --> 00:47:00,280
I still think he's trolling us because he's trolled us a few times before.

664
00:47:00,280 --> 00:47:02,680
I am a huge Kelsey fan.

665
00:47:02,680 --> 00:47:08,360
When I started in tech, I saw Kelsey do a live demo and I was like, whatever he's teaching,

666
00:47:08,360 --> 00:47:13,000
I want to learn and I want to be a developer evangelist.

667
00:47:13,000 --> 00:47:17,000
Kelsey has been influential in my career personally, so seeing him so anti-blockchains

668
00:47:17,000 --> 00:47:18,600
hurts a little bit.

669
00:47:18,600 --> 00:47:22,920
I wish he was more pro-block.

670
00:47:22,920 --> 00:47:27,440
I tried to explain where we were up to, to the smartest engineer I've ever worked with,

671
00:47:27,440 --> 00:47:33,800
his previous CTO of mine, and he went through the dot-com boom, the last time that sort

672
00:47:33,800 --> 00:47:34,800
of thing.

673
00:47:34,800 --> 00:47:38,760
After taking him through all the steps that led us up to where we were, I guess about

674
00:47:38,760 --> 00:47:45,360
like eight o'clock this morning, his conclusion was just one word and it was oof.

675
00:47:45,360 --> 00:47:53,040
He's not a blockchain guy, so he was like, is it proof of work, proof of stake?

676
00:47:53,040 --> 00:47:54,880
Okay, so you've got to validate this there.

677
00:47:54,880 --> 00:47:57,960
We got through all those questions and he got to the end of it and he was just like,

678
00:47:57,960 --> 00:48:00,640
oof, all right, how many forks you got?

679
00:48:00,640 --> 00:48:03,000
I was like, we've got 125 forks, my guy.

680
00:48:03,000 --> 00:48:05,000
He was just like, oof.

681
00:48:05,000 --> 00:48:08,200
He was like, good luck with that.

682
00:48:08,200 --> 00:48:10,920
Anyway, let's get a drink next time you're up.

683
00:48:10,920 --> 00:48:12,920
I was like, thanks.

684
00:48:12,920 --> 00:48:16,920
So it's like, it's a wisdom, but oof.

685
00:48:16,920 --> 00:48:20,400
Would you rather recover this or would you rather recover a downed raft cluster?

686
00:48:20,400 --> 00:48:28,120
Well, I used to work with him on a large project that had a massive Apache Kafka and Cassandra

687
00:48:28,120 --> 00:48:36,360
cluster and our Cassandra driver library went out of maintenance and I had to rewrite the

688
00:48:36,360 --> 00:48:43,720
database driver, which was in Lisp because obviously I'm a Lisp weenie, so that was.

689
00:48:43,720 --> 00:48:46,720
Why did you do that to yourself?

690
00:48:46,720 --> 00:48:49,240
You could have made better decisions.

691
00:48:49,240 --> 00:48:55,200
No, no, why did you do that to your fellow engineers down the line?

692
00:48:55,200 --> 00:48:57,720
Because we're a closure shop.

693
00:48:57,720 --> 00:48:59,760
We all wrote Lisp, we all like it.

694
00:48:59,760 --> 00:49:02,320
We all think it's the best language.

695
00:49:02,320 --> 00:49:07,120
Self-evidently it, well, okay, JVM closure, no.

696
00:49:07,120 --> 00:49:12,760
Closure RS, which I still believe one day might actually have a working version.

697
00:49:12,760 --> 00:49:15,280
Then you're going to see smart contracts in Lisp for me.

698
00:49:15,280 --> 00:49:18,680
That's a promise.

699
00:49:18,680 --> 00:49:21,480
Does it compile to Waza?

700
00:49:21,480 --> 00:49:23,840
Well, it's valid, it's valid, Rust, isn't it?

701
00:49:23,840 --> 00:49:27,840
So yeah, then you could jump it to Waza, I guess.

702
00:49:27,840 --> 00:49:28,840
Cool, great.

703
00:49:28,840 --> 00:49:33,240
I look forward to never reading any of that code.

704
00:49:33,240 --> 00:49:37,480
Yeah, well, but you know what, at least the abstract syntax tree will be exactly the same

705
00:49:37,480 --> 00:49:44,160
as the code itself and therefore it will be everything in perfect balance as is the Lisp

706
00:49:44,160 --> 00:49:45,160
way.

707
00:49:45,160 --> 00:49:55,800
I mean, I'm over here throwing shade, but I actually enjoy using closure and I'm over

708
00:49:55,800 --> 00:49:59,640
here like looking at, you know, reading up on Elixir and other functional programming

709
00:49:59,640 --> 00:50:02,480
languages so I can only throw so much shade.

710
00:50:02,480 --> 00:50:04,680
Yeah, exactly.

711
00:50:04,680 --> 00:50:06,120
It's a way of life, it's a way of life.

712
00:50:06,120 --> 00:50:09,120
Like if it takes hold of you, it's a certain way of doing things.

713
00:50:09,120 --> 00:50:15,280
Just like I think like anything else like programming languages like musical instruments,

714
00:50:15,280 --> 00:50:17,480
whatever, like some speak to you, some don't.

715
00:50:17,480 --> 00:50:24,680
For me, like I can take a leave Python, I've written a lot of JavaScript type script for

716
00:50:24,680 --> 00:50:28,640
money, but I'm kind of broadly indifferent to it.

717
00:50:28,640 --> 00:50:33,840
Lisp is the only language that I actually love, like as in it makes me happy to work

718
00:50:33,840 --> 00:50:34,840
in Lisp.

719
00:50:34,840 --> 00:50:37,000
I don't think there's another language I feel that way.

720
00:50:37,000 --> 00:50:38,000
I quite like Rust.

721
00:50:38,000 --> 00:50:39,000
Rust is great.

722
00:50:39,000 --> 00:50:43,520
I think Haskell is very interesting, although I have to relearn it every time I use it,

723
00:50:43,520 --> 00:50:47,800
unfortunately, because it's just really complex.

724
00:50:47,800 --> 00:50:51,200
But I don't know, maybe not everybody gets emotional about programming languages, maybe

725
00:50:51,200 --> 00:50:52,200
that's just me.

726
00:50:52,200 --> 00:50:53,200
I find it very hard.

727
00:50:53,200 --> 00:50:56,440
Oh no, I definitely do.

728
00:50:56,440 --> 00:51:00,240
I fell in love with Go early and I just like writing Go.

729
00:51:00,240 --> 00:51:03,160
I would prefer not to have to have to write anything else, but I have to screw up with

730
00:51:03,160 --> 00:51:08,480
Rust, I have to screw up with JavaScript, I have to screw up with this solidity nonsense.

731
00:51:08,480 --> 00:51:09,480
Anyway, yeah.

732
00:51:09,480 --> 00:51:15,000
Is there anything else in the Judo Hall that we want to cover?

733
00:51:15,000 --> 00:51:19,920
I think probably pretty much being over what we know so far, right guys?

734
00:51:19,920 --> 00:51:25,040
I think it would be worth knowing when do we think that the fix will start going out?

735
00:51:25,040 --> 00:51:27,000
Can we expect Judo to be running in a week?

736
00:51:27,000 --> 00:51:33,560
I think 48 hours is a little bit more realistic.

737
00:51:33,560 --> 00:51:44,360
My thought is that once we verify that a particular path forward is valid, we should give validators

738
00:51:44,360 --> 00:51:46,680
24 hours and restart after that.

739
00:51:46,680 --> 00:51:57,840
I think that this path of export at 97, increment the height to 98, reset the contract state,

740
00:51:57,840 --> 00:51:59,040
that works.

741
00:51:59,040 --> 00:52:03,360
I think we should go for that.

742
00:52:03,360 --> 00:52:08,400
Getting the network back up in 24 hours as of right now would be, I just put out a call

743
00:52:08,400 --> 00:52:11,480
and a call in court too to do that.

744
00:52:11,480 --> 00:52:16,400
I guess it depends on what kind of participation we get from all the validators in that timeframe,

745
00:52:16,400 --> 00:52:17,400
right?

746
00:52:17,400 --> 00:52:18,400
Yeah.

747
00:52:18,400 --> 00:52:24,360
I think everyone is pretty aware that this is going on.

748
00:52:24,360 --> 00:52:29,200
We were able to get, I don't know if you guys saw this, I'm sure some of you were a

749
00:52:29,200 --> 00:52:33,280
part of this, but at the beginning of the incident, Demi was running around chasing

750
00:52:33,280 --> 00:52:39,280
every, well, all of us were chasing validators and trying to put together a list of what

751
00:52:39,280 --> 00:52:45,480
software versions are on, which by the way, fascinating the way those numbers turned out,

752
00:52:45,480 --> 00:52:48,080
right at the one third, just there.

753
00:52:48,080 --> 00:52:52,720
We had the same thing at most, it was 66.0% or whatever it was.

754
00:52:52,720 --> 00:52:54,400
It was always one away, right?

755
00:52:54,400 --> 00:52:56,200
Same situation here.

756
00:52:56,200 --> 00:52:57,200
Just insane.

757
00:52:57,200 --> 00:53:04,440
Okay, so we were able to get in touch with everyone, I think 98% of the validators said

758
00:53:04,440 --> 00:53:09,440
probably within an hour and a half, two hours of really starting that reach out, so probably

759
00:53:09,440 --> 00:53:12,760
four hours after the downtime started.

760
00:53:12,760 --> 00:53:14,360
That was incredibly impressive in my mind.

761
00:53:14,360 --> 00:53:18,320
I don't think we'll have a problem getting the VALSET back up.

762
00:53:18,320 --> 00:53:19,720
All of the top validators are engaged.

763
00:53:19,720 --> 00:53:22,360
They've had open lens of communication with us.

764
00:53:22,360 --> 00:53:28,480
I think 24 hours, we could do a quicker restart if we really want to push things, but I think

765
00:53:28,480 --> 00:53:29,680
24 hours.

766
00:53:29,680 --> 00:53:35,040
So 5 p.m. PST tomorrow.

767
00:53:35,040 --> 00:53:38,960
Andrew, how do you feel about that?

768
00:53:38,960 --> 00:53:42,640
Are you talking about this live right now then?

769
00:53:42,640 --> 00:53:46,440
Yes, I am literally talking about this live right now.

770
00:53:46,440 --> 00:53:52,560
I think we could all just see this in the corner of the screen, the same notification,

771
00:53:52,560 --> 00:53:54,560
like, okay, this is happening.

772
00:53:54,560 --> 00:53:57,240
So you see a bit of years of the channel.

773
00:53:57,240 --> 00:54:05,800
You are literally seeing some decision-making happening on the stream for what we're going

774
00:54:05,800 --> 00:54:08,400
to be working on tomorrow by the look of it.

775
00:54:08,400 --> 00:54:10,160
Gary, John asks a great question.

776
00:54:10,160 --> 00:54:13,120
Will Luper Kelly be done at the same time while it's down, or will we wait until it's

777
00:54:13,120 --> 00:54:16,160
back up nicely and then try again?

778
00:54:16,160 --> 00:54:19,400
Or you're going to do this while it's down, and in fact, we're probably going to modify

779
00:54:19,400 --> 00:54:24,680
the governance proposal in-state so that it doesn't create an upgrade because we've already

780
00:54:24,680 --> 00:54:27,240
done the upgrade at this point.

781
00:54:27,240 --> 00:54:33,560
Luper Kelly has the nice side effect of fixing the Stargate queries issue.

782
00:54:33,560 --> 00:54:35,520
So we verified that fix.

783
00:54:35,520 --> 00:54:39,880
We verified that the new version fixes the issue with the contract.

784
00:54:39,880 --> 00:54:46,720
So the best thing that we can do is just move forward with the deployment.

785
00:54:46,720 --> 00:54:54,440
Have we rolled anything else into that now, Jack, in addition to what was already proposed?

786
00:54:54,440 --> 00:54:58,640
No, in fact, we haven't.

787
00:54:58,640 --> 00:55:04,040
The proposed binary fixes this issue, and it happened hours before the upgrade was supposed

788
00:55:04,040 --> 00:55:05,040
to happen.

789
00:55:05,040 --> 00:55:06,040
Kind of hilarious.

790
00:55:06,040 --> 00:55:14,480
I think Andrew actually was running the new tag binary which has the Cosmosm security

791
00:55:14,480 --> 00:55:17,800
fix from today rolled into it, right?

792
00:55:17,800 --> 00:55:20,120
Yes, it does.

793
00:55:20,120 --> 00:55:21,120
That's correct.

794
00:55:21,120 --> 00:55:25,400
So we're getting that one as a twofer for free.

795
00:55:25,400 --> 00:55:32,240
So then in that case, so it's 2.3, and it's re-tagged as three, and it includes a new

796
00:55:32,240 --> 00:55:40,720
security patch as well as removes some chaff about that contract that messed everything

797
00:55:40,720 --> 00:55:44,000
up.

798
00:55:44,000 --> 00:55:46,320
So what's happened?

799
00:55:46,320 --> 00:55:47,320
Nothing.

800
00:55:47,320 --> 00:55:48,760
Everyone's on mute.

801
00:55:48,760 --> 00:55:50,760
They're talking while they're on mute.

802
00:55:50,760 --> 00:55:51,760
Oh, sorry.

803
00:55:51,760 --> 00:55:52,760
Yeah.

804
00:55:52,760 --> 00:55:53,760
Yeah, pretty much.

805
00:55:53,760 --> 00:55:54,760
Sorry.

806
00:55:54,760 --> 00:55:58,920
I just instinctively do it just like hit the thing.

807
00:55:58,920 --> 00:56:03,360
Chelsea's the only one who needs to put his thing on mute so we don't hear his clicking.

808
00:56:03,360 --> 00:56:04,360
You know what?

809
00:56:04,360 --> 00:56:05,840
I don't have any clickers with me this time.

810
00:56:05,840 --> 00:56:08,920
So I've been like the entire time over here like, is she a bird?

811
00:56:08,920 --> 00:56:10,920
Yeah, he looks so nervous.

812
00:56:10,920 --> 00:56:11,920
He's a bird.

813
00:56:11,920 --> 00:56:12,920
Yeah.

814
00:56:12,920 --> 00:56:18,320
What time is it of running the side of the pond right now?

815
00:56:18,320 --> 00:56:19,960
It's nearly 1am.

816
00:56:19,960 --> 00:56:21,320
It's pretty late.

817
00:56:21,320 --> 00:56:27,960
So Europeans will be nearly 2am or 3am depending on which part of Europe.

818
00:56:27,960 --> 00:56:35,160
So it's just afternoon restart is a tough time because Asia is still waking up.

819
00:56:35,160 --> 00:56:39,080
Yeah, I would probably say if you could pull it back a couple of hours, it would make the

820
00:56:39,080 --> 00:56:41,560
Europeans happy but you know, such is life.

821
00:56:41,560 --> 00:56:44,360
I mean, 4am Sydney is usually the best time.

822
00:56:44,360 --> 00:56:46,640
It works for everybody else.

823
00:56:46,640 --> 00:56:47,640
It works for everybody.

824
00:56:47,640 --> 00:56:49,800
It's kind of what we're trying to do.

825
00:56:49,800 --> 00:56:53,480
That time slot works exactly for everybody except.

826
00:56:53,480 --> 00:56:54,480
Hey.

827
00:56:54,480 --> 00:56:56,360
Just real quick.

828
00:56:56,360 --> 00:57:01,360
I'm watching the comments here, what I wanted to address for folks.

829
00:57:01,360 --> 00:57:05,520
Ramara asks, any issues of ledger for restarting at a different time?

830
00:57:05,520 --> 00:57:06,520
Short answer?

831
00:57:06,520 --> 00:57:07,520
No.

832
00:57:07,520 --> 00:57:08,520
Long answer?

833
00:57:08,520 --> 00:57:09,520
Absolutely not.

834
00:57:09,520 --> 00:57:10,920
Nothing to worry about with your ledger.

835
00:57:10,920 --> 00:57:12,920
Funds are set for.

836
00:57:12,920 --> 00:57:19,080
Yeah, like that's there's no co-change or anything that would affect ledger operation.

837
00:57:19,080 --> 00:57:26,560
I think maybe do you think maybe that comment was actually referring to the state as the

838
00:57:26,560 --> 00:57:27,560
ledger?

839
00:57:27,560 --> 00:57:29,280
Oh, it's a tribute to ledger.

840
00:57:29,280 --> 00:57:30,280
Yeah.

841
00:57:30,280 --> 00:57:33,640
Not so much the ledger.

842
00:57:33,640 --> 00:57:35,680
Yeah, that's actually a great question.

843
00:57:35,680 --> 00:57:41,760
And in fact, one that we did not have a concrete answer to until like just a second ago when

844
00:57:41,760 --> 00:57:48,160
we tested it, there, you know, these networks are designed to be down.

845
00:57:48,160 --> 00:57:51,920
There have been down to tendermint networks a number of times.

846
00:57:51,920 --> 00:57:56,880
I think that one thing that we're doing differently in this Juneau upgrade is we're incrementing

847
00:57:56,880 --> 00:57:59,880
the block height and keeping the chain ID.

848
00:57:59,880 --> 00:58:06,000
This particular style of upgrade has never been attempted at scale before.

849
00:58:06,000 --> 00:58:09,880
So there were a number of things that we were worried about.

850
00:58:09,880 --> 00:58:15,880
But I think that, you know, we've just kind of verified that we're good to go there.

851
00:58:15,880 --> 00:58:18,480
So that's exciting.

852
00:58:18,480 --> 00:58:22,160
Jack, one thing we were talking about before you rejoined was like how this upgrade or

853
00:58:22,160 --> 00:58:27,400
at least the 210 with the binary patch was slightly different type of approach and upgrade

854
00:58:27,400 --> 00:58:31,800
versus regular ones where people would just, you know, pull the latest GitHub and compile,

855
00:58:31,800 --> 00:58:32,800
right?

856
00:58:32,800 --> 00:58:34,360
It had the library structure and those types of things.

857
00:58:34,360 --> 00:58:40,280
And since it was slightly different approach and you have a very wild, you know, 125 validator

858
00:58:40,280 --> 00:58:43,960
teams at different time zones and different communication and you have some people are

859
00:58:43,960 --> 00:58:46,680
not even in the discord and things like that.

860
00:58:46,680 --> 00:58:47,880
These networks are hard, right?

861
00:58:47,880 --> 00:58:51,880
And so when you have things that are slightly outside of normal, which we saw it at most

862
00:58:51,880 --> 00:58:56,000
and we've seen other types of things or test nets, shit happens, right?

863
00:58:56,000 --> 00:58:59,280
And we saw that even saw that you said you said you surprised the Excel results.

864
00:58:59,280 --> 00:59:05,120
I was completely surprised the Excel results on even, you know, top 10 validators that

865
00:59:05,120 --> 00:59:08,000
were running the wrong binary and a lot of them, right?

866
00:59:08,000 --> 00:59:11,120
Like a decent percentage of that.

867
00:59:11,120 --> 00:59:15,480
I mean, that's actually, I think honestly, one of the strengths of these decentralized

868
00:59:15,480 --> 00:59:18,640
networks is like they're heterogenous.

869
00:59:18,640 --> 00:59:21,400
They're not homogeneous, you know, not every node is the same.

870
00:59:21,400 --> 00:59:22,640
That makes them hard to attack.

871
00:59:22,640 --> 00:59:25,640
Like that's a feature, not a bug in a lot of ways.

872
00:59:25,640 --> 00:59:30,000
Is there any difference of feature?

873
00:59:30,000 --> 00:59:31,000
That wasn't a version difference.

874
00:59:31,000 --> 00:59:32,480
That was a build tag feature.

875
00:59:32,480 --> 00:59:33,480
Okay.

876
00:59:33,480 --> 00:59:34,480
It was a build tag difference.

877
00:59:34,480 --> 00:59:35,480
Right.

878
00:59:35,480 --> 00:59:41,440
And there was a lot of variety on how people provision their servers.

879
00:59:41,440 --> 00:59:43,280
Honestly for network security, that's great.

880
00:59:43,280 --> 00:59:49,160
If you've got a hundred different ways people are deploying, makes it way harder to break

881
00:59:49,160 --> 00:59:52,280
in and steal keys.

882
00:59:52,280 --> 00:59:57,920
And you know, the fact, I think that what we should take away from this as a core team

883
00:59:57,920 --> 01:00:03,960
is that if we have some specific build things, we need to like bold those, make sure people

884
01:00:03,960 --> 01:00:11,320
know what it is because, you know, I've been building tags that go ledger for years.

885
01:00:11,320 --> 01:00:13,480
And I think a lot of other validators have too.

886
01:00:13,480 --> 01:00:19,760
And luckily Andrew read the things the right way and we were running the right version,

887
01:00:19,760 --> 01:00:22,240
but we easily couldn't have been.

888
01:00:22,240 --> 01:00:28,080
So the Frey and I were actually, we've been talking recently about sort of updating the

889
01:00:28,080 --> 01:00:37,360
Juno docs to sort of better represent the upgrade path for Juno given, you know, that

890
01:00:37,360 --> 01:00:39,720
not everything's built from source as well.

891
01:00:39,720 --> 01:00:41,400
And with those default tags.

892
01:00:41,400 --> 01:00:44,560
So we just hadn't got there yet.

893
01:00:44,560 --> 01:00:45,560
Yeah.

894
01:00:45,560 --> 01:00:46,560
Yeah.

895
01:00:46,560 --> 01:00:50,840
I mean, it's, yeah, I think that like anytime there's an issue with one of these networks

896
01:00:50,840 --> 01:00:56,600
like this, somebody's been like, try to make it happen before.

897
01:00:56,600 --> 01:01:00,600
And this time different, you know, a lot of the things that we've talked about in the

898
01:01:00,600 --> 01:01:04,320
past in various channels like come back to buy this year.

899
01:01:04,320 --> 01:01:06,320
Yeah, it happens.

900
01:01:06,320 --> 01:01:10,040
You learn, you learn from it, you put in process and you try to do better the next time.

901
01:01:10,040 --> 01:01:15,080
And what I have learned from this experience is like exactly what I learned during the

902
01:01:15,080 --> 01:01:19,200
Prop 16 experience, the people working on this network are awesome.

903
01:01:19,200 --> 01:01:26,000
We all like working together a lot and we're very effective at, you know, async collaboration.

904
01:01:26,000 --> 01:01:31,880
And there's enough of us where we have full 24 seven coverage on different pieces and

905
01:01:31,880 --> 01:01:36,240
you know, the people who are up at any given time can are very effective.

906
01:01:36,240 --> 01:01:40,080
That's more than we had for cosmos for years.

907
01:01:40,080 --> 01:01:46,320
And I think it's worth noting too that we've got like a lot of incredible developers that

908
01:01:46,320 --> 01:01:49,720
are highly involved with Juno as well.

909
01:01:49,720 --> 01:01:56,280
I think, yeah, just actually looking at, and you know, people passing the baton as well

910
01:01:56,280 --> 01:02:01,720
as the time zones go through is incredible to see as well.

911
01:02:01,720 --> 01:02:09,200
So, you know, handing over the lead position for coordination to another lead dev until

912
01:02:09,200 --> 01:02:13,520
you know, the circles back around as we go through the time zones and people obviously

913
01:02:13,520 --> 01:02:14,520
need to sleep.

914
01:02:14,520 --> 01:02:17,520
You can't just stay awake for 24 seven unless you're the prey.

915
01:02:17,520 --> 01:02:23,320
Oh, I think I think max, I was gonna say, I think max did like, I don't know.

916
01:02:23,320 --> 01:02:31,520
Max is still in this zoom call with me and Andrew and Joe Abbey right now.

917
01:02:31,520 --> 01:02:33,000
He's been there for nine hours.

918
01:02:33,000 --> 01:02:34,000
Yeah, right.

919
01:02:34,000 --> 01:02:39,720
But bear in mind also like max was the first person who identified the smart contract call

920
01:02:39,720 --> 01:02:43,560
like when my, when my pager went off and I was like three hours from my desk.

921
01:02:43,560 --> 01:02:45,040
Oh, it was max.

922
01:02:45,040 --> 01:02:46,040
Yeah.

923
01:02:46,040 --> 01:02:47,040
I had a DM from.

924
01:02:47,040 --> 01:02:49,400
Oh, with dude.

925
01:02:49,400 --> 01:02:52,680
I think I've seen a smart contract that looks suspect.

926
01:02:52,680 --> 01:02:55,480
Here's the min scan TX and I looked at it.

927
01:02:55,480 --> 01:03:01,640
I looked at it and my first reaction was I bet this is that ABCI Stargate thing.

928
01:03:01,640 --> 01:03:04,200
And then it was low and bold.

929
01:03:04,200 --> 01:03:09,160
But it's it's always max max is always first on anything else.

930
01:03:09,160 --> 01:03:14,760
Like he always beats all of the rest of the devs to like pointing to the thing in the

931
01:03:14,760 --> 01:03:17,560
direction of where the actual problem is.

932
01:03:17,560 --> 01:03:23,200
Max is like, it's just like everybody in the team, I think, and Juno like has a kind

933
01:03:23,200 --> 01:03:28,200
of perspective they bring to the table, you know, max is incredible.

934
01:03:28,200 --> 01:03:32,600
The amount of support he gives to the community is just unparalleled.

935
01:03:32,600 --> 01:03:39,760
Well, I mean, it's unmatched like with any of the other, you know, core contributors

936
01:03:39,760 --> 01:03:47,160
or community members or anyone in Juno is definitely puts in the most, you know, go

937
01:03:47,160 --> 01:03:49,000
community support for sure.

938
01:03:49,000 --> 01:03:55,520
And yeah, the fact that he comes up with these fines in min scanner stuff is pretty amazing.

939
01:03:55,520 --> 01:03:58,160
Like, is is max max?

940
01:03:58,160 --> 01:03:59,160
I know you can hear us.

941
01:03:59,160 --> 01:04:00,160
Are you are you a developer?

942
01:04:00,160 --> 01:04:07,200
I don't know if he's a developer or not, but he certainly finds a lot of stuff.

943
01:04:07,200 --> 01:04:10,200
It takes a while for comments to come through.

944
01:04:10,200 --> 01:04:13,520
Yeah, I don't know that that's a great question.

945
01:04:13,520 --> 01:04:17,440
I believe max is one person, funny enough.

946
01:04:17,440 --> 01:04:23,280
But I think that kind of speaks to Juno being this really truly community led project.

947
01:04:23,280 --> 01:04:28,920
Like I saw that like I always kind of knew that max max Max you're in both of the same

948
01:04:28,920 --> 01:04:33,120
streams I'm in right now.

949
01:04:33,120 --> 01:04:37,240
You know, I think I saw that really truly the scale of it for the first time with Prop

950
01:04:37,240 --> 01:04:39,200
16, but I always kind of knew that.

951
01:04:39,200 --> 01:04:42,600
And it's something that Juno does better than any other cosmos chain.

952
01:04:42,600 --> 01:04:44,800
And I think it has been a differentiator.

953
01:04:44,800 --> 01:04:48,920
It's been something that's enabled us to sort of like push through this issue because like,

954
01:04:48,920 --> 01:04:51,520
honestly, this is blockchain dev worse nightmare.

955
01:04:51,520 --> 01:04:55,120
It's about about as it gets.

956
01:04:55,120 --> 01:05:02,640
And you know, it's easy to get into a place where it's really hard to recover that chain.

957
01:05:02,640 --> 01:05:07,000
And it's been cool to see this community kind of push through that, find the answers and

958
01:05:07,000 --> 01:05:08,280
get this thing moving again.

959
01:05:08,280 --> 01:05:12,640
And like, we're going to get it.

960
01:05:12,640 --> 01:05:13,640
Yeah.

961
01:05:13,640 --> 01:05:17,520
I mean, like when the when the chain first halted, like I was with my old man and he was

962
01:05:17,520 --> 01:05:22,200
like, what was that message you just picked up?

963
01:05:22,200 --> 01:05:28,800
And I was like, well, pretty much the worst thing that could happen has happened.

964
01:05:28,800 --> 01:05:31,160
And he worked for Oracle for like 25 years.

965
01:05:31,160 --> 01:05:35,120
And I was like, what would be, I was like, a data center is burned down.

966
01:05:35,120 --> 01:05:37,080
That's I think that's like the equivalent.

967
01:05:37,080 --> 01:05:38,840
Like a data center is fully gone.

968
01:05:38,840 --> 01:05:41,080
And he was like, oh, that's pretty bad.

969
01:05:41,080 --> 01:05:42,560
You don't seem too, too worried.

970
01:05:42,560 --> 01:05:47,760
I was like, well, I'm pretty worried, but I also like, I'm 100% certain that we have

971
01:05:47,760 --> 01:05:51,880
the people in the building to have the chain back up.

972
01:05:51,880 --> 01:05:54,000
So the building that burned down.

973
01:05:54,000 --> 01:05:55,000
Well, okay.

974
01:05:55,000 --> 01:05:59,040
The building, the building was his world.

975
01:05:59,040 --> 01:06:01,200
The chain is our world, right?

976
01:06:01,200 --> 01:06:03,360
So his building's burnt down.

977
01:06:03,360 --> 01:06:04,360
Maybe his devs were in the building.

978
01:06:04,360 --> 01:06:05,520
I don't know.

979
01:06:05,520 --> 01:06:11,680
You know, you have to go ask Oracle for that would be a human resources nightmare, I imagine.

980
01:06:11,680 --> 01:06:16,840
But our building is decentralized and therefore you can't burn it down.

981
01:06:16,840 --> 01:06:20,440
And thus is the power of, do you know, the center.

982
01:06:20,440 --> 01:06:23,280
You can only confuse the shithead of it.

983
01:06:23,280 --> 01:06:24,280
Yeah.

984
01:06:24,280 --> 01:06:28,200
If you're like, whatever, maybe the real, I need to stop making this.

985
01:06:28,200 --> 01:06:30,240
I made the same joke every time we have a crisis.

986
01:06:30,240 --> 01:06:31,240
We've had a few of these now.

987
01:06:31,240 --> 01:06:35,600
I'm like, oh, maybe the real, you know, was the friends who made along the way, but also

988
01:06:35,600 --> 01:06:36,600
kind of.

989
01:06:36,600 --> 01:06:41,400
And I think that the real cosmos is the friends we made along.

990
01:06:41,400 --> 01:06:44,200
They're like music themselves.

991
01:06:44,200 --> 01:06:47,000
They're like through talking.

992
01:06:47,000 --> 01:06:49,720
Zucky used to say that all the time.

993
01:06:49,720 --> 01:06:50,720
Yeah.

994
01:06:50,720 --> 01:06:54,720
They go in good company then.

995
01:06:54,720 --> 01:06:57,480
So guys, do we have anything else we want to talk about today?

996
01:06:57,480 --> 01:07:02,480
We're out about, we're coming up on 70 minutes, which is usually where we start to pull the

997
01:07:02,480 --> 01:07:03,480
pin.

998
01:07:03,480 --> 01:07:05,480
Yeah, let's let's pull the pin.

999
01:07:05,480 --> 01:07:29,480
Yeah, I think that's all.