Very good. Well played. I just got trolled, didn't I? Oh shit, we're live.
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent validator teams.
Okay, so hello, everybody. Welcome to Game of Nodes, a weekly podcast from on the cosmos
from independent validator teams from a quite tired group of independent validators this week.
And what are we talking about this week, fellas? So this week we're talking about the Juno network
halt, sort of what went wrong, what the cause was for the halt, where we're at now, the current
testing that's ongoing, and how we're going to get it restarted, I think, if we have a
solution for that yet, which I think we have a couple of ideas. So, Yusuf, we had some other
things we're going to talk about if we had time too. What were they? I think we had some
questions that came in a couple of weeks ago. So I think we had some interest in kind of
understanding how some of us as independent validators got into this business and a little bit
around backgrounds and what it takes to be a core validator for some of the cosmos. And also,
I think there's some questions just around the full time doing this, is this kind of a side
thing and how do you make a living out of it and those types of things? As well as, I think we
had a third topic that was getting into it was a little bit more around, is it lucrative or not?
And how lucrative is it? What's the necessary investments is kind of a goal on it and those
types of areas. So maybe if we have some time, we can get into some of those.
Cool. Yeah, that would be cool.
So I guess to start off with, the tired, the fray is probably going to be the best person
to give us a leading introduction to the halt.
I thought you were going to ask, was the dumbest thing we've seen in crypto this week?
Oh, yeah. Well, yeah. Let's do the question. I think, well, I think, okay, go the fray.
What's the dumbest thing you've seen in crypto this week?
Well, the dumbest thing I've seen in crypto this week was me designed to take a day off
on Tuesday, which actually was only yesterday, wasn't it?
It was all yesterday.
It's been the longest 36 hours. I've, yeah, I think I've, I think I've traveled 1300
kilometers today while, while I'm telling to a phone, burn through 2.5 gig of mobile
data or something like that.
So you have been buzzing across the country in a, well, across the continent in a train
going 300 kilometers an hour whilst troubleshooting and, you know, just generally trying to get
stuff done.
Yeah. A friend of mine said, don't be pissed off because of the cyberpunk future you got.
Wasn't the one you wanted when I told him what was happening. He was like, when are
you supposed to be traveling today? And I was like, yeah, I'm on a high speed train
literally right now. And he was like, you can't be, you can't be sore because the cyberpunk
future you got wasn't the one you wanted, man. This is exactly what you signed up for,
you prick.
So yeah,
I think we basically signed up to never sleep again.
So I'm beginning to tell that way. Yeah.
Yeah. For every chain you add, you lose another hour and a half of your day.
I'm kind of surprised we're all not gray yet. We're all like surprisingly ungray and
surprisingly handsome. I mean, I'm just looking at Schultz actually.
Yeah. Yeah. Exactly.
I think Schultz is like had a haircut or something.
You know, a different angle does a lot to me.
Yeah. It's a different angle this week.
Yeah. I'm in Puerto Rico right now. So straight on this time. I couldn't do it this time,
I suppose.
There you go.
You're outside your studio. I'm outside my studio. Yusef is still on point. And the
phrase back in his hobble.
Yeah.
Back in the hobble. Yeah. So dumbest thing in crypto this week has to be joking aside
that we got hit by the exploit that we were patching six hours before we patched it. That
is devastating. Like the second that I saw my pager, I was literally like, oh, like not
this.
It could not be that if, I mean, obviously it's good to know what it is, but I was like,
if it's that ABCI thing, I'm going to be pissed.
And really, we just randomly chose an upgrade block, right? It could have been, you give
yourself four days because we've, you don't want to make it too close, right? We made
it four days.
Which is going to do it on Monday.
Right on Monday, like first thing Monday morning. And it was like, oh no, we just, you know,
we're going to be late. We were jumping this upgrade on them, relatively short notice. Let's
just give everybody an extra day. What's the worst that could happen?
You gave us three extra days. It was nice.
What's the worst that could happen? Terrible.
Okay. So it has to do with the exploit that was being patched next time, not the exploit
that was patched previously. Correct?
No. So the LupaKalia upgrade patched the exploit we got hit with. So six hours later, and would
have been home and dry. So the WasmD, whatever it is, CWA 002 2022, which got released this
morning, which is bad. It's a zero day in Cosmosm. We've already got a patch in for that
today, in amongst all the other stuff. And that's a separate thing. So we didn't get
hit by that. We got hit by something we already knew about, something that we had a patch
already 29 or 30 days ago. But this thing you might have heard of, Prop 16, happened.
And it meant that our testing and our original release date for that, which was mid-March,
got pushed back because we all had to focus on dealing with the fire that was in front
of us. And we couldn't, it just didn't seem like we could get the validator set or anybody
to focus on getting that, like the final bit of testing signed off and get it out of the
door. Now in hindsight, it's really obvious to say, you know, that was a mistake. We should
have just prioritized the security release. And that is what, I mean, we're going to do
a full root cause from all of this. We've already started pulling together some notes
on like things that we already know, just, you know, we're going to have to be quite
harsh on ourselves, I think, about this. And I think one of the big things that will come
out of it is that in particular, the core devs were very blindsided by Prop 16. And the,
I don't want to go over it too much. I know we talked about it on our episode on Prop 16,
you know, if you don't know whether you legally have to leave a network, like walk away from
it potentially, which, you know, we've all, a lot of us have, you know, had to go and
get legal advice, even if that legal advice ended up being this is completely uncharted
waters, it's up to you, take your risk profile as you see it, you know, in that context,
your biggest priority as a dev sadly is actually not looking at the security upgrade you need
to get out. It's like, you know, am I still working on this project? Like what is actually
going on here with this level of uncertainty? So there's a big kind of continuity question,
I think, that we were sort of blindsided by with Prop 16, because it's such an unusual
thing to happen, like the fact we've got hit with a constitutional crisis effectively,
and then a cyber attack. I mean, I don't think there's many other systems than a proof of
state blockchain where that combination of things is even possible, you know.
So it was a cyber attack. So did they get anything out of it? Did they get anything out of the
fact that we have lost some sleep effectively? Like there was no funds that were stolen.
There was no, just a point was made by presumably some developer because only someone that's
accustomed to Kazim one would be able to achieve what was achieved. Is that right?
All that happened, the fray right is that basically all of the nodes got confused on
that block and just hold it because they didn't know what to do.
Every single node effectively forked. So we create 125 forks of the chain, which is kind
of cool in a twisted way. And then everybody had completely indeterminate app state. That
was done by calling a low level ABCI query in Tenderman. Now, you obviously have to kind
of, you have to know it's there, but if you knew it was there, it's not that hard to exploit
it. So it's not quite like baby's first exploit, but it is like, it's like Fisher price, my
first exploit or my first cause and my as an exploit or something like that. Like it is
and it just happens so happens in this case, it was actually quite a devastating exploit.
I shouldn't be laughing about it so soon, but honestly, I've had very little sleep,
but I got to see the funny side, you know, but yeah. So what happened with this exploit
is it was found pretty early doors in terror, I think, and they rushed out an emergency
upgrade for it. For all the other chains affected, there was a tracking issue put up which had
like no real specifics, but just said, oh, you know, there's some unit tests that need
adding and the and Confio sort of just got it expedited and got it in. And the second
thing they did, we then obviously put it into Juno's code base and we had it ready to go.
So you kind of had to be paying attention a little bit to know that this was a vulnerability
or you had to be poking around. However, like the, well, the thing is right, you can't speculate
too much because obviously an attacker is going to want to outfox you using social stuff
more than code and stuff, right? Because it's much, much easier to be like, to call the
contract short sees killer contract and then make it look really slick and competent. So
you're like, this is definitely the work of short see is really nicely written. And then
like take down the chain. Everybody's like, well, this contract is obviously written by
short seed because it's really slick and you know, there's lots of code comments and it's
really well written. It just kind of it's quite handsome contract. So, you know, we,
we nobody's going to like be like, Oh, no, no, right, this contract because we all know
that null just passed as usurpers code, right? So you kind of, you know, there's no way
null attack the chain. It was definitely short seed, but, but actually it was null, the attack
the chain, you heard it here first. But, but that's what I mean. Like when you kind of
look at like, you know, what can we infer from the attacker? What can we infer from the
contract? It's cool that we bottomed it out and we know that it was an ABCI query. And
you know, shout out to my brain is not broken is my brain is broken as a mo is his Twitter
handle for working that out. He puts in a lot of hours to basically write a bunch of
stuff in order that he could sort of test those transactions to just ensure that that
was actually our kind of working theory was in fact correct.
Oh, it's off. It's off figured out. Yeah. Yeah. And well, funny, funny thing is I'm
repping secret network right now. So that's that work. Yeah, right. To the, to the fucking
rescue. So that was rad. And then big shout out as well to like the sect out guys and
everything who said our folks, I should say, who, you know, kind of came in and helped
out with sort of like de-compliation against the bottom stuff. There were quite a lot of
people working just on the five forensic side rather than the recovery side, which
was quite, which was, which was really cool as well. Like that was a really interesting
chat to be in. I think in general, like, I think we've probably all learned a lot in
the last anybody who's not sort of, you know, basically Jack or Andrew or a self has learned
a lot in the last 24 hours. Just because like it's, there's just had to be like, I mean,
just in terms of like, not only are like kind of thinking of like, Oh, are we certain this
is the, the attack vector? What are all the other things we can try? What are all the
other things we can look at? That's like one side of it. And then the other side of
it is like, what are all the different things we can try to kickstart the chain, right? Because
it's, it's, it's 125 forks at the moment. So all options are on the table. But we obviously
want to avoid a hard fork because of the IVC connections, right? Right. So it's been really,
really interesting to see how those ideas have developed, but also like we've, we've
had to kind of, yeah, it's been assembling the plane on the way down, but you know, we've
got a few expert engineers in the fuselage, if you like. So the, which I'm not one, I
should clarify, but there are, there are adults in the room who are, who are telling people
like me what they can and can't touch as the plane is falling off the cliff. Well, yeah,
I guess just read along and chip in with the things you can, right?
So basically the methodology that they tested was the one where they replaced all of the
validators. So they were just using the existing state, right? And they just replaced all of
the validators with new keys and then just started it back up to see if starting with
all of the same data state. Yeah. So, yeah. So like one block before the problem block
and then yeah, like removing the Gov prop, remove, basically removing any of the things
that would impede the chain from moving past where it was without any problems. So like
removing the smart contract state, removing the smart contract itself, plus a bunch of
other bits and pieces and then running it.
I saw strange love testing rig. I saw in the, in the handler that it, there was, it needed
to remove the, the Gov prop. But why was that exactly? The Gov prop?
I think mainly, hang on, let me think. Oh, because we don't want to hit the, we don't
want to change the, the upgrade height because we're not going to start, we're not going
to have that binary. Right. Although, we're not, we're skipping that binary and basically
we're going to a new binary now. Yeah. So we're actually wanting to like hit that upgrade
height and then like downgrade to another, the older binary that we'd already done.
So we were, we were, so this is a little bit of jargon for those watching the stream who
may be on. So we, we use the thing called semantic versioning to tag versions of the
software. Lupa Kailia is the code name for a binary, a piece of runnable code, which
is 2.3.0. We're using this opportunity not only to get out something that is a little
bit further along, actually we are 2.3.2 or something, it would be technically, but also
roll in some additional, this new Cosmos and security fix that came out today. And it's
going to be sort of like version 3.0.0. So we've moved a major version forward, I think
primarily as well to indicate that we're actually going to, you know, move past this problem
block in an unusual way, which is by incrementing the block, which, so I actually haven't been
in the ecosystem long enough to, to have seen this firsthand, but I gather is the way that
some of the early migrations were done in Cosmos. And I think that's why Jack and Co
was sort of familiar with the, well, obviously very intimately familiar with this as a solution
path. So Jack, Demi and Ethan Fry of Cosmos and fame with whom some people occasionally
confuse my handle, although he is the original Fry in the Cosmos ecosystem and the first
time my handle is ever conflicted with anybody's name. But those three was quite bullish on
the, the incrementing one block and then moving forward approach. And it's for historical
reasons, it's what it's, it's a way they used to do upgrades in the past. So it was
known to work. We just have a much more complicated state now, what with IBC and smart contracts
and yada, yada, yada.
So we're going to have like, I think we're going to export the state. Does that going
to necessitate a new Genesis then or same Genesis one block forward and then upgrade
handle is going to run, we're going to have the new patch and then we'll continue on from
there. Is that what's going to happen?
Yeah, I think that's right. That sounds familiar. That sounds correct. Any mistakes are purely
my own tiredness. That's the, that's the happy.
Yeah.
Just making sure I understand somewhat.
Yeah, we're using, we're using the, I guess we're using the export. Yeah. Essentially
the Genesis, aren't we? Because we're, we're, we're just going to move it forward one block
as if we're, as if we're continuing, but we keep all that, we keep the chain integrity
of the chain name and the all the state and everything.
But the transactions in the, in the halted block would disappear, right?
Yeah, I guess so. We're just keeping the block in the preview.
Yeah, we keep the, we keep the ones from the last confirmed block, but anything, I guess
we'll get rejected.
And I would assume there'd be some ABC in there as well that might be sitting that might
have to just be reattempted.
Would it not time out?
I don't know.
So I'm not an ABC expert. So that I couldn't comment on, but I would be very surprised if
it didn't time out because of, well, hang on.
We figure it out given enough time because we kind of know, we can sort of puzzle out
how this works.
But presumably, it's a light client on each network, right?
So if the, if the light client on the receiving network hasn't, hasn't actually confirmed the
transaction, right?
That's right.
It will at some point fail it.
Right.
The problem at the moment, presumably is that it's just dialing, right?
So or the confirmation never comes back, right?
So it just fails.
Is that right?
Right.
So it's kind of the Relayer Expert here, I think, right?
The Resident Relayer Expert.
Yeah.
So in theory, that's basically what's going to happen is any, any ABC transactions that
were kind of the process of happening, whether it be inbound or outbound, they are, they
might actually still go through just because like once you go past the block state, it'll
then pass forward because as far as it's concerned, time hasn't changed.
Or just time out.
I don't expect anything that had happened or potentially happened during that block
would cause issues.
All funds should be continuing safe.
Cool.
So I guess you might have to.
I think there's quite a lot of people asking on Telegram about how the IBC would work.
And like, I think me and other people who don't understand IBC very well, we're going,
well, it's going to be fine because obviously these transactions are atomic, but I don't
know the exact details.
But there you go.
If anybody in the stream was wondering, Schultzie, I think has, has hopefully reassured, reassured
everybody that it's going to be fine.
So Schultzie, if, if, if the block never gets finalized on this end, you can just flush
out all those transactions once we get it going again, right?
Yeah.
So that's basically the idea.
Right now, obviously you can flush out transactions because that requires a, like a commit to flush
them.
And so since the chains are moving, you can't commit to, you can't flush them back.
So right now, like if you look on this most, I'm sure it looks like it looks like there's
a bunch of packets waiting because I guess in theory they are, but there's really to
be done about that.
So yeah, I mean, I guess that is, that is our concern of getting the chain going properly.
And that's one of the things that we need to do more testing on before we move ahead
with a, with a solution to, to put it out into production, right?
So all that stuff has been tested at the moment, I believe the fray.
Yeah.
On the testing path, it's like get something that works, tested it with a bunch of different
validated configurations, then start testing IBC.
Once we're sure everything works, then push it out.
The strange love folks are literally testing it as we speak.
I think I did a shout out on Twitter to new Andrew or GNU Andrew, depending on how you
pronounce new, I pronounced it new personally.
But yeah, who's done just a load of incredible work on automating all of this testing so
that we essentially don't have to do it by hand.
And it's the reason we've been able to kind of react so quickly to this has been really,
really heavily driven by, by the strange love team.
And like I say, Andrew in particular.
And then we, there were a couple of other strategies that were being sort of looked into an experiment
with and that was driven very much by DEMI who had essentially a couple of other quite
good ideas that would hopefully excuse me, allow us to move forward without resorting
to a hard fork.
But so ultimately, I think we're going to go with the incrementing strategy as we've
sort of already talked about.
But yeah, that's what they're doing at the moment is they're running a series of tests
for sort of like edge cases.
So now where they're up to is that we've been able to make blocks using that new strategy
or Andrew has specifically, and we were all very excited to watch him make blocks with
the, the genesis, which was based off of an export we sort of prepared from the, the final
block.
And now I think DEMI and Andrew are going through a variety of scenarios, like, you
know, some validators haven't switched over or some validators are on a previous version
of genesis or, you know, these kinds of things that there will be edge cases where some of
the valset doesn't agree or people haven't quite configured it right or, you know, nothing
malicious but potentially accidental Byzantine behavior or accidental configuration or whatever
and that is something that, that we need to test, right?
Because it's quite likely to happen.
So one of the other interesting things that happened, I don't know, Noel was in this,
I know, Shultzie, I think had a, a DEM from me in the middle of the night about this was
quite early on in trying to diagnose the root cause.
We were trying to work out whether or not.
So way back when Manita first came out, we had a security vulnerability that came up
very, very short notice and we ultimately decided to patch it by distributing a binary
from the core team to the validator set that patched it rather than, because it wasn't
consensus breaking in the way that we, if we were distributed it, it just ran it.
It wouldn't be breaking consensus.
So we were able to just get validators to upgrade.
But we sort of realized that we didn't have any way of checking that every validator was
on the same version and if we had more than 33% of validators not running the same binary,
then we were still vulnerable to that security exploit.
So on the first things we had to do, sort of overnight really last night was compile
the spreadsheet of all of the valid, and again, DEMI was amazing kind of getting off the mark
on this and starting to get responses.
And I spent a lot of my evening and night nagging people over DEMs and trying to get
in touch with people and tracking down validators who aren't in our Discord, but I'd seen them
on a different Discord somewhere and going and finding them and going, hey, sorry, you're
not on the junior Discord, but your node's down.
Here's why.
Can you fill in the spreadsheet?
And what we were trying to establish was whether or not over 33% of voting power was on a binary
that was vulnerable to a previously known exploit.
So we could rule that one out because it was one of the potential consensus type breaking
bugs that we might have been vulnerable to.
So just so people understand why some people might not have been on that latest binary is
because it wasn't, most people in all across the cosmos, when they're migrating a node
and then they're building the binary stack back up in CosmoVisor, they're probably just
building the versions from the source.
And so in this case, there was a binary that was released that you couldn't build from
source unless you built it a certain way to include that mucil C, the music binary.
And so what I suspect has happened, and this would have happened to me as well if I didn't
do it a different way, is that when people have migrated their nodes, they've probably
taken their data folder with them and then just built the new binary from the source
code as 2.1 and then they lose the patched binary.
So if you're like me and didn't want to build it and just copied your CosmoVisor directory
instead, you'd still have the patched binary.
So it's probably, I suspect that that's why a lot of those validators weren't on that
patched binary, probably from migrating and also new validators that had joined the set
after that upgrade and then just built directly the 2.1 from source.
So I don't think they can be fully blamed for it.
There's also so much automation going on and anytime that you have a release that doesn't
fit into those automation plans, I think things naturally fall through the cracks.
So they might have Ansible or some other type of method that is looking either A, looking
for new binaries or getting things that are ready.
And I'm just going to go run the job and go pull down and check out the right branch
and run it and go without actually maybe looking at the read me, right?
Because everything is kind of being notified in Discord.
And I'm really surprised that there's Juno validators that aren't in the Discord.
Yeah.
Like, well, some are super unknown and don't like talking to people, right?
You have to talk to people.
At least you have to get some announcements because if you're just watching GitHub, you're
really not getting the full story on what's coming or what's going on and everything else,
right?
Yeah.
I don't know.
I guess there's a Telegram, I guess, and other modes like I guess not everyone uses
Discord.
Is there a Juno Telegram?
I don't think there is a Dev Telegram.
So Juno Telegram?
Because there's like the Juno Telegram and then there's the Juno Price Telegram, but
I don't think there's like a Juno Dev.
Oh, like an announcements.
I don't think there's like, there's like an upgrade is coming Juno announcements channel
on Telegram.
No.
No, I don't think so.
Could this halt have happened non-maliciously?
Could it have happened just somebody's doing a contract that goes a specific way and it
just, it just happened.
Like, does this have to be malicious?
Does it have to be an attacker?
It could just be, hey, there's an X, there's something that somebody wrote something poorly.
So it's possible, but unlikely.
The reason that it looks more suspicious.
Well, so the, if it was an attack, I think there was one of the obfuscation techniques
was using essentially the Cosmos and default template, like the literal learners template
to, as the basis for the contract.
Because you look at it and you see like a chain of executes on it and you're like, some
learner dev is just executing it a bunch of times and something crazy has happened.
Right.
But then you look at it and you go like, oh, no, they've, it's a counter basically.
It just increments by one, right?
And it'd been incremented like 55 times and you're like, nobody has the patience to submit
55 transactions to a blockchain just because they're learning.
Two, maybe three I buy, but we've all been learner devs and like you're just like, cool,
it works right onto the next thing.
You don't sit there and submit 50 transactions.
That's just not a thing.
A learner dev, whatever you do.
You're too excited by the experience to do that.
Yeah, it's like, it works.
It was super suspicious.
But the kind of effort to go like, oh, no, make it a learner template says one of two
things, either they themselves are pretty new or they just want the simplest possible
template they can use or three, it's a deliberate obfuscation technique to make it look like
somebody who doesn't know what they're doing to kind of send us down the wrong path for
a little while.
So we're like, oh, what's going on here?
But I think because we were aware of the ABCI thing, we kind of theorized it very early
on like, like, I mean, like I say, it was pretty much instant.
Like, is it the ABCI thing?
And it was something that got vocalized like again, pretty much straight away.
It just took a long time for somebody with a big enough brain to actually get all the
way to actually proving it was that thing.
Whereas the rest of us were kind of happy to go like, that's probably that.
Let's move on.
Like, right, right.
Because what we were able to do or what particularly Andrew was able to do was actually essentially
spin up a set of fake validators to show that if you had validators on 2.3, I loop
Kali, they were not vulnerable.
But because we had the the WASM, the compiled contract, the malicious contract, we could
run it and see what happens.
So we were able to basically see that current main net is vulnerable, which we've already
seen, but we could replicate it.
And then we could say, oh, well, loop Kali isn't vulnerable.
So at that point, unless it's some completely third thing that we don't know about, it's
likely something that we know about from the changelog.
It's likely something that we know about from security disclosures.
And we're relatively close to the confio teams and we talk to them about these sorts of things.
And particularly the ABCI one was a very, like a known pretty bad thing that we were
very, very quick in getting our fix in as a result.
But obviously events sort of derailed us actually getting it out, which is why the second this
happened.
So have we just screwed it by not getting this upgrade out as early as we could have done?
And that's that's going to be one of the things that comes out in root cause for this, you
know, bluntly.
So yeah, and probably maybe not even that much of a coincidence that happened six hours
before this upgrade, right?
We were talking about like we moved it when we started this, we're saying, oh, we moved
it yesterday or something or Monday, but maybe they upload the contracts on Monday, I think
and we're executing it overnight, just trying to get the same block.
Yeah, with a couple of different transactions.
And so it would have been that and that that mix would have been the right proposer.
Is it proposer based or why wouldn't I think it actually matters?
It just looked like they they had something in their mind, but actually it would have
worked.
If the ABCI query had worked in the right, yeah, if it if it works, it works.
I think I don't think it proposed it was material, but yeah, they were looking for
something and they found it.
That was a question for 1132 after 36 hours.
What time is it hit 12?
It's 1230.
Right.
I thought we lost your connection there.
Pause your whole brain pause on that one.
Yeah, I've been up for I've I got a 530 this morning.
I'm in UTC plus two and I'm in UTC plus one now.
So can you do some quick square roots for us?
Maybe?
Yeah.
I've been up for 20 hours.
I guess I've been up for 20 hours.
Maybe.
I don't know.
Something like that.
So the fray people keep speculating that, you know, this is the whale revenge.
So I think there's a lot of them.
Are they?
Hey, I think the whale is smart enough.
Are they?
Shots fired.
The negotiating strategy so far does not does not seem that of a monster tactician.
So I would say from like what we know that's highly unlikely that it's the whale.
It doesn't really benefit anyone, right?
Well, no, it's what?
Yeah.
What why?
Why?
Key bono, right?
Who benefits?
Yeah.
Well, doesn't benefit because if it's found out to be them, their funds burn out a spike,
right?
If they successfully take down the chain for good, they also lose their funds.
It's kind of seems like a lose-lose.
There's no leverage to be gained from it, right?
A better move would be to go, well, we know you've got, I should know, even then, even
if they knew that we knew about a security fault that they knew about and we were days
away from upgrading, there's still no real leverage.
And they would know that we could recover it anyway.
So what would be the point?
I suppose some people will look at Evmos and go, I think Evmos is an example of, like,
without throwing any shade, it just shows how hard it is to recover a chain when it
gets into an indeterminate state.
And the fact is taken 35 people to get us to a solution.
And what strange love I've brought to the table to help us get to the testing point
of knowing we can get this back online is cutting edge, is a cutting edge piece of test
automation stuff.
It is very cool.
It is very, very clever.
So I think if you didn't know that the devs on Juno were pretty stubborn and also had
enough friends and favours to call in to bring the cavalry, you might think, oh, we can probably
take them down for a couple of weeks.
But again, who benefits from the network being down for a couple of weeks?
Nothing's grown.
Yeah.
No one.
A competitor or a board kid, really, are the two people I can think of.
And is a competitor of Juno really that worried about us that they're going to attack us?
I mean, maybe.
But again, who?
I don't think it would.
Like what's, yeah, I mean, the more activity in the ecosystem, the more attention the ecosystem
gets, then the better it's going to be for all of us.
So why would you say, like, why would anyone want to take down another one of the IBC connected
networks and it doesn't really make any sense?
I mean, my guess would be like, if I had to just completely guess, I would say, you know,
if you were a dev on terror and you had seen that patch, you could easily make the link
to the ABCR, the ABCI queries.
And then you could go, OK, if you knew what you were doing, you could then craft an exploit,
right?
So a board, a board young terror dev, not not as in from the terror organization, but somebody
who knows Cosmosm because they were doing on terror and had seen their security notification.
Maybe that's somebody who's they're just like, you know, they're doing, they're teaching
themselves smart contracts and they're just like a bit bored.
And they're like, oh, this will be funny.
I wonder if I can take down Juno with this thing I've just learned.
Right.
Well, we've all been bored teenagers, man.
We've all smashed windows on stupid shit.
Like this is kind of just a multimillion dollar example of the same thing, isn't it?
Just been a bit surprised then by appearing on the screen.
Jack has joined us.
And a wild Jack has appeared.
You know, I think it just sort of underscores the importance of procedure and sort of more
regular check ins about this stuff.
You know, I think that the engineers who get into blockchain engineering are people who
are kind of security focused.
And I think especially when we're spending up chains and doing a lot of things, it is
the first and only thing we think about.
But I think that a lot of this stuff around sort of like regular network maintenance is
not sexy.
It's not quite as fun as launching new products.
And it's really easy to forget that we just need to, you know, do the basic blocking and
tackling.
And yeah, you know, what I have seen is a lot of great basic blocking and tackling and
tackling this incident, which is fantastic.
And the number of folks that kind of came together to do that just huge, a bunch of
people in this call, which by the way, thank you guys.
Jack, are you talking about specifically around validators not running latest binaries and
just kind of getting lax around those types of things?
Is that what you're referencing?
Well, I mean, that's definitely part of it.
But I think from the core team perspective, too, you know, what about a security council
that meets once a week and just does a dependency audit?
Sure.
And like, you know, and more regular security upgrades.
I mean, this, I think we've effectively found that this was a target query issue where the
attacker was querying the node ID out of the node and then hashing that.
And that was the source of the non-determinism.
Yeah, that was it, I think.
It was literally the ABCI query that, yeah, it was the ABCI query exploit that meant that
terror panic upgraded, you know, whatever it was for four and a half weeks ago.
And then we got a security notification and we put our, we, you know, we patched, you
know, 29 or 30 days ago, something like that on GitHub 31 days ago.
So you know, for Jack's point as well about being sloppy, I mean, we already brought this
out earlier.
Like this is an easy broadside for me to fire because it mainly falls, I think, on my desk
as somebody who works very closely with organizing test nets and validators and does a lot of
the, shall we say janitorial, you know, stuff like just like bumping dependencies, moving
stuff.
Janitorial, yeah, it's like, you know, it's really boring stuff.
Well, I mean, the thing is, you got like, we've got like the name service, we've got
Ed's secret projects in the, in the, in the works, you know, and I obviously want to go
work on smart contracts and work on the fun sexy stuff, but if we don't got a chain, we
haven't got anything to deploy on, right?
So you end up doing the janitorial stuff.
But then yeah, I think we've gotten lax with like focusing on like Jack says, focusing
on the new shiny stuff that we're working on.
And maybe like needing to go like, okay, no, this is janitorial, but just because it's
boring and it takes five minutes to bump dependencies, and then it takes like a week to get it tested,
you know, blocks, isn't it?
Yeah.
What's that?
Andrew, that's blocks, isn't it?
So Jack's, Jack's paying attention to the, to one of the many test scenarios I think
that's being run in the background.
Yeah.
So the height plus plus upgrade is now working.
Is this the one with one of the validators still on the previous genesis?
No, but we don't anticipate that one to be an issue.
This is starting from 99 with an export from 90s of the normalized export from 97.
We have blocks.
This, this 512 gig Ram instance with 64 cores is getting a huge workout today.
I don't think it'd be expected to be, to be running all day.
You got to start licensing that out.
Yeah.
There's probably 20 other chains that would be potential customers for that, for that service.
Well, you know, I mean, this is the, we have an ICF contract that's covering basically
all of our developer time on the IBC testing framework.
I think that the base of what we've built, we've been able to show off today, which has
been really exciting, but this builds on a ton of work that we as a team have done for
years, like the last year, year and a half.
It started with the framework that we built to test IBC and bring that to market that
is in the go relayer.
That spins up to individual node chains and connects them with a relayer and runs sort
of a standard set of acceptance tests.
We've been using that to drive relayer development.
We used it to drive IBC development.
I took the lessons learned there and built out a couple of other tools, but the biggest
one was the test framework for Horcrux.
I sort of inherited this legacy piece of software from the Polychain team.
It was really cool, but it was written by a C++ engineer who didn't really know how
to write go and it was like all standard live, very hard to get through.
I thought the best thing that I can do is just verify the functioning of this and ensure
that it does exactly what it says.
We built this framework where it spun up a whole chain, upgraded one of the validators
to a three-century, three-core-carts node instance and ensured there was no slashing,
that we were missing blocks, stuff like that.
That process of spinning, that took me about six to eight months of development time to
do through last year.
I tried a couple of different Docker testing frameworks.
I tried a couple of different ways of doing it.
First I did it with fixtures, then we sort of did it all with the containers themselves.
The big thing that we did was we switched to job containers.
We're not using any of the underlying Go APIs really for any of this stuff.
It just treats the containers as a black box.
It's like whatever you expose as far as whether it's RPC endpoints or command-line tools,
whatever you expose, that's what we're going to use against.
It is like if you're integration testing the system, that's how you attack it as a
client.
Why this will do it that way?
I've had a number of discussions with folks in IBC about doing this.
The GORAC team wants to contribute to this, but it wasn't until Andrew finished up the
Hork Rucksery factor after working on the test.
He was like, this test suite is great.
I was like, if you like that test suite, we've got this ridiculous one.
It's the matrix of all the IBC-enabled chains and all of the relayer implementations that
we need to make sure that they all work together so that we don't have splits in this network.
It's like, oh, well, yeah, that's a huge job.
Looking forward to doing a lot more with this.
We have another engineer starting here in a couple of weeks who I'm looking forward to
also putting on this and hoping to get some more contributions from elsewhere in the ecosystem.
I think that as far as a developer tool, if we want people to be able to quickly and easily
launch chains and test these complex cross-chain contract call integrations, you need something
like this.
I think it's an invaluable developer tool.
I'm really glad we've gotten to flex it today.
Is that machine that you're using, Jack, is that a local machine or is that cloud-based?
Google compute.
We have some fairly beefy limits.
That's an understanding, I think, right?
How much out of interest?
Darryl asked how much does that cost in compute per second or something like that?
It's, I think, 10,000 a month.
5,000 to 10,000 a month, yeah.
Yeah.
Nice.
I thought my R5s were pretty expensive, but that's a different level.
Yeah.
I think StrangeLive is doing on the order of 50,000 to 70,000 a month in cloud spend at
this point.
Sir, did you come to your house for that?
Did you guys get hand-delivered invoices from?
No.
We barely get anyone to respond.
Kelsey Hightower, the lead developer evangelist, just talks shit on all of our products.
Google doesn't give a fuck about our business.
What did he very publicly, like on Twitter the other day, say, like, hang on, here is
like one use case for crypto that I agree with and everyone's like, fucking, Kelsey,
Hightower's gone D-Gen.
It's all over it.
It's like, not yet.
I still think he's trolling us because he's trolled us a few times before.
I am a huge Kelsey fan.
When I started in tech, I saw Kelsey do a live demo and I was like, whatever he's teaching,
I want to learn and I want to be a developer evangelist.
Kelsey has been influential in my career personally, so seeing him so anti-blockchains
hurts a little bit.
I wish he was more pro-block.
I tried to explain where we were up to, to the smartest engineer I've ever worked with,
his previous CTO of mine, and he went through the dot-com boom, the last time that sort
of thing.
After taking him through all the steps that led us up to where we were, I guess about
like eight o'clock this morning, his conclusion was just one word and it was oof.
He's not a blockchain guy, so he was like, is it proof of work, proof of stake?
Okay, so you've got to validate this there.
We got through all those questions and he got to the end of it and he was just like,
oof, all right, how many forks you got?
I was like, we've got 125 forks, my guy.
He was just like, oof.
He was like, good luck with that.
Anyway, let's get a drink next time you're up.
I was like, thanks.
So it's like, it's a wisdom, but oof.
Would you rather recover this or would you rather recover a downed raft cluster?
Well, I used to work with him on a large project that had a massive Apache Kafka and Cassandra
cluster and our Cassandra driver library went out of maintenance and I had to rewrite the
database driver, which was in Lisp because obviously I'm a Lisp weenie, so that was.
Why did you do that to yourself?
You could have made better decisions.
No, no, why did you do that to your fellow engineers down the line?
Because we're a closure shop.
We all wrote Lisp, we all like it.
We all think it's the best language.
Self-evidently it, well, okay, JVM closure, no.
Closure RS, which I still believe one day might actually have a working version.
Then you're going to see smart contracts in Lisp for me.
That's a promise.
Does it compile to Waza?
Well, it's valid, it's valid, Rust, isn't it?
So yeah, then you could jump it to Waza, I guess.
Cool, great.
I look forward to never reading any of that code.
Yeah, well, but you know what, at least the abstract syntax tree will be exactly the same
as the code itself and therefore it will be everything in perfect balance as is the Lisp
way.
I mean, I'm over here throwing shade, but I actually enjoy using closure and I'm over
here like looking at, you know, reading up on Elixir and other functional programming
languages so I can only throw so much shade.
Yeah, exactly.
It's a way of life, it's a way of life.
Like if it takes hold of you, it's a certain way of doing things.
Just like I think like anything else like programming languages like musical instruments,
whatever, like some speak to you, some don't.
For me, like I can take a leave Python, I've written a lot of JavaScript type script for
money, but I'm kind of broadly indifferent to it.
Lisp is the only language that I actually love, like as in it makes me happy to work
in Lisp.
I don't think there's another language I feel that way.
I quite like Rust.
Rust is great.
I think Haskell is very interesting, although I have to relearn it every time I use it,
unfortunately, because it's just really complex.
But I don't know, maybe not everybody gets emotional about programming languages, maybe
that's just me.
I find it very hard.
Oh no, I definitely do.
I fell in love with Go early and I just like writing Go.
I would prefer not to have to have to write anything else, but I have to screw up with
Rust, I have to screw up with JavaScript, I have to screw up with this solidity nonsense.
Anyway, yeah.
Is there anything else in the Judo Hall that we want to cover?
I think probably pretty much being over what we know so far, right guys?
I think it would be worth knowing when do we think that the fix will start going out?
Can we expect Judo to be running in a week?
I think 48 hours is a little bit more realistic.
My thought is that once we verify that a particular path forward is valid, we should give validators
24 hours and restart after that.
I think that this path of export at 97, increment the height to 98, reset the contract state,
that works.
I think we should go for that.
Getting the network back up in 24 hours as of right now would be, I just put out a call
and a call in court too to do that.
I guess it depends on what kind of participation we get from all the validators in that timeframe,
right?
Yeah.
I think everyone is pretty aware that this is going on.
We were able to get, I don't know if you guys saw this, I'm sure some of you were a
part of this, but at the beginning of the incident, Demi was running around chasing
every, well, all of us were chasing validators and trying to put together a list of what
software versions are on, which by the way, fascinating the way those numbers turned out,
right at the one third, just there.
We had the same thing at most, it was 66.0% or whatever it was.
It was always one away, right?
Same situation here.
Just insane.
Okay, so we were able to get in touch with everyone, I think 98% of the validators said
probably within an hour and a half, two hours of really starting that reach out, so probably
four hours after the downtime started.
That was incredibly impressive in my mind.
I don't think we'll have a problem getting the VALSET back up.
All of the top validators are engaged.
They've had open lens of communication with us.
I think 24 hours, we could do a quicker restart if we really want to push things, but I think
24 hours.
So 5 p.m. PST tomorrow.
Andrew, how do you feel about that?
Are you talking about this live right now then?
Yes, I am literally talking about this live right now.
I think we could all just see this in the corner of the screen, the same notification,
like, okay, this is happening.
So you see a bit of years of the channel.
You are literally seeing some decision-making happening on the stream for what we're going
to be working on tomorrow by the look of it.
Gary, John asks a great question.
Will Luper Kelly be done at the same time while it's down, or will we wait until it's
back up nicely and then try again?
Or you're going to do this while it's down, and in fact, we're probably going to modify
the governance proposal in-state so that it doesn't create an upgrade because we've already
done the upgrade at this point.
Luper Kelly has the nice side effect of fixing the Stargate queries issue.
So we verified that fix.
We verified that the new version fixes the issue with the contract.
So the best thing that we can do is just move forward with the deployment.
Have we rolled anything else into that now, Jack, in addition to what was already proposed?
No, in fact, we haven't.
The proposed binary fixes this issue, and it happened hours before the upgrade was supposed
to happen.
Kind of hilarious.
I think Andrew actually was running the new tag binary which has the Cosmosm security
fix from today rolled into it, right?
Yes, it does.
That's correct.
So we're getting that one as a twofer for free.
So then in that case, so it's 2.3, and it's re-tagged as three, and it includes a new
security patch as well as removes some chaff about that contract that messed everything
up.
So what's happened?
Nothing.
Everyone's on mute.
They're talking while they're on mute.
Oh, sorry.
Yeah.
Yeah, pretty much.
Sorry.
I just instinctively do it just like hit the thing.
Chelsea's the only one who needs to put his thing on mute so we don't hear his clicking.
You know what?
I don't have any clickers with me this time.
So I've been like the entire time over here like, is she a bird?
Yeah, he looks so nervous.
He's a bird.
Yeah.
What time is it of running the side of the pond right now?
It's nearly 1am.
It's pretty late.
So Europeans will be nearly 2am or 3am depending on which part of Europe.
So it's just afternoon restart is a tough time because Asia is still waking up.
Yeah, I would probably say if you could pull it back a couple of hours, it would make the
Europeans happy but you know, such is life.
I mean, 4am Sydney is usually the best time.
It works for everybody else.
It works for everybody.
It's kind of what we're trying to do.
That time slot works exactly for everybody except.
Hey.
Just real quick.
I'm watching the comments here, what I wanted to address for folks.
Ramara asks, any issues of ledger for restarting at a different time?
Short answer?
No.
Long answer?
Absolutely not.
Nothing to worry about with your ledger.
Funds are set for.
Yeah, like that's there's no co-change or anything that would affect ledger operation.
I think maybe do you think maybe that comment was actually referring to the state as the
ledger?
Oh, it's a tribute to ledger.
Yeah.
Not so much the ledger.
Yeah, that's actually a great question.
And in fact, one that we did not have a concrete answer to until like just a second ago when
we tested it, there, you know, these networks are designed to be down.
There have been down to tendermint networks a number of times.
I think that one thing that we're doing differently in this Juneau upgrade is we're incrementing
the block height and keeping the chain ID.
This particular style of upgrade has never been attempted at scale before.
So there were a number of things that we were worried about.
But I think that, you know, we've just kind of verified that we're good to go there.
So that's exciting.
Jack, one thing we were talking about before you rejoined was like how this upgrade or
at least the 210 with the binary patch was slightly different type of approach and upgrade
versus regular ones where people would just, you know, pull the latest GitHub and compile,
right?
It had the library structure and those types of things.
And since it was slightly different approach and you have a very wild, you know, 125 validator
teams at different time zones and different communication and you have some people are
not even in the discord and things like that.
These networks are hard, right?
And so when you have things that are slightly outside of normal, which we saw it at most
and we've seen other types of things or test nets, shit happens, right?
And we saw that even saw that you said you said you surprised the Excel results.
I was completely surprised the Excel results on even, you know, top 10 validators that
were running the wrong binary and a lot of them, right?
Like a decent percentage of that.
I mean, that's actually, I think honestly, one of the strengths of these decentralized
networks is like they're heterogenous.
They're not homogeneous, you know, not every node is the same.
That makes them hard to attack.
Like that's a feature, not a bug in a lot of ways.
Is there any difference of feature?
That wasn't a version difference.
That was a build tag feature.
Okay.
It was a build tag difference.
Right.
And there was a lot of variety on how people provision their servers.
Honestly for network security, that's great.
If you've got a hundred different ways people are deploying, makes it way harder to break
in and steal keys.
And you know, the fact, I think that what we should take away from this as a core team
is that if we have some specific build things, we need to like bold those, make sure people
know what it is because, you know, I've been building tags that go ledger for years.
And I think a lot of other validators have too.
And luckily Andrew read the things the right way and we were running the right version,
but we easily couldn't have been.
So the Frey and I were actually, we've been talking recently about sort of updating the
Juno docs to sort of better represent the upgrade path for Juno given, you know, that
not everything's built from source as well.
And with those default tags.
So we just hadn't got there yet.
Yeah.
I mean, it's, yeah, I think that like anytime there's an issue with one of these networks
like this, somebody's been like, try to make it happen before.
And this time different, you know, a lot of the things that we've talked about in the
past in various channels like come back to buy this year.
Yeah, it happens.
You learn, you learn from it, you put in process and you try to do better the next time.
And what I have learned from this experience is like exactly what I learned during the
Prop 16 experience, the people working on this network are awesome.
We all like working together a lot and we're very effective at, you know, async collaboration.
And there's enough of us where we have full 24 seven coverage on different pieces and
you know, the people who are up at any given time can are very effective.
That's more than we had for cosmos for years.
And I think it's worth noting too that we've got like a lot of incredible developers that
are highly involved with Juno as well.
I think, yeah, just actually looking at, and you know, people passing the baton as well
as the time zones go through is incredible to see as well.
So, you know, handing over the lead position for coordination to another lead dev until
you know, the circles back around as we go through the time zones and people obviously
need to sleep.
You can't just stay awake for 24 seven unless you're the prey.
Oh, I think I think max, I was gonna say, I think max did like, I don't know.
Max is still in this zoom call with me and Andrew and Joe Abbey right now.
He's been there for nine hours.
Yeah, right.
But bear in mind also like max was the first person who identified the smart contract call
like when my, when my pager went off and I was like three hours from my desk.
Oh, it was max.
Yeah.
I had a DM from.
Oh, with dude.
I think I've seen a smart contract that looks suspect.
Here's the min scan TX and I looked at it.
I looked at it and my first reaction was I bet this is that ABCI Stargate thing.
And then it was low and bold.
But it's it's always max max is always first on anything else.
Like he always beats all of the rest of the devs to like pointing to the thing in the
direction of where the actual problem is.
Max is like, it's just like everybody in the team, I think, and Juno like has a kind
of perspective they bring to the table, you know, max is incredible.
The amount of support he gives to the community is just unparalleled.
Well, I mean, it's unmatched like with any of the other, you know, core contributors
or community members or anyone in Juno is definitely puts in the most, you know, go
community support for sure.
And yeah, the fact that he comes up with these fines in min scanner stuff is pretty amazing.
Like, is is max max?
I know you can hear us.
Are you are you a developer?
I don't know if he's a developer or not, but he certainly finds a lot of stuff.
It takes a while for comments to come through.
Yeah, I don't know that that's a great question.
I believe max is one person, funny enough.
But I think that kind of speaks to Juno being this really truly community led project.
Like I saw that like I always kind of knew that max max Max you're in both of the same
streams I'm in right now.
You know, I think I saw that really truly the scale of it for the first time with Prop
16, but I always kind of knew that.
And it's something that Juno does better than any other cosmos chain.
And I think it has been a differentiator.
It's been something that's enabled us to sort of like push through this issue because like,
honestly, this is blockchain dev worse nightmare.
It's about about as it gets.
And you know, it's easy to get into a place where it's really hard to recover that chain.
And it's been cool to see this community kind of push through that, find the answers and
get this thing moving again.
And like, we're going to get it.
Yeah.
I mean, like when the when the chain first halted, like I was with my old man and he was
like, what was that message you just picked up?
And I was like, well, pretty much the worst thing that could happen has happened.
And he worked for Oracle for like 25 years.
And I was like, what would be, I was like, a data center is burned down.
That's I think that's like the equivalent.
Like a data center is fully gone.
And he was like, oh, that's pretty bad.
You don't seem too, too worried.
I was like, well, I'm pretty worried, but I also like, I'm 100% certain that we have
the people in the building to have the chain back up.
So the building that burned down.
Well, okay.
The building, the building was his world.
The chain is our world, right?
So his building's burnt down.
Maybe his devs were in the building.
I don't know.
You know, you have to go ask Oracle for that would be a human resources nightmare, I imagine.
But our building is decentralized and therefore you can't burn it down.
And thus is the power of, do you know, the center.
You can only confuse the shithead of it.
Yeah.
If you're like, whatever, maybe the real, I need to stop making this.
I made the same joke every time we have a crisis.
We've had a few of these now.
I'm like, oh, maybe the real, you know, was the friends who made along the way, but also
kind of.
And I think that the real cosmos is the friends we made along.
They're like music themselves.
They're like through talking.
Zucky used to say that all the time.
Yeah.
They go in good company then.
So guys, do we have anything else we want to talk about today?
We're out about, we're coming up on 70 minutes, which is usually where we start to pull the
pin.
Yeah, let's let's pull the pin.
Yeah, I think that's all.