Like the fray is still in a basement.
So, you know, probably not going to be showing up any time soon.
You just want to roll it.
Welcome to Game of Nodes, a weekly podcast from independent value entities.
Hello, loyal fan and welcome to Game of Nodes.
This is our 111th episode, Critical Fail, which, you know, these
these titles just get came up with without any consultation with anyone and generally make no sense whatsoever.
Except it seems like this week they may make sense with what we're probably going to drone on about for the next at least 30 minutes, I'm assuming.
One of our guys, the fray, decided that he wanted to, you know, jam in a basement instead of come and chill on the show.
So, I hope you're enjoying your music.
It was watching.
Yeah, I was watching a jam, not playing it.
Oh, wait.
No, I yeah, he's actually at a venue though, right?
Yeah.
He's not just in his basement watching it on TV.
That would be something.
Oh, I'm fucking you.
That would be a new love for skipping a God.
Sorry, boys, I'm not staring.
I'm not staring.
I need to watch some TV.
It's like two flights of stairs.
I gotta go up.
And Rahm is already in the comments, hosing us for what?
Sounds like the last episode.
I think that every, remember, it says it sounds like the last episode.
I think that every week.
We circled the drain.
Why would it sound like the last episode ever?
Is it because of the intro?
The intro was, I don't know.
He's a downer.
I hope it's the Lord of the Rings reference because Bilbo on his 111th birthday,
it's his last birthday.
And so this is our 111th episode.
Could be the last episode.
That is some fucking deep reference.
Holy shit.
Dude, that's a man.
So I haven't looked at the news.
Do we have any news this week?
We do.
Yeah, there's a couple things.
I don't know.
Is Rahm contributing to the news?
I don't know.
There's like one link and like two notes.
So maybe he broke a collarbone or a clavicle or some shit.
Oh, well, there's episode 111.
That's the one we're on.
Yeah, there's a, all right, maybe it's time to start with that though.
I don't want to start with that.
No, let's talk about it.
So it's all we're having.
Hey, we're already at the 17 viewers.
So you look at that.
Just fucking amazing.
Yeah.
I mean, I reckon they were always in fire hover over this.
Yeah, a lot of those are bullshit views from X Twitter.
Yeah.
So OK, well, at least actually all of us are having some.
Well, I'm not having problems, but both of you two are having problems
with various node related things this week.
Actually, yeah, well, there's problems later on.
I want to talk a little bit about the magical fix of injective.
And and you guys were talking previously and probably wanted to continue
your your musings about your nodes.
I think she'll see is participating in a pretty high volume
Xion testnet application, which is minting NFTs over IBC,
which is a bit of a newish type of application
and having some issues with with RPC nodes.
And you serve her as a pretty high volume
say deployment, which has also been having some issues lately.
I think you guys wanted to talk a little bit like about that.
And then, yeah, I wanted to talk, like I said before, about the injective stuff,
which is quite interesting.
And anyway, you guys can take it.
Tell me what's going on with burnt.
I don't know. I'm kind of excited about the injective improvements.
I'm honest.
Like, I want to talk about that first, she'll see.
And then we'll sort of roll into your your Xion problems.
Yeah. Yeah, let's do that, because I'm pretty stoked about the injective
signing problem. So right now, for my signer, I have three nodes
that I've been rotating between.
And every single day, a new node resinks.
And that's how I was able to keep up to, like, 87% uptime
with three full machines, full nodes, resinking daily.
That that's what I had to do.
The fact that there was one little config change that had to happen
and suddenly now, like 99% of time, I'm outraged, outraged, I say.
So a little bit of background on the injective protocol.
So as a validator on injective, it is quite frustrating trying to
maintain a high uptime.
There's a lot of, there's like a lot of IO on that application
and a lot of nodes struggle to keep up.
I'm assuming with the head and P2P to be able to
sign all of the blocks and the remedies to date have been that
you need to keep a relatively fresh database that is,
you know, deleting your database and resinking from something
that's either cousin pruned quite freshly or even a state sync.
But I'm not sure if there is actually state sync issues on
injective. So I'm pretty sure most of the snapshots that are
coming through that are smaller made small by cousin pruned.
Well, so that's mostly true, but not entirely true.
There is one individual who has state sync working and that's high
stake. Their snapshots are glorious.
They're like seven gigabytes or something.
And the way they fixed state sync quote unquote fixed is they
turned off errors for state syncing.
And so they modified the binary itself.
Yeah.
So it's just pushes through and that's what I was thinking of from
every day. Every single day I download their latest snapshot and
then run off of that.
Shout out to high stake.
They are glorious.
So my validator nodes there have had pretty good uptime since we
started comparatively speaking to like a lot of other operators.
And I don't know why that is.
I think a lot of it comes down to just luck in P2P and peering.
But we seem to have been able to maintain probably between like
503,000 misses per the window.
A lot of other operators are sort of up around the 20 to 30,000
range or probably at the 15 to 30,000 misses range in the window,
which is what 100,000 blocks, Shilty, I think.
Yeah, 100,000 blocks.
100,000.
So anyway, historically difficult to get a good signing rate.
So keeping a fresh database, there are like certain settings that
are recommended like peer limits and, geez, I can't remember what
the other ones are offhand.
But they typically need just blazing fast core speeds,
like as fast as you can get.
Bigger drives, fastest drives you can get your hands on.
I don't think the RAM has particularly mattered too much in the past.
And sort of RAID zeroing things, just getting as much IO and thread
speed as possible on these machines to just sort of try and keep up.
And Horcrux has helped with my nodes by having multiple distributed
around the place and using Horcrux.
So I think we run three dedicated servers for Injective as well with Horcrux.
And the team at Injective Labs last week, we started testing some settings.
Do you have those settings offhand, Shilty?
I can pull them up.
Not offhand, no.
I don't remember what sections they actually were from, but I will.
Peer to peer, I think.
It was in the peer to peer, was it?
I'm fairly sure.
Was it Oracle changes or binary, like configure app Toml type shit?
It's in the config Toml.
And what did they do?
They changed some signing windows or something?
I'll tell you what they are here in a second.
Peer to peer latency, I think.
If I remember correctly, there is a config for basically disconnecting a little early.
So the flush throttle timeout and the peer gossip sleep duration, basically both of those
are reduced.
So the peer gossip sleep duration probably works as explained.
It's like between every kind of talk, we're having it wait less.
It seems like the throttle timeout.
Yeah, so flush throttle timeout, peer gossip sleep duration.
Yeah.
And well, man, it works miracles.
It's, you know, we're down to like what less than 100 misses.
Is this anything to do with pego and all that shit?
Or is this purely signing?
No, this is just purely.
Who gives a shit?
I do, right?
It's fucking vanity.
This reason we have distributed systems, like the fact that you guys are 90% or 95% doesn't
make a fucking bit of difference in the overall.
Well, or anything else.
But a lot of people are getting down to the 70s and 60s.
So like, Jen, Jen, I mean, we were one of them and that was with the audience.
I know it's.
Yeah.
So like even doing a lot of our best practices, we were going really low and even with most
one in Horcrux, just so that's heading with still and D Jen Horcrux, everything within
10 milliseconds round trip.
Yeah, it was brutal.
And with injective, like a lot of your delegations are based on certain performance metrics.
One of them.
And not only that though, like some operators were getting down to like the wire to getting
slashed because the slash window is 50%.
So, you know, if you're already missing a lot and then you have an issue with your node
that makes you miss for a little bit longer, 100% and then you kind of stuffed.
So yeah, so it was.
What you just proved is that even though everybody makes delegation decisions off of these bullshit
metrics, that that bullshit metric was completely changed by one small config change.
Just one small.
So did it happen?
Anything to do with?
Did it have anything to do with Schultz's ability to deliver this?
No.
Did did his signing ability?
So this whole fucking bullshit, like that you totally just explain the whole situation.
Right.
Like all these.
Deligators making decision off this fucking metric and this fucking metric is bullshit.
Yeah, I mean, that's fair.
Right.
So I said it was like, like I made this one config change on the binary that the team made
and now suddenly we're all at 95% and like, oh, look at, we're all able to sign.
But is it, is it, does that, does the past reflect then Schultz's ability to delivery,
to deliver nodes and RPC or to deliver validators in a secure manner at scale?
And does that make him a worse validator?
It's just fucking dumb, right?
So I would say that like this was a, you know, a collective issue that needed to be solved
because, you know, the more blocks you miss, you know, the more proposals you miss.
And if you miss proposals that affects the chain and its ability to put through.
I'm not saying it didn't need to be solved.
I'm just saying there's so many fucking variables that just boiling it down to like some sort
of signing thing is fucking dumb.
Like it's the whole purpose of the whole purpose of the situation and the whole purpose of being
able to do this is to be able to have a distributed structure that allows for failure.
And in this situation, you had validators being measured against a metric that was totally out of their control.
That's one config file change.
And then suddenly like everybody's metrics changes, right?
So the, well, what I said about the delegations was, I'd say it was a half truth.
It is, it is a metric, but it was, was not enforced because of the blanket problem.
Slashing window would have been forced, right?
Yeah, slashing would have.
But yeah, I mean, this is also, this is also a pretty specific, um, injective problem.
Like not a lot of other chains have this absolutely prolific signing problem.
And I've noticed that a lot of chains have gotten better since IAVL V1 as well.
So, um, the, uh, DYDX was having a bit of a signing problem.
Um, it probably wasn't as bad, but their window is only 8,000 blocks as well.
So there were people who were sort of missing 1,500 blocks in there.
And I noticed as soon as they upgraded, IAVL V1, it cleared up a lot of those as well.
And man, the difference, it was kind of similar to when, um, you know, say DB was implemented,
which I think is an implementation of IAVL V1.
And like the abuse on your drives is night and day.
I think so. Yeah.
I think they took the upstream and sort of fast tracked it and made some other changes.
Um, I think I'm not 100% sure, but the, the, um, the difference is night and day, uh,
on all of the chains that I've seen that have changed IAVL V1.
Um, the abuse on the drives just goes down astronomical, astronomical.
Like the, I've got some bad drives around on some nodes, like, you know, they,
they must be consumer drives and something.
And the, um, the where levels on those were just astronomical, like 30% a month.
And as soon as they switched over to IAVL, like basically the where level flat lined.
So the amount of like reads and writes going on to those drives dramatically changed
between the two software versions.
I'm not sure if, um, you saw the same thing.
She'll see with your nodes.
I got rid of those nodes with those shitty drives, by the way,
because they also had shitty networks and shitty uptime and cost the most out of all of my fucking nodes.
Yeah. Yeah.
Well, that's good.
So the team in it.
So did, did somebody spearhead that or that was at, was that the injective team that made that discovery?
Or is that somebody else?
The injective labs came up with the, the configuration change,
but I think they have been messing with things for quite some time, um, to try and find like the smoking gun.
Cool.
And it appears as if they have found it.
Yeah.
I guess.
Yeah.
I mean, that, I was like, when they, when they asked me to test, I thought, oh, you know,
it might be like a 20% improvement or something, you know, it's just like someone turned the lights on.
Yeah.
It was kind of like, that was like, say DB, right?
Like that was totally different.
Yeah.
It's just completely, completely, completely.
I think like what hardware we choose, like prunings, we had no pruning and like just the state sync structure and like all this shit to,
to get those nodes to do what we want to, and then all kind of just disappeared.
Right.
Now we're doing different issues, but whatever.
Yeah.
Different issues, but similar outcome.
And then just some small change or in say instances, probably quite a big change and a lot of engineering.
Yeah.
But you know, a change and then just makes things so much, so much better.
But I guess that's, you know, that's the iterative nature of these things, right?
Yeah.
You, you have a problem, you work on it, you fix it, you tackle the next one.
Yeah.
I think I think the challenge with, and part of the part that gives me stress with that is that, that, that
realization where, hey, this is all in motion and all that kind of stuff is, is totally true.
Right.
Like it doesn't, it's not perfect.
We find all these different issues.
There's a so many moving parts and like so many things that are changing and you find a new avenues.
Like I'm sure we're going to talk about with this part thing, right?
You're finding new avenues, but at the same time, there's an expectation like this is totally going to work.
And like, like, why can't we support this and why does, why is this issue happening and we're in main net or something
else similar to that.
And I think that idea that this is, there's an incredible amount of untested software that, that new issues are
discovered every day as a note op kind of comes down to the note op, not necessarily the team and say, maybe
not 100% sometimes or from a provider perspective.
And, and it's tough to be able to, it's tough to be able to guarantee a specific SLA associated to a service when the
underlying software to support that service may not give you at all the tools you need to be able to deliver that.
You know what I mean?
Which is, which could be RPC, it could be a validator service like we're talking about the injective could be anything.
But it's tough to be able to be in that position where you're trying to troubleshoot the best you can, but underlying
or something just incredibly broken, right?
Or just needs to be, that needs to be pushed forward.
Yeah.
And I, but I think like, in terms of RPC specifically, like, I think the teams that you're providing it to have probably the
best idea of the underlying issues in the chain that they're working on ends.
Like, I've never had problems with, you know, we have certain SLA conditions and no one has ever tried to
enforce one of those conditions based on a problem with the chain software.
It's more like a clause of negligence.
If you just allow your nodes to continue to have issues when there's no real underlying problem.
But we, we have to, in our contracts, write out issues of the base software because we have absolutely no control over that whatsoever.
Yeah, that's changed.
Well, but I think that that falls back to what Eric, you, you said we were saying before in that, yes, that's true for other entities,
but slashing is an example of a contract that can't always be totally upheld, right?
Like Pega.
Yeah.
That's in effect a blockchain enforced contract that is very difficult to truly adhere to.
Yeah.
So I will say one thing about Pega.
It is, well, it was more so a huge burden of like it was a pain to, to deal with Pega, right?
In that, you know, there were software problems and there were any number of things that made it difficult to deal with.
And the slashing was astronomical, especially if you had a lot of delegations that were not, you know,
people validate as we're getting slashed while their delegators were getting slashed, you know, 100,000 US dollars in one instance.
And you only have two hours to recognize and fix a problem.
We'll had two hours to recognize and fix the problem, right?
And, but in saying that, like it was possible, if you had a lot of people who were not able to do that,
but in saying that, like it was possible, if you had, you know, on call people and you had fucking like just systems of annoyance to keep you up to date with it.
So, you know, but, but it was also very easy to get slashed if the conditions were right.
Yeah.
But we've said that software is a piece of shit, right?
Well, yeah.
It seems to have been improved a lot recently.
Okay.
I mean, you're in that cartel.
So I can understand.
I'm not in that cartel.
So I can say whatever I want.
But I, from what I've heard, from what I've heard, it's been a real fucking bitch.
And it's not the greatest piece of software.
And then number two, you're held to a standard associated to that software.
And there's a huge amount of learning curve to understand the intricacies of how to run that successfully.
Right.
And you guys learn that through testnet, right?
And other types of situations like you, you learn how this, this thing fucking works, right?
Well, no.
That's, that's kind of the fun part of Pego as well is that the testnet is closed.
So if you're going to join in on mainnet, most people who joined mainnet immediately get slashed on Pego.
Right.
That's my point.
The point is the learning curve is so high in this thing.
And it's not like the most like, it's not a great piece.
Can we agree that it's not a great piece of software?
Like it's not the best thing that's ever been written, right?
It's somewhere in between.
There, there are conditions where it just files to operate.
So that, that would probably cost the most current answer I've ever heard of my life.
It's like, it's definitely not the best, but it's, yeah, it's better.
Okay.
It's better.
Better.
It's better.
It's the other way.
It's, you guys, I understand it's better.
It's, it's, it is better.
Right.
We're building.
Yeah.
But, but again, you guys, you're being held to a standard on something that is difficult to understand and, and has challenges to it that
require a high amount of learning curve.
And like to some point, like if you come into Validator today and in Kujira and somebody puts on a million dollars to get in that set and does whatever else.
And they get slashed in the first two because of the stupid fucking thing.
It's not like, is that where, is that why we're here for, for, for literally for this software that is run by a hundred other fucking entities that if one is down, it doesn't make any difference.
Like, what the fuck are we doing with that?
Like, I don't understand.
Like, is that really the need?
My only slashing is from the fucking inject from the Kujira.
Oracle.
No, it's from Kujira.
I'm very fucking traumatized by that experience.
I know, right.
But I'm just saying, like, like, I just don't, I don't understand these standards that are built around executing on substandard software for a reason that has to do with the
for a reason that has no fiscal or health capable like need.
Like the chain will still run those.
Well, I guess I guess you have to have something right.
You can't just say, well, we don't give a fuck about metrics at all.
Like you have, it keeps the, it keeps the operators honest in some respect in that.
At least the modest.
I'm not saying black and white.
Like there's, there's, there's levels of gray, right?
But, but like to say that like, hey, you know, you have this two hour slashing window or you lose, you lose $200,000 because the software is a piece of shit.
There's something between that and like not having any controls, right?
Like, like, I don't understand where the, where the, like where we, where these teams lose their sense of like, understand, even like we've seen like slashing windows at 100 blocks and stuff like that.
Like, what the fuck?
Like, well, that's just because that was the default in the fucking SDK.
I know, I know.
That's different.
But like, in that same situation, like, like, like, this is not, it's not a, like you're treating, like treating this, this ecosystem, like it's 100% figured out and like, oh yeah, we stole the global internet of servers that communicate each other and do hashes associated to blocks and
like this incredibly, and IBC and like all these wasm contracts and calculations and all like this incredibly complicated thing.
And, and if you look back, if you peel back the covers associated to these SLAs, right around, you said, you think you pull back the covers and you actually look at what's running underneath that and you go, holy shit, like there's, there's a lot here that, that has a long way to go to be clearly this like enterprise ready.
I hate using that word, but some sort of like, we can put this SLA associate to it.
And so like things like that, like slashing windows associated to Pego just, I just don't understand that stuff.
It seems like it's, seems like it's way too tight.
I don't know. I'm kind of on the fence, man. Like, I, some, some windows, some standards that you've been held to are quite, you know, strict, but that's the sandbox we choose to play in, right?
Like if you know, no, no, no, it's not, it is, you don't, there's no one fucking holding a gun to your head and saying go validate on injective.
Like that's the sandbox you choose to play in. If, if you want to play in that sandbox and for example, offer a, you know, a slash.
No, well, you know, it's for, for the, for the what? It's a business, right? It's for the revenue.
No, no, no, no, no, but like, but what is that SLA enforcing?
Like that, what you just said, like, like you're, like if you want to hop into sandbox that for the challenge, but the challenge of what?
I'm not saying for the challenge. I'm saying that's, it's a known, it's a known quantity. Like if you, if you want to play in that sandbox, you know the risks and you know the challenge.
So go play in it or don't like there's no one holding a gun to your head. So, or you could, you know, you could go and drum up some support and validate the hub where you'll never miss a block ever because there's six seconds and don't do anything.
So it's not that. It's just like, it just seems like the, some of the stuff is, is engineered around expectations that are unrealistic, right?
For, for the, the level of quality and the level of complexity that's associated to it. And it's, it's literally built to be resilient. Like that's the whole point.
But it's all, it's all beta software, right? But there, there are issues associated with like poor performing operators.
And again, it's not black and white. Like I understand that. Like, like, but a hundred blocks, like two hours to get slashed.
And that's obviously fucking ridiculous. That's just a lapse in people losing. Like if somebody's like coming in from outside, you could get slashed while you were taking a shit.
Yeah, seriously, take it to our shits. Like, but the idea like if I, if I, if I'm, if I'm, that's a fucking different problem.
But if I, if I'm entering the Seco system and I'm learning all this shit and I lose money because, because null, null had a very minor incident associated to Pego and took a two hour shit.
And they get slash because of it. That's fucking stupid. Like it doesn't like, what are we, like, what is that? Like what?
I just don't understand. Again, not black and white.
If I was on call and the alarm went off while I was taking a shit, I would snap one off and I'd go and fix it.
Just pinch it.
I'd pinch it and fix it.
And I've had to do that before. Sometimes I've taken a laptop to the shitter with me. So.
Oh yeah, I've done that.
Yeah. When you're on call, you're on call.
Not like it's not on call.
I'm not in preparation of something happening just like while something's happening. Yeah, for sure.
I've had that. I mean, yeah, you gotta do what you gotta do.
I've even thought about, you know, making a, getting my woodworking skills out.
Get a little stand.
Just pull that thing up here.
Pretty handy with, with some, you know, plywood and some pocket screws.
Perfect. How you get, you get the, you put little steps on the bottom, get your feet up so your legs don't fall asleep and shit.
Like you do something nice.
Well, you know, it's like scientifically proven that you shit better when you've got your knees on it.
I know. I know. That's the, do they have the, here in the States, they, they, there's something called the squatty potty.
Do they have that? Have you heard of that thing?
I have not heard of that thing.
I don't even know. It was probably like, I don't know, it's gotta be 15 years ago or something, but I had a look at it.
It's called the squatty potty. It's just a piece of plastic that sits around the toilet.
There's nothing crazy about it, but it does the same thing where it like lifts your legs up and gets your knees basically at or above.
Yeah. And then, and then I think like it was, I don't know where I heard about me was Howard Stern, maybe back in the day or something like that.
It was a radio jock here in the US.
It's been around for 30 years or something or more. And he talked about it a little bit and then, and then I'm going to have to, I'm going to have to stop you there.
Cause there is a website called squatty potty Australia and, and they are well, I guess this is a testament to the product in that their website says we'll be back soon.
Sorry for the inconvenience. I guess they are not on call because they are probably enjoying taking their shit for four hours.
They might be. They might be. Yeah. Sorry. Continue your Howard. No, that was it. That was it.
So anyway, you could build that right into that. Maybe, maybe that's a product idea. Squatty potty that combined with like a laptop stand, have a little like spot for, I don't know, for, for the dev up elite.
With a little UPS in there.
A little coffee cup holder. What else do you need in there? Yeah, I can sell them to the Google H. Yeah, you get like a little separate like nine inch screen with like some Grafana or like Xabax dashboards in there that automatically scroll through just the problems dashboards up.
Just two, two 20 inch monitors, a little slot for your like a docking slot for MacBook Pros only. Cause yeah, and it's always updated to be only compatible with the latest model, a little UPS battery.
5G backup. Yeah. One of those little white little, you know, extenders for the 5G. A little Starlink backup. Yeah.
And you could sell like a little camp aspect of that too. So if you need to take it on the road, you can just kind of bring that thing out there and dig a hole or something if you need to dig a hole. Set it up.
Exactly. Sunshade for you. So what's been going on with your, your say shit, man.
So we've been dealing with, this is on Arctic one, which is the say EVM B2 chain, which has been going really well actually.
And then an area.
Yeah. Well, well that, that, so that that EVM stuff just moved to Atlantic two that that last upgrade that 501 that was all the EVM stuff coming into Atlantic two, which is why it jumped from like B3 to B5, I guess is why, why that happened.
And so on Arctic one, we have three nodes that are sitting behind a couple balancers and and we've been having this issue where basically, which we saw before, which was that nodes, not necessarily at the same time, but they would eventually
keep growing in RAM, like they would be fine, like, like two or three gigs and they would eventually get to like, at some point, I put a cap on it, but at some point they would run the RAM to the point where the box would stop responding.
Like it would completely, it wouldn't.
It wouldn't.
It would lock the box where I couldn't do anything like it would like 100% right.
Yeah, it wouldn't do anything.
And so I ended up putting some system decontrols in there, which I need to do actually, I need to do that overall, which, which is just just put a memory cap on every single one and I think I'm going to do that in Antswell where I'll take basically, maybe a look at the total amount of RAM in the box and just cut it to say if it takes half of this and kill it, right,
is something like that because usually my box is like all the stuff should have run is probably 128 gigs.
So if you hit 64 gigs, something's fucking wrong on a cosmos change for sure.
And you can't like you can't is there can you use a percentage in that?
Yeah, sure.
Instead of actually having used like a hard number.
Yeah, yeah.
So in Antswell, I would just take I would take the number because that's an it's an Ansible thing that comes in through through get fax and that was just divided by two and just say if it hits that kill it.
So I'm going to do that.
I don't get fax though when well, you know, with my it's why I have get that sound of I do whatever I need because I because I do because right now also what's really fucking this is totally wrong really.
But I usually run Dev 12.
I run Debbie and everywhere.
And then I've been using so many different providers and different things and like Debbie and like not everybody supports and some people just use a bun to and I'm not a huge a bun to fan, but whatever.
What do you have?
I've noticed that you you run Dev 12 like over bun to what's your what's your grub and you bunting it.
I'm over 50 and.
I was born I don't like it pierced pierced no, I it's it's all Debbie and based and I've always I grew up with Debbie and and that was been my thing.
I ran like I mean early days in Linux.
I ran like the whole I went through the whole jet to fucking thing and then I ran all stuff within red hat.
I read about a ton of red hats.
I know I know the young structure really well.
I had an arch period which I ran arch for a long time, but I've always had Debbie.
I've always had Debbie.
So I don't like that.
I know that so I use 22 I for Ubuntu.
It's the same thing, but for some reason.
Yeah, well I is is Dev 12 ship with 6.1.
Does it?
It does the chip is six, but though though all my bunch of stuff I pushed a six anyway.
I use whatever we do package.
What's that package for it basically it's I'll tell you what it is.
Oh, there's like a package that puts all the okay because well I it puts it Linux kind of so much better dude.
It's like it's just I still use kernel six.
I just build it every time.
So I install I'm installed Ubuntu 20 to 04 and then I just upgrade the kernel.
What do you do like like manually?
What do I do?
Well, no, there's a package you can install.
Oh, yeah.
Yeah, sometimes sometimes I build it.
This one is what the fuck is this called Linux generic hwe dash and Ansible distribution version.
So that'd be Linux generic hwe dash 20 to 04.
I think is one of this.
That's what that package is called, but that basically brings in the hwe version of that kernel for Ubuntu.
What the fuck are we talking about this anyway?
Why are we talking about Ubuntu?
How are we in this situation?
You said you're a dev 12 boy.
Yeah, but what does that do with fucking say?
Where do we how do we get down this path?
Back to back to say anyway, you had a problem anyway.
Anyway, what happened man?
Where am I?
This is the story of my life.
Like we started like here and then we ended up like down the road.
Anyway, back to say so we're so we do we have a few nodes that are running on say for artic.
For artic one, which is the EVM type of structure.
And what's been happening is that not at the same time, but over a period of time, we're starting to see that the RAM usage basically shoots up like two or three gigs and it'll shoot up to like 1012 and then it'll continually grow.
Oh, you're talking about RAM controls and then it'll continually grow and then I put in the lamp controls and basically at 40 or 50 gigs.
It starts to slow it and then it'll eventually oh and but before that point once it gets to 4050 gigs what ends up happening is that the stop the note stops responding it stops to it can't keep up with the block RPC stuff shit shut off all this kind of shit right.
And so we're trying to figure out what the fuck is causing this because it's clearly something related to we get a lot of volume on our artic one like we have a significant number of RPC requests.
The say team, their nodes, they haven't seen this but they don't get the volume that we do.
We probably get, I don't even know how much volume they get, but we get we get a decent amount of volume on our artic one.
And so, and so they, I gave them I gave them access to a box and then they've been running a couple of different modifier binaries they've been doing a lot of like core dumps and like other types of peeper off type stuff and like trying to identify what's going on.
And then we had the identified an issue which might still be an issue but it was a separate issue that that on the Web socket on EVM side basically it wasn't like giving up.
I feel exactly how they call it but it wasn't it was creating a socket for every connection that wasn't reusing sockets and some sort of thing where it might have been chewing up ram and it was like doing whatever.
So we did a patch on that and then that was so we spent, excuse me, but spent a bunch of times a couple days ago doing a patch on that and then just kind of going through like what what the issues are and like it only happens every like three or four hours so it's really it's really
difficult to you know, troubleshoot when everything takes three hours between kind of runs.
And so we get through all that other than else and then yesterday or this morning actually we started getting we started getting all of these.
When that was happening we'd restate stinking node and then we were getting app hashes.
And so it like app hash, like after a point and then one time like all my notes app hash or they would all app hash at the same time or similar to the same time.
We're like, what the hell is this now. And so then we got a call is we're going to call for like three or four hours today trying to figure out just focus on the app hash because without the app hash you couldn't solve the other issue.
And we started to realize that the networks were so pretty small and so we're pulling snapshots from like the same couple nodes and one of those nodes is theirs.
And for some reason we determined when I get a snapshot from them it doesn't have all the wasm files.
So there might be 70 wasm files and I get 60 and the other 10 don't have the dot wasm on the name they're just the they're just the like the hash without a dot wasm.
But the snapshot only gives me the dot wasm so I was missing those and then obviously it had passed.
So we're trying to figure out a way to so we couldn't figure out why that thing was named that way and we try to figure that out.
We took us a long time to figure that out.
And then we got to the number one like make sure we like understanding all that and I couldn't get another stint sig node.
It was fucking weird dude.
So so you were sorry you were getting snapshots or you're getting state sync.
I'm sorry they say snapshots stacing we're doing state sinks and then what state sync it wouldn't give me all the wasm because the state sync process would ignore files that weren't dot wasm files.
So the wasm files were in there but those nodes weren't were an app hashing so it would actually run against that wasm file even though it didn't have the dot wasm name on it just had it just had a hash name on it.
And then out but those would never get delivered in the state sync so I would have left of the cop I would have less of those.
So every time I was trying to troubleshoot this RAM issue I would state sync and then then the contract would call node app hashing so it was like all this shit at one time trying to figure out like.
It took us a while to even figure out there was your two separate issues number one because you look at it you say oh this like the fact that the RAM goes up or it apaches that could be the same thing it might be just a different instantiation of that thing right.
And so so that the apache issue we traced back down to a specific say no that I was calling from an RPC perspective.
But it was actually a center I was calling a different say no for RPC data but it was pulling snapshots from one say server that had snapshots.
And then for some reason it was always pulling from that same server and I couldn't get rid of the address book like all this fucking shit dude.
And so and then we figured out that this was the issue so they're going back to confio on the question around that because I'm not sure if that's related to the upgrade that they just went through or something else but it's really weird that that wasm was in there.
So anyway so we figured that out I think that might be solved because we know what the source is and then.
Yeah that kind of sounds like they changed convention at some point and then it does right and then change the state sync at some point to only draw in things that conform to the new convention and then there's just like chaff sitting in there from previous stuff which is.
But but also I looked at that node that node they stay synced early this morning.
And their state sync script I'll send you to copy the script like they wrote dude they wrote a huge pay in new state trip so what they do is which I thought was actually pretty cool.
They saved the private key out they saved the node key out they completely delete the whole say directory 100% they dump it all they re in it with the moniker name they save the moniker name they re in it with the moniker name they move the keys back in and then they run the state sync against a different node.
Why it's totally new can pave I don't know but but to that point I was like oh maybe you guys are like that's what I thought I'm like well maybe you guys are not deleting the wasm directory or you like the cash is still there or something else in that they send me the script was like holy shit like.
I guess it's so that maybe it's just so they are they're always on the latest config.
You know that's what I thought too and and because we do run into that issue because like say moves fast like with say DB or with this with the what's the other thing that just happened.
The.
The concurrency thing that that that concurrency thing where they run transactions concurrency maybe maybe that maybe you guys can see that but that's an Arctic parallelization.
Thank you yeah that was parallel.
Power is it only that's only be able to think right yeah so that's a new config section.
So what I've been doing is like I basically been when I do those things I basically go back and run ansible to delete that whole node which is basically the same thing and then re in it and then reapply right a bunch of shit.
And state thinking I mean that seems like a little bit much for state thinking because I think I would just manually do it but but yeah.
Yeah like I can kind of see but it you know you don't only you don't need to do the new can pave I would think if you had like an upgrade that you wanted to test or something but.
Or like yeah config changes I just thought like because that because.
It was I thought it was interesting because it's cool I think it's totally fine.
I thought it was interesting because we were talking about just the idea of having old shit in there at the same time like they completely.
Like oh yeah there's clearly no old shit in there like it's completely wiping the whole thing out so on on.
You know they might do it because of there's an issue with say where you can't actually use the tendermint unsafe reset or the comment BFT unsafe reset whatever it's called now.
So you know my state sync script actually it'll just delete the wasm and data files and then just create a the problem is is the creation like it'll work the unsafe reset but it won't.
I don't think you can then start a node from it or it doesn't work but if you just delete the data and wasm directories you have to then create the state file and now otherwise the node won't stay.
Which is a bit of a nuance to say so yeah well I mean my yeah maybe that was there fix for that they just like fuck it will nuke the whole thing and start again.
Yeah well I mean my because I don't know if I'm state syncing RPC node or a validator or whatever else right whether it's a signing validator if it's just a node so I always might my approach on because that reset was going away so I just moved the key out.
I'm sorry I moved the state out I delete the folder I recreate the folder I move the key back and I delete the wasm directory if it's outside of data and then it just states in from there right that's the it's just a safe process.
There are some interesting things with the various chains that some of them you need to actually get the wasm directory when you do like when you restore from a snapshot say some you need to get the wasm directory.
How does he don't I don't know so that's a good question actually so I never do it I just get the data directory and it's so sometimes I have problems with that sometimes I don't.
So you should check that because like polka I know I think I think she'll see the same thing I do the same thing if there's if the wasm is outside of.
So my process on taking six snapshots is that I would delete the caches out of those number one is what I would do is do anything that's a cash folder both and whether it's whether that wasm director is in or outside of data that cash folder gets deleted because that's architecture specific and that can fuck things up.
And then that gets added to the tar so if I tar it that would be the data and wasm or it'd be just the data if the wasm is inside.
And then so when I when I when people untar that it would automatically be in there because you're right like if those contracts are not there.
If you add a new contract and the chain is running the obviously the binary will download it right like it has a copy it'll create that wasm file because that contract came on chain and that and the file saved.
But if it's not there which is what I ran today and it's instantiated it will clearly app hash.
Interesting.
So and state sync the one the question that we did ask today is I think maybe maybe she'll say you know this answer.
So state sync when I get the chunks right so say there's 50 chunks associated to a snapshot and we state sync that down and those 50 chunks come in.
Does the wasm structure.
Is it part of those chunks number one is it comes separately and number and number two.
Does it have to come from the same source as a snapshot.
What do you mean does that come from the same source source of the snapshot.
Well I'm picking like that thing it searches around for snapshots right and it finds a peer that has the snapshot at a specific height right and then there's so many chunks associated that snapshot so it says.
Okay I'm searching around I found this peer at block height 100.
Here's the peer ID right CC Web all.
And then it says there's 56 chunks and it'll grab those 56 chunks right it'll fetch those 56 chunks from that peer.
Is the wasm structure in that in those chunks or is that just the data and if not where the fuck does the bottom files come from.
Yeah it's part of those chunks but it's also never dependent there are a couple of numbers I think tears one of them where you do have to restore the wasm directory manually like by via snapshot you can state sync the data but the wasm doesn't come with.
Another important thing to know is that for state syncing you don't stay sync from a single peer you use kind of like so you have those initial state sync peers right those work is for lack of better term like a light client to then connect other peers and then it
like goes through them but the snapshots and stuff themselves come from all different peers.
So that's kind of the point right is that you'll find a peer with the accepted height and it'll find that and then if it finds a lower height from a different peer it'll take that as well.
That's kind of why it's important that it comes from different peers because most of them probably like if you're going 15,000 blocks back.
Most are gonna say 15,000 blocks of state syncing snapshots right so it can go to find to one of the one or two peers that have that and then it can get the rest of the snapshots from other ones that's why you can sometimes find like state syncing doesn't go linearly.
I find that yeah go ahead.
Okay, no I think I'm understanding that the snapshots are coming from different peers but once I select this once the node selects a snapshot okay we have a specific height and there's 57 chunks in that snapshot.
Is it assumed that all 57 come from that same peer?
No.
So it might be that there are other peers that have the same snapshot at the same height because it has to be obviously the same block height right and each individual node operator can set any sort of interval on their own like I could set 666 on the fucking interval and it'll create a snapshot every 666 right.
Then it has to be exactly on that height because it's a snapshot at a height.
Yep.
So it might find another node that has a snapshot at that exact same height with that same hash.
I think you're sticking down the exact same height too much though so it's it's reducing that height as a method for building the chunk like tree.
But if people have snapshots like let's say every 2000 it can be like if someone has every 2000 every 3000 they're going to take more or less the same have all spacing the same chunks right like there's gonna be 1000 apart where they're gonna be different but most of the chunks within that same block are going to be the same you might just be
the same as the whole and so because they're missing a couple they're not going to take them from them obviously they'll go to a different pier.
Does that make sense.
It does but but the but the snapshot is not the whole chain history it's just the hash of that chain at that point in time right like it's not like grabbing all the history associated like 57 chunks doesn't make up the whole fucking history right so.
So I guess it's okay maybe it's more complicated and thought but back to the wasm thing I don't get this is nobody's gonna give a shit about this.
Who wears like we need like who would who would be able to help us out with this.
A soft or soft and soft I know he did quit wasting my time with this question this basic question.
Yeah.
So anyway the and the wasm piece and maybe Simon to would be great.
But the wasm piece was really interesting because it was weird that we had a node that was a state so the node was absolutely still in sync and would run contracts because.
It had the contract right in the wasm directory was there and had each of those wasm files but there was a mix of not that was just be a hash or whatever that is like some sort of ID.
And that ID with that wasm and when I pulled a state when I pulled a snapshot from that from that peer because it showed the peer ID on the snapshot I was pulling it wouldn't pull it would only pull the dot wasm files which means I was getting.
50 contracts even though that node had 60 contracts and so when it would call one of those 10 I would have hash it's crazy so.
What why why why why was it named different okay.
I don't know that that's a question so that we went back to confido to ask like why was this and that knows the state sync today so like why was that node had different level like it had some had wasm on and some did not.
And.
And it looked like it was older contracts that was and so we created a new one and that one obviously had that wasm and my node had it right as soon as I state synced and all that kind of stuff.
But it was really weird so it took it just took a while to figure that out and figure out why why nodes were ap hashing on something that you know just writing a simple state sync on a network.
So anyway, it was like it's just been and that's sort of stressful because there's a lot I mean there's a lot of hype around.
Like anything else right it's a new chain right or this this new iteration of this chain and like there's a lot of teams that are building on it and like I don't want to run nodes that are fucking ap hashing and you know are down and we're getting no server to handle this request type of situations because.
Because of issues that are not outside of my control but like somewhat out of time and control and and so like I'm a I almost I almost pulled no script and you're a lay oh script.
I was gonna watch dog of fortune.
I was literally I was typing it up I had it I had it up in in code because what I was gonna do is I was gonna watch the block height and if it's tall for like.
Eight seconds I was gonna fucking state sync the node I was just gonna delete it all move the script restate second and just get a second like get some rest because.
Even overnight like it would it both all three nodes would would hash out or not hash out either hash that had hash out what's happening more randomly but they would you know fail and memory or something else like I can't I don't live like this right I don't have these issues.
Well that's what I'm doing with science watch dogs for the win just put a watch dog turn off monitoring fucking nuke and pave everything yeah I don't give a shit yeah.
Yeah tell us what's going on with Zion I want to hear about that.
So for Zion.
There are so we are running a.
Load balanced endpoint for the team as well as like the main Zion team and they're also kind of like subletting it to other teams right sure.
And so it's taking a lot of a lot of heat which that's not a big deal.
We haven't figured what it is yet but I think there's a query coming in that when it does come in it just apaches the node and.
Is it it's probably concurrency if like if I'm has it in a guess I'd say it's concurrency because that's every time I've had this same shit go on another change even say in the early days it was always concurrency and it'd be a contract call which comes in through the g rpc does some concurrency bullshit and the apaches the node anyway that's my two.
What do you mean by concurrency tries to do like two transactions in wasm at the same time which are related and there's like a concurrency issue.
Oh many like it's shoving them both in the same block and once dependent on the other type of idea.
Yeah I think there's like two VMs running that using the same shit and then they just you know they're related they use the doing that are related.
I believe that's what it is and then something just puts up concurrency issue and that patches and then it's fucked and you have to restart so.
So then what what what did you do to resolve that so the way it's been.
Complain and complain to the team until I fixed it.
So that you like it now.
Yeah, back to our point.
Yeah, exactly.
But yeah the way we've been resolving it right now is I've got one note off the side that takes a new snapshot every hour because stacing doesn't work on Zion so.
Why are you.
I'm clear.
I think they may be like is it.
Is it errors off bro.
Does it does it.
Does it apache on after a state center it never grabs stamps or doesn't whatever.
So get snaps it fails if I remember correctly there's an IVL store error that happens to the very end.
Got it.
So good and thoroughly nuked something snapshots very regularly as soon as an apache is detected and it skins every few minutes stops the node restores a snapshot and kicks it back going.
Fucking fucking.
I'm gonna grab a little and bro.
What a glamorous life this is.
I'm glad we're sharing those everybody.
This is what it is folks it's fucking you think it's like a 100% oil machine.
Meanwhile, fucking watchdog scripts that new can pay.
It's a never ending saga of no sleep and last resort typing at bash.
So you're so that those nodes only apache.
Those are the nodes that are actually receiving the transactions.
Right.
Yeah, so that's a different app hashing.
Consensus issue.
Yep.
Correct.
I've gone through and looked at caddy logs.
I've looked at all proxy logs and there's nothing that stands out.
They also I've been piping the logs to the team and they also haven't seen anything that stands out.
Everything looks fine.
So it's like, go ahead.
So is it how many nodes are in the bouncer?
10 10.
10.
That's a lot.
Yeah.
Fucking 10 at one endpoint.
Yes.
Are you doing like geo one endpoint?
Holy shit.
So 10.
Yeah.
Are they all same region?
How many fucking how many fucking transactions are you seeing?
Like how much how much RPC requests are you getting?
60 to 100 requests per second.
Oh, that does not match.
So what's what's what's your balance strategy?
Round rolling.
Pretty simple.
And then if one fails and it just retries to the next note.
So is it just the node that is so what what actually have app hashes all of them?
No.
So the one that so typically what happens is one lap hash and then it'll app hash the next one in the line because it'll retry.
Can you can you balance based off of source IP so that in that situation to Mel's point, like, is it because they're I'm not sure I'm not sure why the first note would have has to be honest.
But but is it because those two transactions need to go to the same node?
Wouldn't think so.
I wouldn't think they'd be going to the same node.
If it's currency issue, you'd want them both in the same you want them to both in the same mempool.
Right.
If if like they're probably batching transactions though, if they need to be done together, if they need to go on the same block that'd be batching them.
Like on the VM side and the not so much on the tendermen side to be honest, but on the VM side, I make sure that based on source IP, they're going to the same node because on the VM side, I know there is a there is a tremendous amount of there.
It's a little bit more complicated.
I don't really know it well enough, but I've seen this and also on other nodes like associate to geth and some other types of things even the Ethereum side, even though that that that's a little bit less of an issue because the block times are so longer.
But on other nodes that are that are geth based that have a much shorter block time like I was a good example of that.
Like I make sure that those notes are going the same one.
So so you might want to change the balance to not be around Robin.
You might want to say at least as a test.
Well, you can round Robin.
You can round Robin, but just have.
What is it called affinity IP affinity?
So that I mean, yeah, it'll round Robin to new users.
But once it assigns like a server to a user, it'll stick to that until it becomes unavailable.
Like HAProxy, I would I would balance based off of the wall.
Are you using HAPho?
Shilty using catty.
No, no, but catty catty can do that.
Yeah, yeah, I would look at this.
But is it behind Cloudflare?
No.
Yeah.
So you have to look at you're using Cloudflare and you've got multiple proxies.
You'll have to put IP affinity in Cloudflare.
And IP affinity in HAProxy or catty.
Yeah.
In catty, you need to look at the header called connecting dot dash IP because the IP is actually the Cloudflare IP.
So you have to look at the header for connecting IP.
And then that'll pick off the header, which is the client IP to make sure the client IP gets routed to the same node.
And then yes, right.
And forward IP, but yeah, same difference.
No, not forward IP.
You want to look at the header.
There's a header called connecting IP because that none I even in catty.
No, that's if it's proxy through something like Cloudflare, I'm pretty sure in catty, you need the forward IP, which is like before it's the previous proxy.
Otherwise, if you use the if you use the source IP, it will just give you the proxy source IP will give you the proxy.
But Cloudflare sends a specific header with the client IP.
And that's called connecting capital C dash IP.
That is the Cloudflare connecting IP.
That means that's the IP that can ask the request.
That's what you want to that's what you want to balance off of.
We can still tell everybody.
I can look at that.
Yeah.
And then and then the nulls point in Cloudflare, you want to make sure that you have affinity to the just means that like, hey, send it to the same balancer.
Yeah.
To be the right, the right term, 10 to the same origin.
And then the origin server would then but if you only have one origin, you have one balancer multiple balancers.
Oh, sorry, multiple balancers.
Okay.
All right.
So then if any of their well, one, so hang on one, one balancer in Cloudflare, but that goes to multiple origins, right?
Like you've got multiple caddies running.
Yeah.
So you need session affinity to the, the caddy that it originally sends it to.
Yeah.
Okay.
That's a cool idea.
I will do that.
I don't know.
I think it's like this idea me sir, because we had this problem like years ago, right?
And that's when we came up with that fucking affinity thing.
I've had this problem on say ends osmosis.
Not so much.
Yeah.
I haven't had so much on the tenement side.
I've had a little more new VM side, but, but it's not bad.
I mean, I mean, I don't, I don't really enjoy that too much with like, I mean, we have a lot of clients that have like fixed IPs.
We get a lot of traffic from the same IP.
I don't know if I want just, you know, whatever.
It doesn't really matter.
I have to be honest.
And then, and then HAProxy, which is like a, like, like a child to me, but that HAProxy,
then, then I do balancing based off of like a ton of different shit.
And then I love the failover aspect of that and the controls in there.
But that might, I, so it's a long shot, but in those types of situations, if you are getting a shiller
transactions in a very short time span, that is a good example.
The other thing you could do is just expose one port on one node and, and have them try to just one node.
That's another way of getting around this.
Like you have this giant fucking ten node, whatever the fuck you're doing, but just expose one thing and say, hey, try to get into here.
And if it fails there, like you just have one, one node with a, you know, 57 port open, it fails there.
Then, you know, this isn't the issue.
That's a good point, except the fact that there are a bunch of different teams involved that are hitting it.
And so I have to go to each team saying, Hey, only use this and see what happens.
I think that's where the issue is coming from.
Cause I think that if I were to, if I were to point fingers, this wasn't an issue until no most came up and it's happened basically since then.
And so if I had to guess, it was the no most team, nothing to do here in Fisher or something, just that whatever's happening with their utilization has exacerbated the issue.
They are doing lots of crazy shit though.
We think it looks 10,000 fucking IBC channels.
Yeah.
Well, that's expected though, because ICA is a new channel per user.
So that's, that's totally normal.
What?
I was waiting for it.
Look at what?
Yeah.
And they got just an insane amount of channels and what is your goal?
Wait, wait, wait, wait.
Per what, what kind of user?
Like me, like a person?
Yeah, so for every ICA account, yeah, like if I create an ICA account, an account over ICA, what's the entertain account?
Yeah, then you'll get your own channel for that.
If I do, if I do one transaction, that channel is created and then it's there forever.
Yeah, basically.
What the fuck?
That's, that's an overhead.
The strange love know about this.
Like, you know, is it built for that?
It's not really built for that.
Is it kind of it?
It handles it fine.
Like you basically just do a wild card for channels.
The problem is that, I mean, they need to stay open, right?
And so the more I see channels you have, you're pulling now for more and more channels and keeping more and more channels alive.
It's kind of like a recursive hellhole.
Could you imagine the, the bloody chain?
What do they call it?
Chain the poor registries for, for that chain.
The channel is just 400 pages long.
Yeah.
The poor mid-skinned, IBC channel team.
Oh God.
Oh my God.
But, but once that channel is open, doesn't expire?
Yeah.
So that's why we had to keep pulling for it.
So in theory it should, but no, wait, wait, wait, wait, wait, wait, because it has to,
has to expire.
It was a two thirds of the unbonding period, right?
So does that mean you have to run?
Wait, I don't understand this at all.
Does that mean you have to run a transaction through every one of those channels, every
two thirds of the unbonding period?
So I'm not sure how that works for ICA specifically.
It might have acquired that or they might have done it differently.
I would guess it to have it differently.
You're right.
It's not true.
IBC, right?
It's not a regular IBC channel.
It's truly an IBA.
It's definitely different, right?
Yeah.
Okay.
All right.
I was thinking in my head, it was like an IBC channel per user.
I was like, what the hell is going on?
Well, I mean, it is, it is a channel per user.
I just don't know if it has the same rules around channels closing.
Yeah.
And I think strictly speaking, if I were to guess, I would say that it does.
But for IBC, you can just do an update channel transaction, which are very simple, but it
would still, you know, increase state.
Right.
Okay.
So, this is like a technical fucking episode, man.
We're like deep in it.
Phrasing on here.
Maybe we, is it?
Is that what gets our viewers?
Hey, there you go.
Sup, rockaboy.
Look at you.
I like this.
Let me, let me get, let me fix things here.
Yeah.
Different this way.
And what's with this big ass couch?
I can't use the computer upstairs because everybody's already asleep.
Where's the fucking mess, man?
Where's the mess?
There's no racks of shit.
This is like, I'm taking a chin view, not the, not the, this is like a whole new fray.
Yeah.
I mean, I'm not on my desk.
I'm not on my desk.
So, obviously that's not so.
There you go.
Maybe turn half the lights off and shine a flashlight.
Oh man, you look like a completely different person like straight on than the chin view.
So funny.
How was your, how was your prog rock situation over there?
Give us a little.
I would say that the band prognosis were substantially better than their band name.
It was kind of technical, modern British, I would say British progressive tech metal
in that kind of, it like kind of post taken post Tesseract kind of vibe, like a little bit of atmosphere,
but still like lots of quite heavy poly rhythms, but not like quite as Gonzo in your face as
the American stuff, like the periphery animals as leaders type style.
There was one particular riff that was very animals as leaders.
It was very like modal over like a poly rhythm thing.
And it wasn't like, you know, Scandinavian heavy, like, you know, in the sugars and whatnot.
I didn't have that kind of dissonance to it.
They know it probably short set, but it was, yeah, it was interesting with good, much better than their band name.
I without my glasses and I think the other guitarist was wearing a silk shirt, not a kimono as I had initially thought.
But I think they have, they have, they have things ahead of them in the UK tech metal scene.
If they change their band name, that would be my review.
Such a fine line between silk shirt and kimono.
I basically I left the venue and I realized, I realized actually I look really tired because I am.
And I saw one of the bands smoking outside and I just went, I'm sorry, I'm probably literally never going to see you again.
So I've got to tell you, you're like an eight year young tech metal band and you need to change your name.
Is that what you're saying?
Yeah.
I was like, the name is maybe the name word. What do I know about anything, but I think you need to change your name.
Peace. And then I literally disappeared into the night, smoke bombed away.
You actually said that to them.
Good feedback, man. You got to get good to them.
I do see.
They were talking to like somebody else, I don't know, or a friend and that person just look at broken every rule of British.
That person looked mortified that I was being that honest is we don't do that.
Your name sucks. You need more 17 eight.
Peace.
See ya. Where's the tube?
They do have albums on Spotify.
They have one with a huge eyeball on it. So that's always like a red flag, but you know, so is the name.
So, oh, there's prognosis radio. Somebody actually built a radio station around it.
I'm all over this.
Yeah, pretty good.
Say that was what I was doing.
Nice.
We've been talking. We've been actually in pretty deep technical talk to be honest.
Been deep in it.
Yeah, we've been in it. We talked about wasm shit. We talked about
Schultz working on weird concurrency issues on g rp. g rp. See was a g rp.
API.
I was able to disable g rp. See just to kind of reduce that.
We talked about maybe some load balancer structure, which I'm sure he's working on right now and ignoring the rest of this podcast.
We talked about
He's got something that might be promising that will eventually disappoint him like the rest of the shit.
We talked about injective pego and shitting on that a little bit.
We missed a lot. It was a good show.
Yeah, but there were more riffs where I was and as a bonus, the basement did smell like piss.
Did it?
Like all good.
Very authentic.
Was it a real basement?
Post.
Because the hip thing is right. It used to be that you could smoke indoors and that was obviously really bad for all of our health when we were all like dying.
And that was bad, especially if you worked in the venue. Obviously you were like dying and that was super bad.
And I do recognize all of that.
But the venue smelled like cigarettes and not like piss and now they smell like piss.
Are you suggesting that the smell of piss is worse than cigarettes?
Like is that where you're coming at me with that?
I think the smell of piss and especially gig venues because it's actually kind of piss and strong bow, but it's like strong where it's been on the floor for a while.
Yeah, it's not like new piss. It's old piss.
There's a difference.
No, the piss is just a background miasma, but the strong bow is like fermenting again.
Yeah.
Strong bow is a cider. Not your cider, hard cider, like actual cider, like the meaning of the word.
Not apple juice.
So funny story about smoking in piss.
In Australia, there used to be basically every bar you'd get.
Bars have changed a lot in like the last 10 years in Australia.
It used to be every bar you'd go into.
Well, it's probably been almost, it'll be 10 to 15 years ago when they banned smoking in pubs, right?
And dance venues and that type of stuff.
So when I first went to college, you could like be on the dance floor burning the fuck out of everyone with your cigarette and come home with like your dreadlocks smelling like fucking smoke and then your bed and your mrs.
Hold on.
Smoking ban, notwithstanding, you had dreadlocks?
Yeah, man. The banging is fucking dreadlocks.
You had white with dreadlocks.
Oh my God.
That's bullshit.
I think I need to see that.
You're fucking, you're bullshit.
I can, I can find some evidence.
I can find some evidence, but not with the band because that was in the period.
No, it would have been, I know it would have been a skate metal band like like corn, something like that, like a new metal band.
I don't know if I should be giving you the evidence.
Not right now, but literally, yes.
I don't even know if I should give it to you privately.
I feel like it's something you all like screenshot immediately.
Did you?
Anyway, so.
The block media is what will happen.
Yes, so NFT.
So the anyway, so back, back, back, back when you could like smoke in the pubs.
There used to be at every bar that'd be like an astray that ran around the perimeter of the base of the bar.
Did you have those over there?
I'm sorry.
There would be like a dished masonry at the base of the bar.
So, you know, in Australia, you have big, long bars, right?
Big corner like the old school bars, pubs, the old school pubs.
The bar in the pub would have like this dished masonry bowl that just went right around the perimeter of the, the bar.
Okay.
And people would like stand there and drink their schooners, having their cigarettes and then piss in the astray.
So that because they could do it in a crowd, like if there was just shitloads of people there, you could get away with it.
I don't understand.
Like, what the guy, you just stand at the bar and then they'll just piss against the bar wall and it would go down into the astray.
Oh, I see.
So that was at the base.
Yeah.
Yeah, bunch of fucking, fucking, I'm out of rage.
This is why you all got deported, man.
You'd wonder why the one guy had been standing at the bar drinking schooners for fucking seven hours and not moved.
Like how, what was he doing?
I was going in the astray.
It's a whole new world to what's been waking right, doesn't it?
I have no idea.
We'll give you a laugh for a moment.
Did it drain or would it just like as the night went on, get deeper and deeper and deeper?
Yeah, just like, just get deeper and smell like this.
Just get deeper.
God.
And then probably never get good, probably just evaporate the next day, you know?
Yeah, I'm sure that's exactly what happened.
That's pretty interesting.
Just evaporated.
Oh my goodness.
What the hell are we talking about?
We have a minus.
I can't believe I probably spent some hours going on.
I was like, hey, news.
You want to do news? The one line of news?
Or fuck it.
We've avoided news already today, like say, we're just,
we're on episode 113 news.
There's literally no episode 113 really on like,
but it will have news.
It will have news.
Yeah.
And that news is that CZ got, got sentenced finally.
So, so CZ from Binance pleaded guilty.
Oh, let me find that.
Let me find that.
Let me find that better piece of news or source.
There's actually a pretty funny quote in this news.
I'll put it up with a link to there's a, there's an article on CNBC,
which is not the greatest source, but whatever.
I thought it was pretty hilarious.
But CZ got sentenced.
He was really guilty to money laundering here in the U.S.
and a couple of other things.
And so he got sentenced.
The prosecutor was looking for three years.
They ordered Binance to pay 4.3 billion and Zao agreed to pay 50 million.
Kind of a gap between those two items.
They wanted, they wanted him to be in jail for three years.
He got four months,
which really means that here in the U.S. four months probably means you're out in 60 days.
Maybe.
And it's like, it's not even worthwhile.
Like you're, you're done.
Right.
Um, he did have to agree to step down as part of his, of his agreement,
but he owns like fucking what, like for, he owns like 90% of Binance, right?
So like there's nothing, he can't be CEO anymore.
Um, yeah, it's more about that though, isn't it?
It's more about the administering the slap on the wrist.
And it's the psychology of saying we are in charge.
We are in control, right?
Oh yeah.
So yeah, and like they're, obviously they have a new CEO and, and all that kind of structure
associated to saying that they're going to build more robust, robust compliance structure
and all that kind of stuff.
And like, which, and like, I'm not defending CZ like, like it was, it was definitely Wild West,
right?
And there was a ton of shit within there.
He's also not operating out of the U.S., but whatever.
Um, there, there was a, there, we find this great quote that came from the prosecutor.
I think this must have been before the, um, the actual decision.
Um, he was just saying that, that's, prosecutors said that Zao violated U.S. laws on unprecedented
scale.
Zao bet that he would not get caught and that if he did, the consequences would not be as
serious as the crime the member random said, but Zao was caught.
And now the court will decide what price Zao will pay for his crimes.
And that price is a fucking Malibu beach house and four months and he keeps four billion
dollars.
Like it's totally worth it.
Like I would do it for four billion.
Like it's fucking 60 days.
Like no problem.
I'm in how hard are you going to be 60 days?
What's the damage to people's lives?
There's actually, this, this article goes around a lot because I mean, I mean, look at an SBF
like Sam caught 25 years and CZ here is at four months.
And I think this, this article does point at, I mean, as FTX in terms of causing harm
to investors is one thing.
And the idea around just like regulatory and compliance bullshit.
Um, honestly, juries don't give a fuck.
And so I think that is kind of the, the message here, which is, which is, Hey, individuals
were able to do what they wanted to be able to do and as a free market for them to be
able to do that.
And whether he's in compliance or not, this is a, I mean, obviously the dollars are huge
and they're, they're big dollars in here, but like the jail time is nothing like this
is, this is pretty significant.
Now he's still, I think he still faces some other shit.
If I remember, because this is only, this was only federal, right?
And he still has FTC and something else I think coming.
So it might be, it might be a bit more of this, but this kind of sits atone.
I thought, um, so anyway, interesting.
I mean, I think the, the, the interesting thing out of this is the, like it is what
you just said there, right?
Which is in a lot of other jurisdictions, the, the threshold for even doing anything
is to use a harm.
That's the thing.
Yeah.
Um, and that's kind of what this, this ruling reflects.
I think the, the more twisted bit of it is it's once again, kind of, you know, like
you said, they don't, they're not based in the U S like they happen to operate.
Yeah, a lot of their markets in the U S, but I think that's the bit that is, I think a
bit insidious because it's like, well, for any of this party to work in the long term,
um, it's the, it's going to be the question of who sets the rules, right?
Because it's a multilateral, we're in a multilateral world and we're building software systems that
can work in only a multilateral setting.
And the idea that the U S gets to call the shots, like, yeah, you have aircraft, aircraft
carriers, but, uh, like, I mean, well, it does mean you can't control data sensors.
So yes, you can't control the internet, but like, I don't know, it just feels, it just
feels kind of weird to my side for punk bones to be like, I don't think your aircraft carriers
mean that you get to control and like, I don't know.
Fuck it.
I don't know.
It's late here.
No, I, I, I mean, I think you've seen that Napoleon film like to bring about the films
because that's what we really want to talk about.
Have you seen the Napoleon film, the Ridley Scott one?
No, that's a new one.
It's shit.
Don't watch it.
Oh, okay.
Well, it's not shit.
No, it's, it's worth watching, but it's like fine out of 10.
It's like a bit of a fuck up and it's way too long.
But there is an amazingly bad line in it, like 10 out of 10 dog shit writing, how did this
get past the, the writing room?
Is that the, is that the Joaquin Phoenix?
Is it?
Yeah.
He literally just shouts at the British ambassador, you think you're so great because you have
boats.
That is the line they went with.
It got past the writers room, it got past the editors and they put it in the film.
They gave it to Joaquin and then and said, yes, this is the line.
This is the line.
Please read the line.
This is historically accurate.
Taps a piece of paper.
Please read Mr. Phoenix.
Please read the line.
And that's kind of what, that's kind of what I feel like.
You look at this, this ruling and you're like, you think you're so great America because
you have boats.
Yeah.
Well, then again though, they probably are using Binance US as their cause of jurisdiction,
right?
Which makes it totally bad.
And would it also explain more?
Although the story is to never incorporate shit in the US ever under any circumstances,
otherwise you would get four months in prison.
Exactly.
Yeah.
Four billion.
Four months in prison.
Four months.
Yeah.
That's how they get you.
And that's it.
That's the news.
I think we covered it all.
Also, and for the record, Robot Jocks is definitely better than Napoleon.
And in addition, my other half, having walked in only halfway through Robot Jocks.
This isn't, wait, wait, wait, wait, before we go, this isn't like Dark Side of the Moon.
The only view of Napoleon was that Robot Jocks had a clearer plot arc.
Just saying.
This isn't like Dark Side of the Moon Robot Jocks.
You're talking about like true Robot Jocks.
You know, like sit down and smoke this with me and then we're going to start over and
start with a fucking money or something.
I tell you what I will do.
I will try and Torrent Robot Jocks, sink it up to Dark Side of the Moon and then like
put it on YouTube and get and get immediately taken down.
Robot Jocks is a good fucking movie.
It is a clear, has a clear purpose.
It's fun.
You got a good, it's got a good like, you know, protagonist.
Like it's all right.
You know what I did watch this week?
I watched Arena.
That movie, were you talking about that?
Yeah.
I was recommended it when I went looking for Robot Jocks.
Somebody was like, make Arena.
I think I think I watched that in the 80s because I remember it because I remember that
dude, whatever his guy's name, but he was like battling this giant grasshopper.
He had these huge fucking legs and like, I mean, literally it's like it's just a dude
beating up actors and in rubber suits and and like the robot from from Lost in Space
and shit.
But that that actually that movie was, I mean, the quality of the of the action in that movie
and like the sets and everything else wasn't bad for the that time.
Like it was a pretty decent movie.
I liked it.
You should go watch it.
It's on my it's the whole things on YouTube as luck would have it.
So they give that one away for free.
Not even hard to track down.
It's a it's on my to-do list this weekend.
Do it.
Highly recommended.
Well, all right.
Maybe the highlight.
At least a little bit much.
But it's recommended.
It's it's okay.
Like you put on the put in the background while you're doing something else.