We actually...
Okay, it's live now.
I can see it.
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent validator teams.
Hello, welcome to Game of Nodes, a weekly podcast on the cosmos from independent validator teams.
And this week we have Dan from Steakfish joining us to hopefully enlighten us with how the professionals do things.
Now, we've actually wanted to make this happen for quite a while because two principal reasons.
One, behind the scenes Dan did a lot of...
I guess like sort of talking to different validators and finding out what people were doing,
and stuff around Prop 16, that's where I think we first got talking.
And I was just super impressed by the way you sort of approached everything.
And I was like, right, we should get Dan on Game of Nodes at some point.
And then, and we were just talking about this in the pre-show,
when some of the Juno devs dropped the ball a tiny bit on Prop 20,
and may or may not have sent $32 million to the wrong place.
Which we did resolve, but we did do that.
Dan was quoted out of context in a coin desk article, which was not a good day for Dan.
But I think within context, actually, the points that you were trying to make, I think,
was a very, very good one, and we talked about it on Game of Nodes at the time,
which is about the actual responsibilities we have as validators.
So Dan, what was the, can you remember what the exact quote was?
So maybe I should preface, like, maybe, yeah, like, I am from Steakfish,
and then maybe some of my opinions will not be representative of the entire company.
I'll say that first.
Yeah, I think what you're saying is, right, the context was,
we as all 125 validators on Juno did not catch, like,
let's say, an act, a misplaced copy and paste error for where $36 million of the Japanese syndicate CCNs funds
would go into a shared smart contract that would be governed by the community,
got sent to a debt address, and none of us caught that.
So, right, collectively, we basically, you know, fucked up big time, like, collectively.
That is to say that...
Collectively.
Steakfish didn't fuck up big time.
We all really were validators fucked up big time.
Yeah, exactly.
And it brought up a really, really interesting question that I think quite a few of us
had been discussing for some time about some of the responsibilities that we should hold as validators
under the ethos of sort of this decentralization where should validators be validating the code they push out into a live production environment,
or if that is even a reasonable responsibility that we should hold,
because many of us are multi-chain validators, are we able to, in a sane way, audit every single upgrade that we push out?
Right.
Do we have the money to do that?
Do we have the human capital and engineering talent or even time to do that on other chains?
Or is there just priorities that we have internally for the different chains that we validate
where we can kind of take up that mantle?
So I think the issue with what happened on Prop 20 where none of the 125 validators did a double check
was probably not necessarily like a completely a failure or completely like our responsibility as validators,
but maybe it's just like the system that has been designed was not designed in favor of sort of the realities of what we have in a multi-chain system.
Because I would even go as far as to say that a lot of these POS chains were perhaps designed from a lens that there would be sort of like these maximalist or loyalist validators
that only support a single chain. And so all your attention is focused to that versus having many, many networks that you're making money off of
as you add new networks, but are you able to pay the appropriate amount of attention?
And I think before the show started, we were talking about how do some validators have hundreds of people employed or the resources for that.
So maybe it's like a quality first quantity thing, priority thing, the system design thing.
Because like some people would even say like maximalism, right, we're in this sort of kumbaya multi-chain phase,
but some would probably say that, and maybe I am included in that some, that like loyalty or maximalism is quite important to the security of a network
from like the attention that we spend as validators. And even like the participants on those chains themselves, as we clearly saw with the CCN Japanese Syndicate,
where they hold like almost a non-trivial amount of the entire Juno network in terms of stake.
They should be according to some of the POS security guarantees in a model of like sort of financial brinkmanship, not be willing to sort of act maliciously against a chain by dumping a lot of rewards
or having claimed a lot of those things. So maybe they had like another chain that they were more loyal to, but then even though they had more of control of Juno,
they didn't treat it as such. So maybe there's some interesting conundrums there.
I feel you have a question, Yusef. You're doing the thing you did before you ask a question.
I would say that Juno has some of the most loyalty of the chains that I'm involved with, and at least.
So if something like that does not get caught and included, inclusive of that is some of the best WASM developers and overall developers in the Cosmos ecosystem,
a heavy amount of loyalty among the validators, you can see that in the test nets and those types of things.
So if this happens within Juno, I would think that this is one of the best cases. Would you guys agree with that?
Best case scenario.
Best case scenario of having the right people involved to actually find an issue like this before it occurs, right?
Versus other chains that might not have nearly as much involvement or as much loyalty to it.
Yeah, I would have expected. I think what's going on in Juno is absolutely incredible in terms of the amount of participation that we've been seeing.
I don't think ever in history has like 99% of a chain or stake have voted. And then I think there's consistently, you know, a large majority of all staked assets voting,
which does lend to some of the majority.
Maybe it is also perhaps size of the ecosystem that helps with catching some of these bugs, right? With King Nodes, like, I know, is Kiwi is New Zealand.
Okay, sorry. So there's just like, I'm sorry. I was about to say like, oh, in your neck of the woods, there's like this privacy thing called KiwiCon.
And there's a really great like computer scientist and security researcher Peter Gutmann, who'd famously said, I think, or had talked about like this concept of like all many eyes makes bugs shallow,
where and then he called a lot of us, a lot of the security ecosystem out when we had found some when we had found like these pretty critical errors in the Linux OS that was not caught for five years.
And so something as widely spread as as Linux and and as carefully combed through the many eyes did not catch those bugs.
And maybe that can be something said to to what happened on June as well, even though there's all this loyalty and very great participation.
So this is also commonly an argument for practices from, I guess, like extreme programming that kind of school of thought, you know, the, the, there's a great website, I think it's called Never Work in Theory,
which is essentially, you know, every time you're at a programming meetup and somebody says to you, oh, yeah, you need to do this thing because great.
And you're like, okay, where's any actual methodology or stats behind that and it's it's never work in theory is is I forget the name of the person behind it but basically trying to come up with the supporting research or say no, these things don't work.
And it's interesting because there are a few things that do come out of looking at looking behind the numbers right and one of the interesting things about extreme programming, the argument is always about bug defects right is the you should pair
program and collaborate more closely on tranches of work, because you find more defects, and your time to live is lower.
And there is actually some evidence for that. And it's but it's largely based on the fact that pool of pool requests and code reviews don't catch any bugs.
They literally don't they like they catch a very, very like you said, superficial layer of bugs like, but more often they catch things like formatting.
They do catch design problems, you know, like architectural problems or conceptual problems. But largely speaking, they don't catch critical security vulnerabilities, those things take a lot of time to get into your head to understand the code, then the implications, then the potential
bugs, which you tend to only appreciate when you're co located on that piece of work like directly working on it.
So there's a whole bunch of really interesting stuff actually on the the never working theory side of things. One of my favorites actually is the addendum to pair programming is if you make people pair all the time, they're less effective than people that never pair.
If you let people decide when they want to collaborate shock horror.
People come up with better results. But, you know, the interesting thing I think related to maybe the cosmos is thinking about how the actual core software is written as well, which is everybody is distributed, everybody is remote first.
And it's quite often done heavily asynchronously by people in different time zones, working on their own. So we rely very, very heavily on essentially text based code reviews, both at chain level and an SDK level.
And that's also something that we should be worried about. So a question we very commonly get on game of nodes is should all validators know how to code.
And it's like, well, maybe, I don't know, there should be technical people in the room probably. But if the people who are writing the actual software can't find those defects, then it becomes the evidence is from the research that it becomes increasingly unlikely that those bugs are going to be found anyway, right?
Yeah, I'd actually love to talk about that specifically. Because we've, we've had pretty lively discussions internally at at steak fish and I'm sure other validators who after the aftermath of during the aftermath of like prop 20 what happened.
It's an interesting question because I feel like it is firmly within a decentralization ethos of the the responsibilities of of where responsibilities come from.
There's like so many ways you can you can you can look at this let's say, let's say for example, if you're just operating nodes, and you don't have any responsibility, you don't have any governance responsibilities, then you can remain agnostic just as like maybe AWS can be
agnostic to the type of websites that it serves and if there's hacks on those websites, it's not ad of us is fault. Right. But then the moment that you do expose this sort of governance, you inherit, I feel personally, I feel like you inherit responsibility,
because you now have like this, you have the ability to vote, and that voting ability gives you the chance to direct the chain for how it gets developed and what does get passed or not.
And that is on now on our laps. And if that's on our laps, then we are not just like AWS we're active participants in this direction. And also we are not we are explicitly independent from the core foundation teams that started the chain themselves.
Maybe if we're under the umbrella of the core foundation team, that was pushing a lot of this code to then they would be responsible right maybe the person who's pushing out the code but we're supposed to be these like checks and balances, right against these
foundations.
And so that's why I feel like we need to figure something out, maybe it's not maybe as you're saying like, we don't have the capacity to all audit, or check the code maybe there has to be independent third parties that do.
But then right how do they get paid, it costs like 100 plus $1000 sometimes to audit some of these things and like all the auditors are always busy. They're at various quality and so like blame keeps getting shifted sometimes but yeah.
I just I just wanted to chime in with just one point about that then with. So from what I understand that you're saying that because you can make a choice and you can vote, then, you know, inherently we're taking some responsibility for the direction of which way the
chain goes right. But I would put forward an argument that for example, you know, with Ethereum, whilst they don't have like an on chain governance, you're still voting by picking up the new software and running it.
Because that's essentially a vote for that new client. Yeah, yeah. So if you don't run it then you don't agree with it if you do run it you do agree with it.
So I mean, I feel like you're still voting in some fashion, whether it be like, you know, it's a bit different because on on a POS chain with governance, you have obviously other holders that can vote not just the people who run the nodes.
So, yeah, I guess, so you two doesn't really agree with me.
I wonder if, yeah, like it's interesting. Ethereum is an interesting case because I feel like the Ethereum Foundation has done a pretty decent job of like distancing itself from, let's say, even like client development, right, we have like many different clients that are pushing out the
software and we can obviously make a choice, whether to use like the Prism client Lighthouse, Techu, Nimbus, and then they themselves, if there's for example, a bug in their client implementation, then maybe right.
It's not the foundation. It's the client team and then the client team has to make sure they're doing a good job. And then when we as Ethereum validators choose to run that code, should we also be doing like a double check or as a participant in the network?
There's two interesting things I'd like to pick up on here. So we've had a comment saying that that's a terrible argument to Null's point previously from Psy2 Studio, who says, what's the alternative to not running the update? Is it to exit?
That's not really a vote.
But is it not a vote? It's like you're voting, it's like the power of exit is key.
Well, exactly. The power of exit is assumed in a lot of voting systems as part of the checks and balances of the system. And we saw this with Prop 16 and Cosmos itself has a similar mechanism, which is bonded tokens to a validator.
The validator can shut down the chain. We saw that on Terra very recently. And in addition, I suppose there's also this low level concept or not low level, but the governance concept of No With Vito within Cosmos, which assumes that if you...
The general assumption is for No With Vito that if you do a No With Vito that you will exit the chain should that proposal pass. Whether or not anybody actually would completely say, no, I'm done, I'm leaving.
I don't know. But that's what it's kind of convention has it that that's the case. So No With Vito is sort of also not only the fact it has obviously a different threshold for sort of turning over a vote.
It also has this sort of signalling property, which is that if you're getting to around 30% of people who are saying no with Vito, there's also a danger of a chain halt.
Besides, if you get to around 30%, you're into chain halt territory regardless of the actual upgrade itself, purely because not every validator comes back online straight away.
I mean, this is something that we track in the Juno Core team very closely. We'd like to know ahead of time who's ready, how quickly we can expect to get back up and that sort of thing.
And Steakfisher very, very good at responding when we do ping people and ask whether they're ready. But yeah, it's one of those things where there's actually two mechanisms on the surface look the same, but they're actually quite different.
So just before we go too far down the rabbit hole, I just wanted to submit a further argument to Sawie too, in that I see what you're saying, but if enough people vote no by not upgrading, then no, they don't have to run the update and exit or exit.
So you still initially, like when they do the upgrade, you have to have a certain percentage of the network do the upgrade, or you end up with a hard fork.
So, you know, if they enough validators refuse to update and others do update, you end up with a fork, right? So it's a little bit, I would still say you're still voting with your willingness to upgrade.
So sorry, I just wanted to go back to is still talking about it in the chat there. But anyway, I'd be interested what you guys think.
Right. In these cases, let's say, in most cases, the these decisions that validators make when making critical upgrades or infrastructure level governance decisions.
There has generally been like, it's been good, right? Like nothing crazy has happened where everything like burned down to the ground. But I think the Juno case for prop 20 was, although it was not like a crazy technical problem.
It was a mistake and no, none of us caught it. This is kind of like it was, I felt like it was like a slap in the face like in giving us a wake up call as as validators to figure out like what do we do in these situations, or everything has been going good until this
happened. And so we clearly miss that edge case, maybe like the protocol, the POS systems designers, or maybe validators as organizations communicating with each other and how to properly manage these chains.
What do you guys see as, yeah, like what do you guys, what are your guys's opinions on that? Like, because there's an extreme, right? One extreme is every single update that we push into a live production environment should be vetted by us as a validator.
The other end of the extreme is a complete agnosticism. And then maybe somewhere in the middle is like, you, we have separate entities, like third party entities that are going to do audits for us is the foundation's responsibility to pay for those audits or something like that.
Because remember, like, that like the whole like, right, if you're familiar with like Ethereum, like the roll up ecosystem or like, like fraud proofs or even like in Polkadot when they used to have fishermen, right, doing these these checks.
What do you guys see on the spectrum of like how validators should be behaving, maybe it's chain by chain, different.
But so I think, I think overall, like the behavior of the devs, the foundation, the community, and the validators all lend itself to the success of the chain as a whole, right.
So I think like, it's no one's particular job, but it is like the community as a whole should have checks and balances and it's in their interest to do those as time goes by.
Like, I think from a validated perspective, you don't necessarily need to be a coder, but perhaps, you know, you should have a policy internally to at least look at the diff of an upgrade and see if there's anything that jumps out to you or at least try to learn through that process.
I mean, it's not wholly down to validators for the upgrades, either, because I mean, security upgrades are a bit of a different story where we jump on that straightaway and do it out of outside of governance, but for major upgrades that go through governance, you've got everyone, all the participants in the network
except usually foundations, I think they try to steer clear of, you know, influencing votes. So, and you have to imagine being crypto that there are a lot of people in the community or coders as well.
And for the most part, the code bases are open source and they can go and look at a diff and they can look at the code bases as a whole.
So I think it's the community as delegators responsibility as well as validators to do checks and balances on the devs who are contributing to the networks.
But there's there's two other interesting ways of looking at this as well. I mean, so number one, like I am on record. Number one, I'm being gone rust rather than go. Sorry.
And also, as somebody who's worked with very, very large complex code bases before like the SDK is complicated, it is hard to understand what is going on. There's a lot of boilerplate. There's a lot of misdirection.
That is the way it is, unfortunately.
But back to what Dan was saying earlier, which I think is an interesting point is that for a lot of this is a kind of thing that we see, because of the the world we're in, which is app chains, right, you should be able to build an app chain relatively easily.
So here's a toolkit for doing it. Whereas on larger monolithic chains, there are multiple clients, right, and they actually are potentially moving quite differently, right.
And you can even fork and rebuild them and whatever. And there's a whole bunch of different ways of interacting. And there are actually models for different types of blockchains where your validator set is randomized, say, and as long as the
as long as the integrity of the packets matches the right software version, then they can be proposed, right. So it's also kind of the not only where we are with the technology, but it's also the way the cosmos ecosystem works a little bit that we're
we're talking about this one monolithic piece of software, which is sort of run in this really particular way, because of how it the consensus model works, you know, if there were like a situation where there were multiple clients or multiple SDKs or, you know, like,
okay, this is very hard to imagine. But if there was a position where you could have where it was common to have and this would not make any sense really because why would you do it in two code bases but in large companies, sometimes you have two parts for projects
and two different code programming languages, like what if one part of you know you had a Juno Rust implementation and a Juno Go implementation and some running one summer running the other, you know, we're just always assuming that the the the whole thing is like each
is a different app chain is like this monolithic stack, which is an architectural design choice, 10 to minute itself is is modular. So, you know, that's not particularly helpful thoughts throughout there but I've just sort of said it just I think what Dan was pointing out about clients
is quite an interesting one to think about for the future you know. I also think we go through a couple different types of upgrade types I mean we have functionality based upgrades like introduction of wasm other types of things which usually run through some sort of testnet that could be incentivized to be able to find ways
to do like bounties on defects or the types of things because it's a much slower pace and we're delivering functionality or upgrades over a period of time. There's we've had a lot of wasm kind of last minute upgrades that those binaries or even that code hasn't been released to maybe an hour before scheduled upgrade.
So, there really is no team no time for that because we're trying to avoid a chain halt or some sort of you know bad actor who's going to take a look at that and try to do something associated to a chain so that that really has no time for that review that's that's not something that's going to happen.
So, I think it's quite rare is this June upgrade which was very governance focused which frankly we had all the time in the world for sort of right there's a little bit of a clock ticking related to this idea of maybe a possible unbound bonding but that never really happened so that one that one I think there was no incentive around that I don't think there was
an incentive around that review so maybe I think we also look at this even from a governance perspective we've talked about in the past I think on Game of Noses around either there's one governance timeline and it's always a you know X day vote and then that same timeline gets applied to any of those three situations.
Well those three situations might have much different timelines and when a governance vote should occur maybe some of those should be longer some of those should be shorter and so hoping that governance module also has the ability to expand to be able to deal with those types of things separately.
So I think taking my point is trying to take one cookie cutter approach to each type of these issues doesn't necessarily work.
Yeah so would I guess yeah maybe that as you guys I mentioned there's security upgrades there's community related like upgrades and then let's say like referendum and and so if we were to figure something out let's say with with Juno.
Knowing that we had had missed things should and and from a perspective of we don't want to kind of operate in a.
Without we don't want to operate without robustness.
Given that we know that there are potential edge cases me.
It's kind of like isn't there's like this this this psychological experiment or like this phenomenon called like the prisoners dilemma where if everyone is expecting to act a certain way.
It just it just requires one person not to act that way and then everything falls apart.
If if we're operating on this assumption again that we expect all 125 validators at least one of them to be checking the code and then voluntarily saying hey I caught this like typo or something or or problem.
But then right the bystander effect kicks in and then none of us do that is not very robust and probably like it's not let's say upholding some of these sort of fiduciary duties we have to like the people who stake with us if we're responsible for running this chain and then also like.
Supporting that I feel like maybe they're yeah there has to be something more robust than hoping that other people are watching it as you said right like.
Well, I could I could say something for the future that could be.
You know something gets created out of to make it more robust so we were talking I think last week potentially the week before they all seem to match together about getopia.
So there could be some mashup of getopia in cross train cross chain transactions that might be able to result feeding into governance about checks and balances on the code because you could you know prove it with a signature.
So you know just thoughts while you're talking maybe maybe someone's working on that.
Yeah, that would be awesome if you have more details on that that'd be pretty cool I just think like.
There's we have to probably also understand even for large validator service providers, like we have to understand like there are limitations that we have from resources, both capital and human.
To try to do the best that we can to make sure that like no flaws are going out into the chain to live production of environments.
Because like if I had my like if if in an ideal world for me personally speaking my own opinion not reflective of steak fish but in my personal opinion it would be wonderful if every single validator was validating.
Everything that it was pushing out into production environment but that's idealistic it's not practical.
And maybe there are some solutions that we can kind of come up with.
We as let's say validator ecosystem can also pressure collectively foundations if they have all this money right to be like, you know, this is your chain that you guys have designed we're running it for you but we're also cognizant of the fact that the foundations are
supposedly meant to decentralize and it's all we're going to be in charge now of sort of running these things or other decentralized independent entities.
Where yeah like maybe there has to be an auditor operator or something existing or something that are contractors that we have money that we're given from the community.
Just practically speaking with.
I don't know I think it might just be that the whole I keep I keep saying this like I need to stop it.
I stop being negative so I want to preface whenever I'm critical of proof of steak.
I just I realize I've just kind of made incoherent noises for about 20 seconds there without actually beginning to say words.
It usually happens before you talk about bashing.
A friend of mine used to describe it as Hugh Granting, which although I'm not anything like that, it's a kind of trope in TV and film isn't it British people just go except we actually do do it before we talk.
So, but yeah, there's the other saying whatever it is like closest friends harshest critics that's that kind of thing.
I think we prove a stake that's like kind of where where I'm at with some of this stuff but like that.
It strikes me that probably there's there's there's a whole bunch of different ways of approaching this that maybe we haven't considered which is that it's much easier to establish integrity of data than it is to establish like integrity of data integrity of contract and face by which I mean like.
What data is flowing into and out of the system I mean we do this with packets with this ICS all this kind of stuff is very very well understood.
We talk about protocols all the time, but in reality we don't actually, well we do we do build and work on protocols at a low level but that's not necessarily the way we think about the software we're rolling out right.
We do that's what an app hash error is at the end of the day.
I think a lot of the way because we're kind of tied into this very low level world when working on blockchains. There's a whole bunch of stuff at a higher level, which has happened in software development last 20 years with you know contracts contracts run time schemas all that kind of stuff and you
literally see it in cosmizing contracts. You literally see it in you know swag is generated when you run an app chain you know those are code gen tools obviously but they're telling you what the contract interfaces for a bunch of stuff.
And there is a question I suppose for me a longer term whether actually it's only really interface between things like that needs defining because you can guarantee an API the rest of it.
Not so much right and the actual integrity of the software the intention of it isn't as important as what it actually does.
Right and and and and okay those those people who are listening along and a software developers you've gone yes okay very well you've just described the difference between state and data.
State of run time very good. They're still the whole thing problem and you can't just wave your hand about that and yeah okay it is literally the most intractable problem in computer science is what you eventually reduce this down to.
But I do wonder if we're actually thinking about the problem in long term the best the best way right. It comes down to the ergonomics of actually establishing what that what it is it's supposed to do.
Not necessarily what the what the code actually says because that is try you know get a go developer who's a good developer give him a bunch of the SDK code and say what is this do.
I think it's like yeah I think there's like maybe a few maybe we can draw from like some real world examples to like.
What what you're describing with like let's say the the intended use of of the code and what it actually does right and perhaps in a decentralized setting maybe open source makes a lot of sense as well.
Like we've seen also sort of that was let's say inherited responsibility come from the particular clients that are served.
So for example like we have like red hat Linux which draws from like an open source Debian project but then they are they are like they license their software to people who and then red hat will be responsible for any defects that happen from from a an institutional client that that uses their software.
Not not necessarily not necessarily Debian perhaps.
Maybe something like that and and you know I saw like soy to saying that maybe there's like use the stick instead of a carrot.
Right because we are definitely fed the carrot in POS were rewarded for behaving properly and there's like a very small set of of let's say rules that we have to uphold like you know that that are ingrained into the protocol but maybe there is an opportunity for the stick to also exist.
And if a stick exists where if we push out bad code we get punished. Then if we know that we're going to get punished we're definitely not going to want to push out bad code which then requires us to do all this auditing ourselves.
Or maybe just like a follow the leader kind of thing where someone a larger validator maybe is more well resourced pushes it out first.
And then everyone's like OK like they probably had the resources to do it or maybe there's like a signal or flag that comes up. But it but it's interesting like because there's it's almost like we pushed out all all 125 of us validators on Juneau we push that bad code out.
None of us took any like maybe I did a bit because of my comments but like it like nothing happened like we pushed out.
We screwed up and then and then like like like all of us right like pushed out that bad code and then nothing happened with no consequence.
Yeah.
And so right I took quite a bit of stick. But yeah the junior the junior court team took a lot of stick but then rightly so.
Yeah. So so I think some of these suggestions are cool like maybe Commission hit maybe not allowed to like deduct Commission for a week.
What happens if all 125 of us are now 135 of us are not going to commission for a week. Maybe that goes into like an audit fund or like you know we go into time.
Yeah. Right. Right. No no Commission. Yeah. Yeah. Maybe we need both care like it's it's interesting because some of the there have been quite a few validators with talking about like governance and right.
How do we self regulate right there's like parallels we can draw to like the financial ecosystem like FINRA being like this sort of self regulatory organization that is made up of like industry not necessarily of the government but it complements like the SEC for example.
And then I think FINRA also does even have fines or something like that.
Like like confined people and you have to have like charters or maybe certifications and stuff. But yeah like we got to figure this out.
And I was even maybe this is a very extreme case where if someone was malicious enough or wanted to teach us a lesson and we were all still going about our our validation of Juno as per the status quo of like we're expecting someone else to catch it a bug.
If let's say the core team wanted to teach the validators a lesson and they pushed out intentionally bad code that let's say jailed a random validator.
And none of us caught it. Then it's like right we've been fooled once we've been fooled twice like it's our fault then like because like if that is in our responsibility like you can you can intentionally like.
Like maybe we maybe we will never listen learn our lesson and we're just going to keep pushing out code blindly into these live production environments and maybe like the people like the stakers will be be hurt.
Maybe validators will be hurt. Maybe our reputation will be hurt as a chain.
But yeah like that's what I was thinking like we can be taught a lesson if we don't change or decide to do something differently from now on after what happened on prop 20.
Yeah.
So there's an interesting question right which is we're using these tools to build economic systems right so like Dan says you can build a system where you disincentivize behavior you see is undesirable.
Which is I mean something that we've been experimenting with like as in there are now ways of building chains with.
Custom end blockers and all that kind of stuff in and I know I've been banging on about fungible validator sets for a while something I've been tinkering away with my spare time.
And there is a whole bunch of incentive things you can do as soon as you assume that however unlikely it is that a validator actually finds themselves being pushed out because of their own.
Behavior in action that it can happen.
And that delegations have nothing to do with it that that there is there is some kind of writing mechanism in the set that is doing.
Activities on some deterministic order so there's that way of going about things which is.
Cool and interesting and I think is pretty neat and there's the other way that I was kind of hinting at before by saying like you know we think about the wrong way and it's actually a case of data validation right which is that.
Is the system itself perhaps should be more robust to the fact we're in trustless environment right so.
We're terrified of the idea of a hard fork right and other chains other technologies just go which chains longer.
And then off you go you resolve and you move forward so it is peculiar to proof of stake.
This need for some element of trust and then that comes back to like well you know who the validators are.
And that comes back to other validators fungible.
As soon as you stop assuming that the validators are not fungible or at least a very very slowly fungible.
Then you have to completely rethink your model in terms of what is the thing like then you have got probably a higher likelihood of Byzantine actors probably a higher likelihood of nodes being down probably a higher likelihood of many other.
Situations that wouldn't take place on a current generation proof of state chain.
And then it becomes a case of okay well what what is the barest minimum guarantee I need to make a block.
And that is essentially that the wire data is validated in a certain format.
Sorry that wasn't meant to come straight up on screen.
I think you described such an interesting concept that perhaps gives rise this idea of like how much agency validators have where we can we can operate within the the rules of the system that had been defined defined from the beginning.
Like the system exists as of now as you're saying like you know extraneous to a validator staying within the set because it has a lot of delegations or not.
And so I think that within the rules that we have defined like we can we can operate a certain way versus also changing the system in a way that is more in line is more practical with what we've been seeing because because I would even say that right.
And I would not to say that the original creation or the setup of how POS chains is not like let's say like cannon or sacred.
We can change it like we as validators and and with our delegates literally have the ability to change the the way these chains work.
And so we have the power of governance and that that is a feature of it right and and we've we've clearly seen some of the assumptions that some of these POS chains have had been created with the assumptions fall false.
Like if the assumptions of let's say some security guarantees of these chains are false then we should probably fix it or or or work together right obviously closely with the community to make it better.
And what I mean by some of the assumptions is that like if if the if you have a large amount of stake on a chain you should probably not like the assumption was that if you have a large amount of stake on a chain and non trivial amount.
You are less inclined to act maliciously against it but we clearly saw that not happening.
So that went out the window.
And so like then it's like how do we how do we encourage some of these security guarantees.
If that went out the window then we have to rethink certain things and we have the agency to we as validators have the agency with our delegates to to fix those things to make it right.
Yeah I mean I guess this is also the problem back to that we're just trying to create economic systems here.
I think also a lot of the tendency with the level of maturity we're at now is to still be thinking in very simplistic terms like oh well it's in somebody's self interest to do this so why would they have a not do it.
And it's like well I mean look at 150 years of economic history 200 300 400 years it's full of academic studies and reports on people doing stuff that is wildly not in their interest.
And that's that's why it's so complex to actually say well what is the correct balancing set of things that you can do.
And the answer might be it's not actually it's not it's not intrinsically possible or it's not it's it's like in completely intractable to actually make a true trust of the system and it is just social consensus.
It's an existential crisis.
I mean I think it genuinely I because because the thing that's exciting about crypto right is that if you believe the Internet is a real place.
The real place needs governance and currency and it needs these things in a in a trustless highly adversarial highly hostile environment where you need to encourage people to collaborate because you were talking about the prisoners
like the fundamental truth of the prisoners dilemma is that in any situation that resembles the prisoners dilemma collaboration despite looking like the thing that is not a good idea is actually the optimal strategy and if it's a repeated prisoners
dilemma the dominant strategy is to always be optimistic even if you get screwed the first time because the second dominant strategy is to follow what the other guy does and you see that play out in real life.
So we look at those we look we have we have a lot of knowledge about how the system sort of can play out but we still haven't had the time to actually like.
Work out whether we are actually incentivizing the right behavior in the large and you know back to them the validates it's such a small set that the big scale mathematically this should happen.
Definitely doesn't happen because it's only 125 people.
It's incredibly irrational it's too small a set to be even statistically useful is it if you were doing research and evaluates that.
Yeah yeah there's so many important things that we can draw from is like like the financial system has evolved alongside the legal system for hundreds of years and we've like tried to create a financial system without like a.
Like a regulatory system like we we at least at Statefish we we we reached out to quite a lot of few law law schools and we like said hey this is what's going on with some of these chains governance like we issued out grants for them to like.
Like I'm not like a legal scholar but then we as validators here like we're in charge of kind of helping some of these foundations evolve and create these robust systems and perhaps like sometimes it feels like we're recreating these wheels.
Like if validators all have to get into a common forum to communicate and we're sort of like delegated this power to vote like we're literally Congress in some sense of the word and like there's checks and balances that exist for that.
Maybe there's very fundamental differences that we have and with crypto with like immediate transparency and auditability but it's yeah we there's there's there's a lot of I feel like from the more radical beginnings that I've had when I entered into the cryptocurrency ecosystem in the first place.
Finding that maybe like it's not me being jaded but there's a lot of things that we can recognize that that we can use from from let's say the old old world I remember someone mentioned like like the ETC and.
The Dow hack and the ETC and ETH fork like back then I was a very strong like ETC maximalist like I was all about code is law and I was and obviously I had to give up that because.
Ethereum was was the fork that went out but then and then I was had like Trump like this PTSD flashbacks when we were seeing with Juno prop 16 happened where we're kind of like confiscating someone's money where we're self sovereignty in that perspective point or or.
But then what we what what the the ETC and ETH fork taught us or at least taught me was that it was maybe code is not law.
To some extent but like community was was really like the governing factor because without even though you are right let's say we have like these blockchain principles or.
Of decentralization self sovereignty censorship resistance immutability all that even though we like we forced those principles and hard coded them into like how we decide to.
Govern or vote on on chain governance if no one wants that.
And no one uses that chain and that chain is dead regardless of how like strongly you believe in or how true those principles are.
Like if 99 out of 100 people in a village vote that the sky is red.
When when the sky is blue then like that's that's what's going to happen and I did see that with Juno like I and that's I respect it because.
It's it's what people like we we have no power beyond the delegators that give us our power so.
It's there's a lot of a lot of like sleepless nights that I was like really questioning like what was what was happening was going on that maybe.
We can't be so like and anti like existing structures that we think are maybe.
Antithetical to what we believe crypto should be maybe there's a lot we can draw from it but.
Yeah I mean the most obvious is the like an explicit social contract right because you pay your taxes because the government will do will give you defense.
In the case of our country that Universal Health Care not dental sadly impossible to fucking get a dentist in the UK.
It's the one bit you have to pay for basically it's the most complex mess and do you get insurance or do you pay for yourself and it's just like wow OK this is literally what everybody else has to deal with for everything.
But like you know the social contract is like an important thing like it's the basis of it's the basis of the trade off for like oh you know.
If I'm called up should I go to fight in all these kind of things where it is both you know it has a route all the way back to sort of like antiquity so then democracy and antiquity.
But you know it had this practical reimagining in the 19th century you know around nation states and you know what what does it mean in the sense of mass participation.
And we still really don't have a clear idea of what so back to the thing of like the exciting thing about crypto is that we're making a jurisdiction right we're saying the Internet is real it is a jurisdiction it is jurisdiction zero it is just this place cool.
What's the social contract for that place.
Like in the absence of like this is a very like slowly over thousands of years of history it's very it's very dependent on the place it form like the type of the type of government that you have in different places around the world the type of think the way thought around how government systems works and everything.
Is inherently tied to place and if the Internet is a place then. You know we also can't it doesn't just come out of nowhere it comes out of this huge discourse we haven't yet found.
A lot of the forms that are going to be relevant for I feel like they're still we have all we have is jurisdiction zero and we don't know what to do with it yet.
But like you said Dan the one thing that we have learned in the last few years and we can definitely put a pin on is that the only thing we're definitely accountable to is each other.
But we might not be accountable to much else like and if that's the case then that's kind of like okay well should be we be pushing things back into more of a trustless space.
Because it seems to me the more trustless the space is the more you are just accountable to one another or should we be going running the other direction and saying oh no we need to actually build civic structures.
But you know in these virtual spaces in jurisdiction zero.
I wonder maybe in the last few minutes like is it like because like I there's this this concept of like in good behaviors incentivized versus like good behavior cannot be relying on altruism.
But then as you're saying like in the prisoners dilemma example like you have to like the desire behavior by folks there is that you are sort of coordinating and cooperating because that's kind of that is somewhat altruistic right.
But then there's also this concept of if crypto is like a dark forest and we're all like PVP each other right because we're all sort of self sovereign and then no one else is watching out for us then like how do we operate that there.
It's I like idealistically we'd like to have like an altruistic system.
But yeah I don't know what you guys think but curious.
I mean the comments here look yeah you suffer you got anything to add but going back to the social contract pieces and the social contract kind of built on a layer of trust though in those situations like does that does that actually align itself well to this trustless.
Every person for himself herself type structure that well yeah I mean it's between I mean we get it gets really complicated if you want to call it a social contract doesn't it because it's between asymmetric.
It's between asymmetric actors in a situation where you don't have the right of exit or easy right of exit right and a lot of your a lot of your a lot of the I suppose dynamics that play into your ability to exercise your rights within the social contract are fundamentally non fungible non portable.
Like your past like everything from your passport to the land you live on.
And I think that's kind of what you sort of bottom it out at is like not really a social contract but it's not really a social contract.
And I think that's kind of what what you sort of bottom it out at is like not once you look at it from that perspective from a cynical perspective.
But just like you know what is the minimum expression of this idea.
You know take out the asymmetry take out the the the the you know the fixed goods take out all of the other stuff because because a lot of that stuff actually doesn't hold true unless we make it true.
I think Pop most said something in the comments saying well you know like nation states can't just fork the nation and walk off to a new nation with an exact copy of all of the record of the resources and the goods and the land records and and everything.
Like there was a case in Rome I think it was called like successio plebis where like everyone literally just like walked out and then left the left the rulers to fend for themselves.
Okay so probably wrong about that.
So maybe there is a just just that one time you know that.
If I could interject so Dan has the role soon so did we have any burning questions, whether from us or whether from the audience here that we wanted to ask them before he has to shuffle off.
Do we have anything in our in our spreadsheet.
Well we had we had some biographical stuff but I think to be honest stands been far too interesting to just ask some questions about you know how you got into crypto and stuff which I think to be honest we've also we've also gone back as far as etc you know I feel like we we've covered we've covered some of how you got into crypto as well implicitly in that haven't we.
You should hear my you should hear how I got into crypto it's it's quite a wild story.
Well hang on a minute hang on a minute you can't just say that.
I used to work in a law for my rage quit after the 2012 congressional hearing where it just BC got caught money laundering and tears financing.
And it was like my like burned down the banks moment.
So that's how I got into crypto that's why I said extremist tendencies where I was all about like anti government or anti legal system anti bank like complete that's the total basis around crypto right.
Yeah, yeah so when Craig Wright got so angry with the financial system that he created.
I'm not like right.
Unfortunately, but somebody somebody in the comments.
I think it's only I think it's Kevin has asked where did you get that sweater.
Oh, this is a fairy.
Funny story like.
I did a sweater swap with Dylan Francis the DJ and then I emailed Fairty because they were out sold out of the sweater and I told them the story and then I got one from.
I couldn't find it anywhere but it's a nice.
Nice.
It's extreme loyalty to the sweater.
It's very extreme. I think the one other question we've got is not not specifically for Dan but we talked about as a tiny bit in the pre show.
A question we got during the week was about how in practice a proof of state chain in cosmos would actually die and whether it's possible for a chain to die or just a zombie on forever.
This was in the context of Kerberus where there are validators dropping off.
I think the reason that I got this question through is the DM for for game of notes was because obviously needlecast said we're going to pull out of Kerb because it's not worth it.
Focus on things instead.
But I'm kind of interested on on on your thoughts on that.
What does it mean meaningfully for a chain to die can a chain die.
How does it die.
How does that play out.
Is it just the perception that it is dead because as long as it's still making blocks right.
Is that a question for me or.
I mean the whole room but I'm interested in your take on it as well Dan.
Yeah there's like I had some really interesting conversations with a friend of mine who we're talking about we were talking about zombie chains in the beginning before the show started.
Where he described the punishment of let's say bad chains acting as like ebbs and flows.
So let's use for example like XRP where.
Right people who knew XRP there's like a lot of difficulties with with XRP would leave and so it would it would ebb away.
But then the people who are not able to exit in time because we have like sort of these zero sum games would would would then be incentivized to pretend like everything is OK or even show the people who who ebbed away.
Don't have any more incentive because they don't have any XRP bags to kind of to show to show or and so they won't be like complaining or trying to speak reason to the people who are looking at XRP now.
And then right the people who are shilling XRP are going to show to these newbies who don't know any history of of XRP they'll eat it up and then it'll flow they'll flow in they'll go back up and then people realize it's it's a scam or anything it'll flow back down and up and up and up and down and down.
And the people who are originally there to show will find perhaps that they've done such a good job because of whatever cycle it's in we are in these markets that that they'll now they have the opportunity to leave but won't.
And then it just doesn't become about fundamentals is just a bunch of showing and so like you can never sort of like kill a chain up and down.
Maybe with some of these other chains.
You get too big. And then right the bigger you are is sort of the heart heart you fall sort of sort of situation but
maybe it's it's like it has to be a complete catastrophic failure for a chain to fail versus chains that are not necessarily as exposed.
They'll suffer the same zombie ebb and flow sort of situation.
But even in Cosmos you reckon it's more likely there's going to be this again question for everybody like you reckon it's more likely that we zombie chains rather than you know some chains literally gets the point of not making any more blocks.
Yeah, yeah, or validators all exit. I think I think there are some Cosmos chains that we're starting to see just no more development on.
And eventually, Validators will see that like if if let's say a large validators maintaining many many networks and they miss something and they're just continuing like continuing on maybe maybe they'll leave.
So.
So it's interesting like for chains that have no purpose with a token with no custom modules is you know, I guess they could live on if they just IBC'd everything to another chain and then everyone just switched it off right.
The token could live on anyway, the spirit, the spirit of it.
But there's definitely yeah, there's definitely chains that aren't that have no that won't have any economic value like stall development because it's easy to launch a chain right.
And but if you if you launch the chain with the expectation that there's continued development on that chain and then that doesn't happen.
Then what's the point of continuing to run it and you know the investors in that chain of essentially not going to see any value from it. Do you just keep burning the candle?
You just keep throwing the money or you know, is there likely collective agreement to just turn it off.
So there is a chain that I've asked the you know the developers of should we mothball it.
It's in a situation where it can't continue at the moment to further its development because of where it was founded and where its agreements were.
And I asked the question well, should we just keep kicking the can or should we you know put it on ice until the development could be furthered later on down the track.
Anyone have any comments on that?
I mean it's more of a coordinated exit is what you're talking about like having a developer lead exit versus waiting for top validators or the consensus to be lost through people shutting machines down.
Yeah, like for example, what if you put through a government's proposal a text government's proposal and said you know what's the appetite to mothball this thing for a year.
For example, I guess the the hard part of that is restarting it because then you need to get participation of all the original.
Yeah, or yeah, they would probably create a new Genesis off of it or something similar.
Yeah, because we got all the because all the power is to be bonded right.
Yeah, yeah.
There was also a funny comment in the in the column who posts a lot of shit post a lot during game of nodes popped up in the comments and came out with a very DJ and comment, which I love.
Couldn't IBC dead chain tokens become collectible. Yeah, I've got that Gen 1 loon C token.
Anyone anyone want to buy my surplus.
Yeah, right. So and then and then I think Kevin Garrison said well you know it does a token convert to a de facto at FT.
When the chains finally dead. I mean, there were I guess there will be IBC denoms hanging about weren't they like for a considerable period of time.
Yeah, I do think that that they'll have me. I don't know if we've had any that's happened in the cosmos yet. I can't think I mean servers might be the closest.
But then they're trying to bring that back through. I think the governance to bring wasmin or something similar that they're trying to generate trying to you know, turn up some excitement there.
But I think it will be slow and then sudden right like it'll you'll have some validators just coming out and say hey move office those types of things and then that will happen for a large set of validators where there's nowhere left to move or people don't see that or something else and at some point that
thing gets shut off because they just think it's dead. So I do think it's definitely possible that one of these is going to die just based on not being able to reach consensus on the topic of exit.
I think I have to leave now but thank you so much. I really enjoyed this conversation.
Thanks for coming on down.
Thank you for joining us then.
Great to be here.
Thank you.
So something that I was just going to show because we had the question earlier about looking at the diff for for a new release.
So I actually just brought up. This is the Juno Cosmos contracts Juno page where the development lives for Juno.
So I actually just wanted to show if you go to the releases page.
You've got your changelog what's changed in this new version but you can also click on the full changelog here.
And this is an option you can enable when you do the the tag but then you can go down and you can see the the diff between the named tags here but you can also change and you can see what you want to diff between on tags.
Well, if you click that release, I'm pretty sure.
You know, release v6.00 the tag.
Are you going to tell them that the screen is not changing or am I going to tell them that the screen is not changing.
Oh, okay. It's not changing.
Well, there you go.
So that's just an image I think.
Well, that didn't work well.
I was actually doing that on screen.
So I was just like, why is he talking about a figure dot bearded?
That's because you know, it's one of those things where I see magic for the people listening on podcast players.
I love it.
Yeah.
What you just what you just saw was a screen share not updating so that null sounded like a crazy person talking about something that none of us could see.
Can you add it back in there now?
I don't want to get off the screen.
It's up.
It's moving.
All right.
So this is this is for Tag release 6.0.
And here is the full change log.
Right.
And now we can see all of the comments that were put up all of the commits for for this new version.
And then you can go down and you can just see all of the diff between versions to this new version.
But, you know, just if you want to play around and see the difference between various versions, you can you can look at the different tags here.
And you can diff between tags quite easily with this interface.
Realistic question.
What percentage of the 125 validators could look at those diffs like say related to the wasm contract structure, those types of things and actually have a useful comment associated to those?
What do we think?
10% 20 30.
Yeah, maybe five, five, five percent.
Yeah.
So I and I think somebody I forget who mentioned channel there in the in the structure.
I apologize.
But I think there was something really I forget who said this was X on kids.
It's something along the lines of this is a this is a incentive based infrastructure.
So let's build some incentive around it.
And I still going back to that.
I think that's the way to do it.
Right.
There has to be something that is whether that's carrot or stick.
There's got to be there has to be a you know, some sort of kind of incentive built around that.
I think the sense of community in those types of things is positive.
But ultimately, capitalism, right?
So maybe, you know, maybe one thing we can do better as like community leaders as validators and, you know, social, you know, like we all have social accounts for all have a voice in the social realm for the network and we have very
various platforms and we have responsibilities as validators like maybe something we can work on doing better is, you know, education for other validators which we tried it already, but as well as just the community and these types of things like maybe we can put out tutorials on how to check.
Yeah.
Various parts like, you know, because I think I think for the most part, it's assumed knowledge, right?
So we think, well, you're a validator.
So you know how to do this.
Well, that's not necessarily the case all of the time.
And so, you know, the, the base that knows how to do these checks and balances might be, you know, for one network might be the 11 people watching our podcast.
Plus, you know, the three people on the screen and a handful of other people.
So, I mean, I think, and it shouldn't be too hard to do for Juno because at the end of the day, the majority of Juno changes are basically the largely dependencies or integrations like so the way the SDK works, it's very, very dense, but it's module based and most of what Juno very little of Juno is custom module.
There's no Juno custom module, it's just integrations with other stuff and some additional configuration code on top.
Yeah.
So there was, there was custom stuff.
There was custom stuff.
We're actually ripping it out wherever possible because it's just not a lot of it's not necessary.
It's too hard to maintain, right?
Because you have to.
Well, it goes out track.
Upstream.
Yeah, it goes out track.
So we did have a custom fork of WasmD for a period of time and it was just, it was a night, it was a nightmare to keep in track.
And so I was a big, was a big bugbear of mine to get that killed and move back to mainline WasmD, which has been a lot more straightforward in terms of security fixes since we have done that.
So it's made my life a tiny bit easier, which is something.
But because of that, though, I think it means that it would be possible for us to educate validators to, at the very least, like a good bar would be looking into the upstream dependencies that have changed that are cosmos based and establishing whether or not their consensus breaking, let's say.
Yeah.
Because if you can establish that for yourself, you know, that's, that's already a pretty big thing.
And then you can then actually start to dig into the code and see what's changed from those different stuff.
So did you want to quickly touch on what the parameters are for consensus breaking?
So what are the things that will break consensus?
So it's anything that affects the state machine, basically.
But more broadly, I guess, conceptually, it's kind of the thing I was talking about earlier, which is the contract, like the data integrity bit.
Right. If it cannot be agreed upon in the state machine by implication, the validation is not correct.
The the the the commit of the block can't be agreed upon consensus breaking.
So that is that when the the binary goes to compare app hashes, or you know, you send out your app hash to the network, the hash of the state will be a binary that will result in a slightly different
state to everybody else than it is consensus breaking.
Yeah.
So I mean, the easiest way to show something that's consensus breaking is what happened when Juno got cyber attacked, right?
What's the node save the node to state boom 125 different chains.
Everybody else.
Yeah.
So this is back to we were talking about it earlier, like what's the hardest problem in computer science other than naming it's the halting problem, right?
Which is that it is computationally impossible to reason absolutely about what a program will do or what a machine or what a hypothetical imaginary conceptual mathematical machine will do, because there's two things that happen.
There's state and there's well, there's actually several things, but there's there's state and there's also that you can feed a machine to another machine.
So you can obfuscate a machine inside another machine via recursion and functions and anything.
And there's also state which affects runtime state, right?
So in this case, the state is the chain is the database, whatever.
And then the code is actually the thing that's doing the modification.
So if you reduce is all the way back down to like Lambda calculus is a really, really simple computation model.
Yeah, the state is that was a state for Lambda calculus, the state would be the blockchain providing inputs to the functions which are the code running, right?
That's your computational model.
And of course, can't reason about it, mathematically cannot reason about it.
So, yeah.
But this is why implicitly though we have this block and we have this app hash and it's all deterministic so you can make the subsets of your code deterministic otherwise obviously blockchains wouldn't work.
And that's what app hash errors are.
That's what consensus breaking means.
And so we have this inbuilt assumptions.
We have a bunch of inbuilt assumptions about data integrity that are enforced by you app hash if you're, if you, you, you, the app hash and you crash out.
If it's not right.
And that's why you kind of go okay well but are there other models where you're still doing a similar model for consensus.
But you're able to negotiate that error case more gracefully.
Because if you can negotiate that error case more gracefully, you can handle upgrades completely differently.
You can handle the client completely differently and you can actually handle the validator set completely differently and you have to trust each other less.
So there's a whole bunch of interesting stuff there and there are other chains and models and stuff that do very different things which are very interesting.
Holler chain is a personal thing that I think is fascinating that I only started reading about a few months ago and was like wow okay that's an interesting way of looking at it.
And I think consensus models where.