like the falling inside of your room instead of a dark background.
I know it's too bright here. Stupid.
Welcome to Game of Nodes, a weekly podcast on the cosmos from independent validated teams.
Okay, hello and welcome to Game of Nodes, a weekly podcast
by independent validators on the cosmos. This week we have Asaf from Secret with us,
and we're going to ask him probably some really basic questions about what he does
and what Secret do, despite the fact that, wait, two, three of the people on this podcast validate
Secret? Two, I think. Probably two at the moment. You don't, Serp? I do not.
You don't? Okay, that's my mistake then. So apologies.
He wants to be in the Secret game, but he's not there yet.
The next expansion.
Yeah, okay. So apologies in advance for that, Serp. First off, no, we have some,
although I'm guessing actually you don't have the spreadsheet up.
Oh, no, I do not.
We have some follow-ups. We had a lot of questions over the last week, and our new format,
our continuing changing the format every week so that we don't actually have any kind of
regularity dictates that we're now going to do some of the follow-ups for previous weeks right
at the top of the show before doing questions at the end. So we have quite a few follow-ups
related to what happened with Prop 21 on Juneau, but also some questions about what that means for
the centralization of validator sets as well. So I will read through these, and you will answer them
other people who aren't me. So you have to understand.
That's one way to get out of answering the questions.
You have to understand that I am so incredibly fucking tired right now that I'm running on like,
I don't know, something that's not energy.
Vegemite?
Yeah, something for you to feel.
Why are you so tired?
Because everyone has now, everyone's taken to doing upgrades in the middle of the night and
podcasts early in the morning. It's 7am. And for some reason, everyone decided to string
upgrades together last night. So eventually, I just left it up to Cosmovisor.
Don't we all?
Right. So follow-up questions from last week.
So the first one we have is, how could it be that 125 validators didn't thoroughly check the code
properly?
I mean, I can touch on this one a little bit. So there are a handful of us that are on the
testnet and are very committed to running on the testnet.
Did anyone hear the question?
Yeah. Can you hear the answer?
You're not hearing us?
No. Is someone talking?
Yeah.
Can I be? Am I heard right now?
Yeah, I can hear you.
Yeah, you're good.
Okay. Okay. So from my perspective, there's a handful of us that are really committed to running
on the testnet. And we make sure that everything happens correctly on the testnet.
And that's kind of what happened here. We had ran like, basically did a dry run on the testnet to
make sure that everything worked as it should. And admittedly, due to a lack of due diligence, I
suppose, I didn't confirm that may not have the exact same functionality as the testnet. That's
basically how it went awry from my perspective.
Yeah. So I think also just the context in this question, I think, is about what we're talking
about here. So this is on the basis of prop 20, where within the upgrade, the aforementioned
funds were sent to the incorrect address through just a basic, a basic mistake, a pretty complicated
mistake, right? In terms of trying to be able to identify or be able to paste in the hex code
associated to the right contract address or the right wall address or something similar to that,
right? So I think that's what that's going on. And so the question here is the prop 21 was to
be able to go fix that mistake. And so the idea would be how come 125 validators did not review
that code in prop 20 for that error, right? Yeah, that context sounds right to me. Yeah.
Checking.
Frey, you're going to give an answer for this?
Yeah. Well, I think there's also, and then there's also a final, I think prop 21 is unusual in the
sense that there are also developers who had, and we're one of them who had recused themselves
from working on that code for legal and regulatory reasons. And literally, we don't have get commits
against it, haven't reviewed the code. We've helped run test nets and done all the things that were
the kind of the normal stuff that we do for Juno. We've helped run test nets. We helped stage and
write instructions and coordinate the validator set and whatnot. But insofar as actually working on
checking, being responsible for the merging, tagging, and QA on that piece of code,
we left it to other people. And so that didn't mean that there were radically fewer eyes on that
particular piece of code and that particular handler than there would have been otherwise,
but there were fewer. And so things, things unfortunately got missed from, I think,
from the developer end because of, in some cases, just being very tired after the preceding 10 weeks
and also just being a bit fast and loose with the actual merge itself, which we've had a talk
about how that happened. And we've done some, we've done a little bit of, we've done, there's
a whole bunch of root cause analysis that will come out on everything from the chainhole
to Prop 21. There's a lot to cover because it's been an insane period of time. But that we've
already had a discussion about that and what steps we need to take to avoid it in future.
But because it's such an unusual situation, it's also very hard to see how we would
end up in that situation again, effectively, where there were people who were sort of out of
the equation in terms of manpower as well. So that's going to presumably, and kind of quite
fairly piss off some people who are like, well, you should be looking at this code or you, or
nothing, you should either be working on everything or nothing. But, you know, it's easy to say that
until you have potentially regulated breathing down your neck. So that's the kind of important
date to point, I think on that one. I think 21 was pretty unique in that a lot of us didn't
want to be working on it just because of other reasons other than the network. So it was left to
the people who could work on it. But also keep in mind that it's hexing coded. So it's not only
looking at the code. It's, I think it was an assumption that it was the right address. So
like it's not just reading the code, then you have to then extract that and then decode it
and then check it, assuming that you know what that address is supposed to be. Like making sure
that address is correct is like a bit of a task. And you need to know that the address is what the
address is. And maybe I'm not sure if the address was published before. There were instructions,
but they were not well circulated. So that's another thing I think that contributed.
Yeah. So that was probably a falling over in itself that if there was an intention, I mean,
it probably should have been in the proposal to put that address so that people could check.
Yeah. And I think there is actually also one more thing, which is that the hex was used rather
than just the back 32, which is a bit of a weird design decision. But again, like, I'm not going
to throw stones about it because we literally didn't work on that code. We didn't look at that
code. We didn't help review it. And so we can't turn around and say, oh, we wouldn't have designed
it that way because, well, we could have chipped in and helped, but we decided not to for the reasons
mentioned above. So yeah, it is a bit of a tricky one, I think. And there's no one personal group
of people to blame. I think it's a collective failure of, do you know, in a pretty tough situation?
But it does, I think there are a lot of interesting things here around like, you know,
the Valset checking code, right? And this is not just this is kind of why I thought this was a
particularly interesting one, because obviously Stargaze has got some upgrades coming up now,
and they have governance controlled smart contracts, right? So, you know, what's the
betting that every validate there is check the hex on, sorry, there's a hash of those smart contracts.
Yeah. But also, like, it kind of leads into the next question here.
Well, I mean, and we've got a surface. So like, you know, on secret, when you run a when you run
an upgrade, like, what's your expectation there for your validator set, you know, in terms of
checking the code before they run it, they can't check the code.
Well, it's not close.
Isn't it? I thought it's all pre-composed, isn't it?
What is it? Is the I thought the secret code was all just pre-compiled.
Yeah, it's pre-compiled, but the code is open source.
Oh, is it? Okay. Because because all the instructions were just downloading the
binaries, I think I just assumed that for security reasons, it was closed source.
So, I'm wrong, but I believe the reason why we download the binary set of compiling ourselves
is because it's because of SGX. In order to actually compile an SGX binary, it requires
having like a certain enclave permissions or Intel permissions that average validators just
won't have. And so my understanding is secret labs compiles them for us so that not all of us need our
own SGX T effectively. That doesn't have to do with the code itself. Is that correct or so? Is that
at least in spitting distance? Yeah, yeah. And also the code is open source for security reasons.
Yeah, totally. I mean, I just assumed that there must have been some encryption key or
something associated with it that would prevent it from being open sourced. But, you know,
we're still very new. I only entered the set last week, still very new to secret. So, that's good to know.
Yeah, so, where'd it come? Nice. Yeah, thank you.
So, do we want to move on to the next question, which sort of leads into that in that, is it safe
that validators don't have any program experience? I think that should read. Is it safe that some
validators don't have any programming experience? I think a lot of them do, but perhaps there are also
percentage of people who can't read code or maybe aren't proficient in that type or language. So,
yeah, anyone want to grab onto that one or want to listen to me ramble for like 10 minutes?
I could touch on that one. I don't think that having program experience necessarily benefits
you much really at all. What really benefits you is having, you know, Linux experience,
having sysadmin experience, experience with distributed systems. Knowing code only helps
insofar as because you're a software engineer or have software experience, you probably are more
acclimated to Linux and other distributions just by virtue of that. But really, that's the only leg
up that being a software person gives you when running validators. I don't think there's inherently
some magical key that being an engineer or software person gives you in the validator game.
I think in terms of the previous question, though, if you're expected to check the code, you do need,
so I guess there is a sub question within these two questions, which is that do you need enough
technically experience to read the code and to determine what it's going to do to a reasonable
satisfaction in order to be a validator? And is that something that foundations and teams should
be looking for as a qualification for a validator? Yeah, I think it's fair to say that,
you know, as time goes by, you probably, and if when you're recruiting, you should be looking
for someone with those kinds of skills in addition to like DevOps. I think in terms of
actually running a validator, that, you know, being a bash engineer is probably more important than
being able to read the code. But at the same time, there needs to be enough eyes across the
validator set that can read code to make sure that we're not injecting malicious code into the
validator set. So I think is it important for every validator to have coders in their team?
I don't think so. Is it important that there is a decent amount of coders across the VAL set?
I think it's beneficial if you don't fully, you know, because we shouldn't put our entire trust
in the team because A, they make mistakes and B, it's crypto, anyone can be, you know, trying
to rug everybody or anything like that. So they definitely should be people checking it and they
definitely should be coders that can check it. But I don't think everyone should be or needs to be.
So then just becomes what a question of, you know, how big does your VAL set need to be
statistically to make it likely that one team has definitely checked the code or two teams or
three teams or whatever your metric is for a reasonable amount of checking, right? Yeah,
and I think like this is a thing where it can help that people advertise, you know, their
capabilities. And if that's something that they engage in, then people should be supporting them
for that work as well. So with delegations, you know, because people have to realise that
there's teams out there that are just barely hanging onto a validator so that they can make more
money off their other scheme that they're running. That's not the case for all those types of things,
but there's certainly ones out there that is aping to running a validator code
to, you know, coincide with their marketing. So if you think that actually running non-malicious
code is important, then you should support validators who undertake that work on behalf of
the rest of the validator set. Cool. Yeah. Go back to that question. I think we glossed over a
little bit, but on secret, like when you guys are producing, you know, builds or you're producing,
you know, tags for releases, what is your expectation in terms of what is your expectations
from testing or from validation or from quality control from the validator set? Or who do you
look to for those types of roles? Yes. Up until recently, no one really looked at the
actual code from the validator set. I think it's a pretty new movement that validators also
develop the code and test the code themselves. Like in a deep manner, not just like running it
on test methods and stuff. Right. So how to do that right now? Yeah.
Well, that's because a lot of times people are able to just do that and get away with it because
there's been no consequences. So it's interesting to see a time when there have been consequences
because people haven't checked the code. I think when we had Shane and Cistler on here,
we also talked about there was a early Stargaze testnet where they snuck something into an
upgrade handler. And it said, if you've read this code, contact the team. Right. And then it was like
one person out of 50 or something actually looked in the code and was just like, oh,
hang on a minute. What's this? That's pretty good. So yeah. Like you say, maybe that's the thing
that wasn't really expected in the past, but is expected now for the purposes of decentralization.
I don't know. Yeah. So for Cic, for example, we've been on for like two years now.
And at first the variator set was like only DevOps guys and then teams that are building on
secret started to join the variator set to raise money that way. But they were building their
own project on top of secret and not building the protocol itself. So I don't think that they
are looking closely at that. Yeah. So I think that covers all of our follow ups. So we have a
topic of the week as people were probably guessed from our choice of guest. It's secret. And obviously,
first of all, we have a big thank you to Saf for being the one who bottomed out the root cause on
the Juno chainholt. You may remember him from such cosmos incidents as the Juno chainholt
where he came through as a bit of a hero of the whole episode.
But so I'll kick off here because I've noticed the first question on our list is the explain
like I'm five and I do that role pretty well because I am genuinely pretty clueless about the
benefits of secret. So imagine I'm a newcomer to the cosmos ecosystem, be that a user or maybe a
developer. What are the primary benefits of secret as compared to say, well, I mean, before today,
maybe terror or I suppose you know, equally Juno has cosmism, but the use case I think for Juno
and secret is very different. So maybe terror is a fair comparison, I don't know. So explain
like I'm five. What is like the big draw for secret for developers and for users?
Yeah, so secret network is a cosmos based chain with permissionless cosmism.
With the only difference of having private inputs, outputs and state for contracts. So
you can build pretty cool stuff like private dollars with private voting private private
governance. And for example, dollars can store private keys in the contract state, and then sign
transactions on other blockchains or validate validate blocks on other chains, for example.
And you can store, for example, private entropy in contracts and then generate random numbers
on the fly. So it's very good for games. For example, you can generate like random mazes and
in a shuffle like a deck of cards, something that you can do anywhere else, because you just see
the entire deck on chain. Right. So you could do like a generative NFTs or something like that
without needing an Oracle or some other source of randomness. Yeah.
Wait a minute. Storing entropy? How does that work? Can you explain that just a little bit more to me?
Yeah. So for example, let's say that you want a shuffle a deck of cards for poker. Okay.
So you have, let's say two players joined to the table and each player, when they join,
they input a secret number. They can do that because inputs to
to contracts are encrypted. The input parameters, the variables. So the contract receives two secret
numbers from two players. And then it can use that with a pseudo random number generator to
generate random number and for example, to shuffle a deck of cards and then give cards to the players,
for example. Yeah. So yeah, I'm a little confused about where the entropy comes from originally,
like to be stored in the first place. So you say it's secret, but it has to be generated initially,
right? Yeah. So everything that happens on chain must reach consensus. So you can just generate
a random number from like nowhere, I don't know, from the nodes themselves. You have to
to input the entropy from users inside transactions. So it's cool. If for example, two players join in,
each of them input a private like secret number or something like that. Assuming at least one of
them is honest, then you can take both numbers and hash them together. And then none of them
know the output of the hash. Yeah. So the benefit of secret being that neither party can see what's
in there. It's like the black box, which is another cool application of secret, which is kind of on
topic. Is the black box stock cash? Is that what it's called? Yeah, that sounds right. Yeah. That
was a pretty cool project that I used the other day. It was not clear to me initially how the
hell it was operated. But after about half an hour, I asked someone and then it was pretty good.
Yeah. It's nice to, if you need to seed new wallets anonymously or anything, it's like a
good little application for that and entirely wouldn't be possible if secret didn't exist.
I don't, there's no other way that you can anonymize a wallet anywhere else in the Cosmos that I know
of like that. Well, I guess you could do it with Snarks, right? But it's the approach is different
on secrets, right? It's not using ZK Snarks or Circle or something like that. It's using
different. No, but are there Snarks in the Cosmos at the moment? Well, I mean, there is
supposedly a tornado cash clone coming to June at some point, but there's no reason why you can't
compile Circom circuits. And then it's just requires quite a bit of fiddling to actually
wrangle them into a smart contract layer, basically. I won't pretend that I'm fully
up to speed on the details of how that actually works. I just know that you can do it. You can
compile the circuits, there are Rust libraries for doing it. And obviously, then you can
actually interpret what is happening to say that input does actually match output.
Thus, the two did happen. Thus, you can release funds or take some further action or whatever.
But I suppose my question would be like the approach that secret uses for like,
you know, the built-in security of the smart contract layer and where it fits users,
presumably doesn't require lots of extra developer effort. Is that like the,
because obviously ZK Snarks is a lot of overhead if you want to actually work with them, right?
Yeah, so for Secret, it's just like as a developer, as a contract developer, it's just like
a version of Cosmos and just a few tweaks. Also, it's a bit of an outdated version of
Cosmos and because we're running it since like September of 2020.
But as a developer, you don't need to do anything special. You just need to take into
consideration the privacy model of your contract. So what you want to keep private and what you
want to expose via queries and whether you want to implement like an access control mechanism
in your contract and stuff like that. For example, CW20 contract on Secret,
we call it Snip20 contract, is a privacy token because balances are private and when you transfer
a token, the recipient is private too. By default, unless you expose the details we acquire it.
Right. And how is practically speaking, how is that encryption at work arrest kind of done?
Like again, explain like I'm like I'm five, I'm not a crypto expert.
Yeah, so we're using something called SGX, which is a hardware chip inside like every
Intel CPU today. And what it does, it allows you to run the computations on encrypted data
and we require every validator every node on the network to have an updated SGX hardware.
And it means that they are running computations like Cosmos and computations without seeing the
actual data that they are running. But the output is also encrypted, but they reach consensus only
because it's exactly the same for each of them because the computation is deterministic.
So there's two layers to it. There's encryption at effectively the programming layer,
where you're essentially building in developer tooling that says use this API, your inputs will
be encrypted, they'll be decrypted at the output, don't worry about it, just use the API. And then
there's also the hardware, which is ensuring that node operators effectively can't be
inspecting that data as it's in flight or as it's at rest, right?
Yeah. Yeah, so outside of consensus, the users encrypt and decrypt the inputs and outputs.
But obviously we do that for them with the CLI and with the JavaScript and Python libraries.
Right, okay, cool. So yeah, it's a combination of the hardware and like thinking about the
developer, like the holistic developer experience and stuff, that's cool.
Okay, I've seen some of the, again, like from talking to other Cosmos teams or interact with
other people who are using Cosmos, I've seen some of the secret tooling around like
essentially developer tooling and like bootstrapping and whatnot. And it's all pretty slick and
joined up. So I kind of had guessed that you guys had spent a reasonable amount of time sort of
focusing on that side of things and the ergonomics of it. So that kind of does make sense.
So what's it like working on the core team and stuff? Like what's your involvement,
what's your speciality like there? What's the thing that you kind of work on the most?
So we are like a pretty small team. We were four engineers up until recently. And now we're on
board in formal. So we all did everything. And we built like the since like February of 2020,
we started to work with the Cosmos team to bring Cosmos into a state where it production ready.
And pretty early on we discovered that we need to rewrite the entire core of Cosmos to work
inside SGX. So we kept the APIs of Cosmos but
rewrote everything inside. And all along the way, we shared feedback and code with the
Confio team, Dayton and Simon. And then back in I think September of 2020, we upgraded mainnet with
the version of Secret Contracts we called it, the Cosmos inside the SGX.
And I think all of 2021 we worked on like building blocks for contracts. So
the CW20 equivalent, which is the 20 token and then SNP721, which are private entities,
which have public and private metadata. We also built an AMM and bridges to
Ethereum and Binance matching. So yeah, because we were just four people, we worked on everything together.
Nice. And what's your background as an engineer? Like before Secret, what did you do?
You like mainly a kind of low level protocol guy or you're going to tell me you sort of
worked on mobile apps before this or something or did advertising bots or something.
I was in the Israeli military for six years. I was doing cybersecurity work, mostly defending
against the nation state actors and mostly building the infrastructure of the team that
is supposed to research malware. So my toddler is waking up.
So just a second. Do you need to go for a second?
Yeah, maybe. We can wait, mate. We'll talk amongst ourselves.
Thank you.
I remember those days. Well, I guess we should wait before continuing the Secret conversation,
but how about that UST back to 70 cents? We have on the sheet UST, I'm not saying all caps,
but then obviously it's an acronym. So it would be an all caps anyway, but it does have a lot of
exclamation marks after it. Yeah, and it's just scattered around the sheet everywhere. It's like
links to Duma, YouTube videos and just UST and Scribble across this spreadsheet. How did it come
like this? Yeah, well, Luna looks like it's at $1.03 at the moment.
Yeah, it seems to be. I imagine that is how the UST is making a very slow recovery. I guess
they whoever was trying to push the UST and to the dirt ran out of powder.
Yeah, something like that. Or maybe the goal was to just buy some cheap Luna.
I mean, do you buy the conspiracy theory that it's somebody trying to make
money on shorting it or whatever? Do I buy that? No. I think the legs just,
it's had an incredible pump in the last preceding months. I think maybe it just ran out of steam
and then it had a hiccup. Do you remember the iron finance fiasco last year? Oh, yeah, absolutely.
It's outside of the cosmos, so you guys might not be familiar with it.
Yeah, no idea. That's not the cosmos. I've got no clue. Obviously, I don't own or hold any crypto
outside of the cosmos. The idea that I would is completely ridiculous. So basically, I mean,
it's going back a bit now, but I think it did get a mild attack on it and it busted its peg
and lost confidence immediately. But it was an Algo stablecoin and it was pumped by one of those
football guys, right? Or maybe one of those, Schiltz, was it a football guy or was it like
one of those sharp guys or something like that? I don't remember. I just remember it was some
external personality. Yeah, so some dude was pumping it and then it had a hiccup and went to
zero incredibly fast. So it was a mint. I think there was too much lag in the mint algorithm,
which was meant to increase the price of one. I can't remember exactly how it worked, but I think
it's similar to how USD works, but it had a bad oracle and there was too much lag in it.
And if you attacked it quick enough with a flash loan, you could send it to zero, apparently.
It just lost confidence and everyone bailed on it.
Yeah. Well, isn't it more that there is a piece of price action that changes, that initiates the
attack, but isn't the fundamental issue that as soon as an algorithmic stable dpegs enough,
that people just fundamentally lose confidence and that's just like a bank run effectively?
I don't know if that's fair though, because if you look at the charts, almost every stablecoin
has dpegged at some point. Like, Tether went down to like 60 cents at some point, right?
Right. Because there's no algorithm keeping them pegged on Binance and Kraken and Dex's,
they have their mechanism where people can ARB to. But if there's no ARBs or if there's not enough
liquidity in the pools where they're being smashed, they'll go down anyway. So it all
sort of relies on being able to make ARBs back to the original algorithm that actually stabilizes it.
And for all of those places where it's traded to have enough liquidity to be able to sustain a
prolonged attack on it. So I think that even though there was UST as high as it was,
still not deep enough liquidity to defend against something like that. And I think the same thing
could happen with Tether and the USD and the rest of them. It's just that people are of the belief
that they're backed one to one. So they will ARB between pools, not necessarily back to the
algorithmic pool, which actually keeps the peg. So I think there's something to be said for the
belief in a system as well that will keep people. So because people believe that, and it might be,
Tether is backed one to one with a US dollar, then if between Kraken and Binance, there's
a difference, they'll arbitrage it to bring the peg back. But if there's a system which doesn't have
100% confidence like UST seems to have shown, once you break the peg, if there's not enough
arbitrage back to get it back to $1 quickly, people can lose confidence and then just start
bailing out of it. And then you get an avalanche of volume. So you need really deep liquidity to
absorb it and a lot of arbiters. And when you then can't access back to the original pegging
pool, as was the case, because the bridges were clogged, then that causes another problem.
If I can reduce a very complicated situation down to like a literal five-second meme,
do we think the peg is going to be regained? I think so. I think the longer you leave it,
it'll just regain from the algo. Yeah, I think you'll still regain.
I think it will as well. I do question. One of the big points of faith behind it was the
fact that Luna was above UST and market cap rate. I'm starting to wonder if maybe it was a wrong
decision for it to be tied one to one. So if you look at Silk and Shade, which are on secret
networks, so it's a little appropriate, they're looking at actually having a different system for
it's still going to be an algorithmic stablecoin. But the peg isn't going to be just to their
Shade derivative, which would be like Luna. It would be, let's say you can swap secret for
the UST as well. And so if you try in multi-different cryptocurrencies, one dropping doesn't
necessarily undo the system. And so I'm wondering to what extent that's true. The fact that there
were covering the peg right now and there's such a discrepancy between the two.
But if they're all positively correlated with one other, then one dropping does bring them all down.
So you're back in the same, you know? It's true. But a reduction in the pool of money.
Well, central banks do this with credit issuance and they do it with, well, historically, they did
it with gold or dollars or whatever. They actually held the money somewhere because they knew that
there was a fundamental difference between the two types of things. Like one of them is
one of them is essentially, well, it's real. It's real. It can be used as an actual store of value.
So like there's, or I guess to bail out 2008 or to bail out during wartime, long-term government
bonds, long-term government loans are also used. So there's like a variety of different
tools in the toolbox, which are all about, well, increasing liquidity, right?
But not all of those things are positively correlated with one another, right? So it means you have
different tools available to you. So I guess that's the question. I'm kind of wondering whether a
algorithmic stable doesn't yet have all of the tools it would need to actually
grab something out of Finnair that's not positively correlated, right? Because that's also the
problem with Luna here. As soon as you lose confidence in UST, you lose some confidence in
Luna and Luna is trying to dig UST out of the hole, right? Yeah, I think Luna's taken the brunt of it,
for sure. So I mean, Asaf, do you think UST is going to regain its peg?
Yes. Okay, so we're five for five on the call. So there you go. I mean, this is not
investment advice, but five out of five members of the Game of Nones podcast tonight,
I think UST is going to regain its peg given time. Well, now let's throw a little bit of a
hardball in there, right? So Osmosis just put out their update, right? And they're considering
hard forking so that people can pull out their UST and their Luna pools. So there's going to be
a governance program for that now. If suddenly another three or 400 million enters the market,
what happens then? I mean, that's going to be even more difficult. Explain that again,
Chelsea. What are they putting a proposal for? Yes, they put up a proposal to fork Osmosis such
that they all the pools and all the bondings within their UST Osmo, Luna Osmo, and one other
pool become unbonded so people can immediately pull them out and sell them if they want to.
Right now, you're tied into your one day, seven day, four day bonding, and they want to undo that.
So what you're saying is Luna price is going to go down even more?
I'm saying it's a possibility. So it's actually pretty interesting. I think I'm against them
doing that. Like we took the risk in bonding. But basically the idea is if so we're trying to get
all the values in on it and voting if it hits 66%, they're just going to cut it early and hard fork
Osmosis with all the tokens unbonded. Is that something that's come from the team or is that
something that's come from users? Oh, no, that's come from the team. I'll post the, well, I guess
it can't, you're also into the discord and then you guys can post it in YouTube chat so you can
read up about it. Yeah, I mean, I think essentially the damage is done there. All that can really
happen now is that UST can slowly regain its peg. Luna's done in terms of price for the time being.
So, you know, I don't see the point now.
I mean, unless Luna's going to zero, what do you think it's up?
I think like they can probably land in the top 50 on point market cap after regaining the peg.
So not a bad spot to land on, I think.
Do you think Luna will have a substantial recovery or do you think we're going to see it
pretty low for a long time? I think the confidence is pretty busted in
terror at the moment. I don't know. It's like purely speculation at this point, but after looking at
their market code and seeing that they're willing to sacrifice Luna for regaining the peg. So I
think that Luna eventually will bounce back. I don't know if in a week or in a year, but I think
that it has a good chance. So I don't think, you know, the turmoil over the last couple of days
doesn't change the fact that terror have a good product and good marketing. So.
And good developers. I think within the cosmos, they have the most developers.
And I mean, that can't be understated how important that is.
Sorry, I'm just reading comments.
Yeah, just because I think Asaf was saying earlier, right, you guys are four and you've
just added four more developers, right? So to give an idea, like for the listeners,
eight developers would be quite typical for a very small startup, maybe a couple of hundred
thousand dollars annual recurring revenues, something like that that's just received a series A,
or like a very, very first seed round or something like that, a few million dollars,
a couple of million dollars from a few investors to see if the product has legs.
Typically, then you would go from like two to four developers to
six to eight developers, something like that. So eight developers is a small but decent, smart
startup team, right? And secret is quite a bit bigger and more substantial a product than like
an early stage startup. So, you know, obviously, I mean, the staff correct me if I'm wrong, I don't
know if that's because it's hard to hire or if that's because of, you know, it's difficult to
bring new people on and not be unproductive. I don't know what's been the reason that you've
scaled in that way. So it's hard to hire. And I'm not sure if you're familiar with the secret
history. Like it started as an IGMA and they did an ICO back in 2017. They raised like 45 million.
Then like when I joined back in 2019, they were building a layer to privacy solution for Ethereum
and it was not going that well. So they decided to start fresh on Cosmos. And they also had like a
very, I think they had like 10 or 15 developers. And they decided to scale back because it wasn't
working for them. Right. So, okay, that makes a bit more sense in terms of in terms of developer
size. But this, I guess, bringing us back to why it's important that Terra has so many developers
is that most of the other teams in the Cosmos are operating on probably
a similar number of devs to what secret were before the scale up that as staff says, they're just
doing at the moment, right? Which is three or four engineers who are practically doing the
day to day. I think Stargaze is about four. Most of the other teams I can think of are between four
and maybe 10 if they have a lot of people. And given the market cap of a lot of the chains in
Cosmos, I know obviously Cosmos is a lot smaller than some other ecosystems, but those are very
small numbers for team sizes given the scale of these companies. If they were traditional companies,
they would have 100 engineers, you know. Speaking of Terra, are you guys familiar with
the mentor mint? I'm not familiar with it. With what, sir? A mentor mint. No. So, mentor mint is a
special node that Terra developed. It's essentially a node without a tender mint. It sinks state
and they replace the entire tender mint stack with a much faster DB database engine. And
I was talking with one of the engineers a few hours before she hit the fence. And he was saying
that they're serving constantly like 1.4 million requests per year for Terra. Yeah. So,
do you think that other projects should try and pick that up or try and include something like
that into the SDK? Yes. I was talking with him about trying to port this to Secret.
And he said that it will help like a week after they...
Can you tell me the reason that, for example, when you jump onto Mint Scan and go to the Secret page,
there's a delay in loading it? Is that something to do with Secret itself or is that a Mint Scan thing?
Yeah. Okay. I know it's only seems to be on Secret just talking about RP season stuff. So,
this is like a kind of light node that you're talking about with all the tender mint stripped
out, all the consensus stuff stripped out. It's basically just... Does it still check hashes against
the validator set? Yeah. It's still like receiving state and validating the headers, but it doesn't
run transactions in blocks. It just sinks the state and then you can query it really fast.
And I think that he said that they have a cluster of 35 nodes and they serve like 20k queries per second.
Wow. That is decent. Is that... What kind of load balancer did they get into that type of discussion
with you as well? No? Because I know that I think Stargaze is now using... And Serpent might be
able to help out with this. I think they're using the Cloudflare smart load balancer now. Is that
right? Yeah. We use regional setups that are both pruned as well as full nodes across a couple
different regions and then Cloudflare bounces based off of geography and other types of things.
Yeah. So if you have pruned nodes along with full nodes behind a load balancer, how does it
figure out where to send the requests? Is that programmable or... It doesn't. It doesn't.
So it just throws them around until it gets the one that...
Sort of. But I think the goal is not so much around being able to provide archive nodes to
the community. It's really more that... I think with so many validators running prune nodes now
that the chain is obviously wants to make sure that there's plenty of archives. So
there's such a race on the validator side to bring it down and to make sure that we have
a short block times and things like that. There's some concerns that how many archive nodes are out
there. So for Stargaze, I think they have a couple... I think there's like four or five
that the team runs that are out there. Then we take backups and all that kind of stuff too.
Yeah. I think that's sort of getting lost in the soup, the fact that one of the jobs of a
validator is to maintain the history. Yeah. So I know almost everyone's running a prune setup
because otherwise, it's a way to help making sure that you sign all the blocks and that type of
stuff. But definitely in your infrastructure stack, you should be keeping history as well
if at all possible. So I know that we're working on having hosted hardware elsewhere
to make it cheaper for us to be able to keep a lot of state on slower drives and stuff. It
wouldn't be able to keep up with validating, but it'll be good enough to put lower cost drives,
just fill it up with the data forever, sitting in a colo somewhere where it doesn't matter
if it goes offline. Just like cheaper setup on your own hardware just so that people can
sync from it if they need to someday. Yeah. I think it's becoming more important. I think
I think also change is trying to figure out a way to incentivize that as well because there's
so many valuable roles for validators to play from RPC and API nodes as well as archive and
other types of things. And there's not really a clear incentive structure around that, at least
not on chain at least, right? Everything else becomes off chain, which becomes more difficult
to track and everything else. But yeah, it's important, I think. It is important and it's a huge
problem that in proof of stake, it's not incentivized by the chain and somehow natively.
It's a shame that there's not an easy way to make sure that the people who are contributing to the
health of the network get the delegations instead of people who have the best marketing or something
like that. That is a totally different topic. Yeah, yeah. But it ties in that we can all go
to the effort to ensure the health of the chain as is our responsibility. It doesn't necessarily
mean that you get rewarded for it. So it's good to see that Stargaze are working in that direction.
Yeah, for sure. This is also something we talked about in a previous episode. I remember it was
the one with actually with Shane or whether it's a different one, but it's all about incentives.
And I think the way incentives have been designed in Cosmos for a variety of things
were effectively a good first attempt and need to be upgraded. Governance is a prime example
of it. It's the first stab at governance and it doesn't work. Sorry, it's done fine up to now
and it's now at the point where it's creaking and it's breaking because 10x users, everything breaks.
And another thing is the economic incentives around the validator set in
distributive proof stake do not scale for the long-term health of the network, I think.
And I think there's something that we seem to come back to quite often on this.
We seem to come back to that topic in one shape or another quite often. And I think that's also
something that we found very interesting. I saw there was a comment about Akash down
a minute ago. And I think it's also something that came out with, although I was in a bit of
delirium of sleep deprivation thanks to the hospital and whatnot last week, but that came
out with our conversation with Greg is like, there are a bunch of ways that you can start to
devise more interesting incentive mechanisms. And I think that's something that we found very
interesting. We need to devise more interesting incentive mechanisms using the code inherent
in a blockchain and then go to a different chain that has an infrastructure provider and
do things based on those incentives. And there's other ways we can cut this.
But to bring it back to our guest who I realise is actually that way,
is we were just talking about query speedups and we were talking about the speedup stuff that
Ter had been doing. I gather there's been a substantial query speed increase on
secret. Is that something you guys have done just for secret or is that something that can be applied
to any Cosmos chain? I'm sorry, why do I talk about it all the time?
This is actually a short question, so perhaps you can take that one.
Yeah, so the question is about the 500X speed increase for query nodes. I guess what's the root
cause of that? Is that secret specific? Is it just from adding threading? Can it be translated to
Juno and to Terra? What kind of cross-pollination can happen with that?
Actually, I think Juno already had this feature. We back-ported some new APIs from Cosmos in version 1.
The crypto APIs for verifying signatures, like cryptographic signatures.
Because on secret, we're doing a lot of cryptographic operations inside secret contracts for access
control purposes. For example, when you query a SNP20 token for your balance, it's private,
so you have to prove to the contract that you are the owner of the account.
The way that you do that is by signing something off-chain and then sending it to the contract
to verify. Inside the contract, it verifies the signature, which is a very expensive thing to do
inside of Cosmos. What we did is we added an API for the contracts to offload that computation to
the chain itself instead of doing it inside the contract. Right now, it should be much, much faster.
This is something that came about. It can't be translated, is what you're saying, because it is
effectively a reuse of something that already exists and was created previously. Is that right?
Yeah, it already exists with newer versions of Cosmos. But I'm not sure if the use case
translates to June, for example, because you don't have to implement the access control mechanism
to access your data. But for example, to verify blocks inside of contracts,
that is something that can be translated to June. For example, if you would write a contract that
is essentially a permissionless bridge, for example, just like IBC, that needs to verify
blocks on other blockchains, then that contract would need to do a lot of cryptographic
computations, which in Cosmos version one, we already have that. We didn't have that because
we were in with Cosmos in 0.10, which was released back in September of 2020.
Got it. That makes a lot of sense.
Cool. I think we've got one more question for yourself, which is a big picture stuff.
What's the vision for how secrets are going to grow and change and how the use of it
also might evolve over that time? Where do you see the network going on a five-year horizon,
like if you guys even plan that far ahead? Yeah. Hopefully by then, we will not have to rely on
secrets or any other hardware solution because it's very expensive to run an order right now
compared to other chains. It's a vendor locking, which makes us dependent on Intel right now.
I guess with secrets, there are a lot of use cases that you can't do anywhere else on blockchain.
For example, build games with incomplete information,
or build DAOs with private voting, which I think is very important.
Hopefully, we will see a lot of novel blockchain applications built on secrets,
but half of it to imagine right now what these use cases will be.
What are you most excited for that's coming up on the secret roadmap?
In terms of apps or infrastructure?
I'm excited about bringing the newer version of Cozenwaz into secret,
like having IBC contracts. Then I would want to see apps built on secrets.
I think games have a lot of potential on secrets because of the privacy aspect.
I have a private voting which I already mentioned a couple of times.
Hopefully, we will see a bunch of them this year, maybe in the next six months.
Private voting, something you're working on, is it?
It's something that already exists, like the CIFI DAO, they have a private voting, I think,
for over modern...
So, you plan to implement that on secret, so the validators or anyone can vote
on governance proposals with anonymity?
I think that we are also discussing, but maybe further ahead.
Because I think that's... I'm probably going to crucify myself here.
I think that's important because a validator doesn't necessarily...
What's best for the network? Is it necessarily always the popular vote?
I completely agree here.
Yeah. When you couple that with public forums like Twitter,
people's businesses just get ravaged when they...
A lot of times, you can't make the right decision. You're damned if you're doing,
you're damned if you don't. I think a lot of countries have anonymous voting,
like Australia does, so you can't be discriminated against for your vote preference.
Right.
And I think in terms of proof of stake, a validator's preferences, like I already said,
aren't always aligned with the community because community tends to be more emotionally driven
in a lot of instances, whereas a validator has, in a lot of cases, a set of values that they apply
to blockchain or whether they're blockchain purists or just want to do what's best for the
blockchain doesn't always align. And I think to be able to have the ability to
vote anonymously is important, actually.
I also think even during the vote, away from even that, even to see somewhat results posted
as the vote is going, I think completely sues people in terms of how they're going to vote.
So getting a proposal that's open, and if you have a huge amount of people that vote one way,
to turn that tide, I think is really difficult unless you have obviously validators who are
looking at maybe a little bit deeper than maybe the community. But I think from a community
perspective, if they see 97% of voting positive, even if it's only 2% of quorum, guess what? A
lot of people look at, that's what I should be doing, right? So that's also what I think is a
real benefit here. Even if post-vote, those results are public, I think that still gives
it some options in there. Yep, absolutely. So I mean, most democracies for a very,
very long time have had private ballots. And that's been traditionally, I think,
to avoid the coercion of the upper class or your employer saying, tapping you on the shoulder and
saying, this is how you should vote. And that's before you even get into the inherent problems
of the tyranny of the majority and whatnot, which I think you just identified, you said,
but which is that if you see that a certain number of people have voted a certain way,
it makes it more difficult to form some kind of an objective view. I mean, none of us vote
objectively, right? That's a complete fallacy anyway.
But people, it's weird that even in situations like that, you don't want to be in the losing side,
which like even for votes that don't directly affect you or they get else,
like people don't want to vote the loser. So it's that type of thing, I think, is a huge thing.
Well, so at a case in point, we got asked a couple of times, what we think of Prop 22, right?
And that's an example of one where it is overwhelmingly positive.
But what do we think of that prop and how we vote? Because I mean, from our point of
view, we think it's probably not economically viable with the market downturn we're now seeing
for those final 10 validators to actually break even. But it's like 90 something percent positive.
And it's like, well, do I want to die on a hill of validator expansion? No, I've done
on a hill the last 10 weeks over the CCN drama. I'm done with that, you know?
Well, and those that do have the dissenting opinion during a vote, they are expected to put
out a message on why they have a dissenting opinion. And so it becomes you have to put so much more
energy in to not go with the crowd. Look at Cosmos had Prop 69 with adding Cosmos into the hub.
I had to flood a lot of questions and put out a statement about why I voted yes on it,
because everyone else is voting no. And so that dissenting opinion, it shouldn't matter until
the die is cast. Right. Yeah. And I think the overall fallacy of increasing validator sets
equals decentralization, I think is a different topic that we should talk about also.
I think going from 125 to 135 does absolutely f all for decentralization, but that's a different
topic. We're going from 125 to 225 does the same thing. Exactly. It's literally the same amount
of decentralization. So speaking of decentralization, I saw I know Secret keeps the validator set
pretty small. Like I think you guys just went to 80, right? It was at 70 or it started at 50.
I don't know the history here, but why do you guys, why do you guys kept that tight? I don't know.
Like I'm not involved in this type. Maybe shorty, we'll know more about this.
Yeah, I can, I can throw a lot of heat in that direction. I was probably the cardinal dissenting
opinion about expanding the set recently. So the cardinal reason was when the network is under
load, you start to see a lot of like pairing off of missed blocks. And the reason why I
dissented from increasing was because we don't know the root cause yet. There are, there was a
bunch of work that chill validation and I did kind of doing a B testing on for what all the
different things, peering, different hardware types, different locations, all sorts of stuff.
And we never like all these different things we were, were changing. We like it into a
mental improvements, but we had no good final solution of this. This was the cause. And
because we haven't had a big load event since us, do having done this testing, I was resistant to
this set actually being increased until then, because during, I think it was during the shade
meant a couple of months ago, block times, I think we're approaching 10 minutes. And
that doesn't seem huge. And yeah, increasing set by 10 won't change that too much,
but it's still that much more risk. 10 seconds. No, no, no. It hit 10 minutes per block.
Really? Oh yeah. Okay.
That's insane. I didn't realize it was. I didn't realize it either.
Yeah. Yeah. Now, I mean, it was for like a full day or two that it, that it slowed down to that
point. And so there was a lot of effort that I wanted to try and figure out what the cause was,
right? Yeah. But I mean, maybe, maybe because all the effort that we've put into resolving the issues
is why there hasn't been a huge event like that since, because there have been multiple
entertainment since then and things like that, but still.
The block state was because of that. Because of the size of the monitor set.
Yeah. So it seemed like one of the core issues was when the block started going down,
because it takes so long that like they're became like consensus hubs, you would see them
rotating who was actually getting consensus. Yeah. And I think that was, that was the biggest
issue. And then hardware improvements or like hardware optimization was, was, was the next big,
big answer. Do you think like, I know is a stuff, how much overhead is there in the,
in the encryption? Because yesterday, I switched over, like I'm running bare metal,
like decent hardware. And I switched over to Cosmovisor, right? So I left.
Well, actually, I switched over to Cosmovisor and a remote signer for secret. And I had just
the hardest time catching back up to the head. So I initially switched over to the remote signer
and kept the just running the, the daemon normally. And I was able to catch up, but it was slow.
And then I switched over to Cosmovisor. I missed far fewer blocks when I switched over to Cosmovisor.
But it took, I couldn't catch the head by the time there was the upgrade. And that was probably
a good few hundred blocks out. Is there like lots of overhead in the encryption that makes
bringing the blocks in slow? And should that matter when you're just trying to sync state?
Or is there like some other issue that makes syncing slow? Because I know that I wasn't the
only one who had that problem yesterday. Yes. So there's of course overhead with the encryption
and with running inside of SGX. And because it's running on encrypted memory. I'm not sure of the
recent figures, but I think the bigger overhead is with the WebAssembly engine.
So Cosmovisor are using a WebAssembly engine that's called WASME. And back in 2020, it was not
compatible with SGX. So we went with a WebAssembly engine called WASME, which is a lot more
slower, like 200, maybe 300 times slower than what's currently in Cosmovisor.
We were just trying to make, to implement like secret contracts, you know, making contracts running
SGX. So right now, WASME engine is far slower than the vanilla Cosmovisor was the engine.
And for example, with what shade we're doing, they were doing a cross-chain
add-on. And because they were doing a cross-chain add-on with secret, the hub and the atom and
Luna. And because each of the chain has different coin types, then you can just like convert between
addresses. The same seed would produce different addresses on secret, on Terra, and on the hub.
So what they had to do is have the, if you want to claim the shade airdrop from your
Terra address, you have to sign a proof that you own the Terra address and send it to the shade
contract. And then the shade contract had to do some heavy cryptographic computations to verify it.
And when a lot of people were claiming a lot of shade at the same time, it made like blocks
stretch to be like 10 minutes. So is there any light at the end of that tunnel? Is there any
plan to move to a different WebAssembly engine or is there development in the WOSMA that maybe
will allow use with SGX later on? Yeah, so since then, WOSMA completely reward the code base.
And now that we're finished with the upgrade, that was like a few hours ago, we're going to
start working on migrating our WOSMA engine to be just like what Cousin was using right now.
We should be 200 to 300 expester. That's a pretty big speed up.
So did you say, sorry, sorry, I was just ready comments. Did you say that
it's what was going to produce that 200 to 300 time speed up?
Switching to WOSMA.
So they are switching to WOSMA. So there is like SGX available on that now? Are you guys going to
do that work?
Yeah, we are going to do that work. Like I said, they completely reward the entire WOSMA code base
like in the past year. And it should be compatible with SGX now.
Awesome. That's great news.
Just had a message from usurper saying they've just had a power cut. So that's why he suddenly
has disappeared from the call. So I've just looked just double checking what we've got through all
our questions here. So we've talked about increasing the VALSET. The other question I think that we
this is kind of interesting from the perspective of we've already talked about secret voting,
which is that does it make validators uncomfortable in the community, asking for reasons
on the way they vote on proposals? And I think you've shortly made an interesting point on this
one already when he said that generally speaking, I think people generally only ask you to justify
your position if you are offering a dissenting view, which I mean, I guess makes sense, right?
I can maybe add input on this. We had a couple of projects building the staking derivatives
on secret, which will make you a staking private. And the contract will stake the secret for you.
They also implemented private voting inside the contract to let you validate or know what you
want to vote or to let the contract vote with the voting power, like in the direction that the
stakers are wanting to vote. So that sounds like almost like a contract version of like
orth Z or that kind of functionality, except secret. Yeah, sort of like it's a contract that
that takes the secret for you privately. And then you can for each governance on chain proposal,
you can vote in the contract, what the contract should vote on with the main governance.
Right. That's really cool. I didn't know that that existed on secret, learning a lot tonight.
So I think we've trying to keep it to 90 minutes, I think now is our optimistic goal,
because we like to chat too much to our guests. So we've got one last question, we've got a wrap
up question, which is, what are you most excited about this week in the cosmos?
Shorty, do you want to kick off first? Because you're usually the most positive and least tired
sounding out of all of us. Yeah, I mean, I hear first, let's see. I can't tell you how excited
I am for the improvements from shockwave. I know I've been saying shockwave upgraded for the last
couple of weeks, but secret networks like my home network. So anytime I see improvement there, I'm
like, yes. So I guess there's that for now. I'm also hopeful about UST coming back up. I think
that within a week, it's going to be back up. I've got some strong hope, even though, like I can,
I have that just directed, a direct inject into my veins. So don't listen to me. But I think it'll
be back up. I think that by that time, the market is going to recover, not recover like
stable, but I think that it's going to at least level out, there's going to be less,
people are going to be more happy to be here again.
I think you're right about the, I think USC will definitely repag. I think it will have
probably more volatility going forward though, because of thinner pulls and less backing.
But I definitely think it'll repag. I know we didn't sell anything of our UST
that we still had when it started to depag. We had fortunately, from a risk perspective,
actually sold almost all of our UST just the day before this whole thing kicked off. So we were
lucky in that regard. But that was basically just a, we do like a bit of a risk mitigation
rebase every couple of weeks, just to see where our assets are sitting and what the
risk of that is to our business. So in my case, it's better to move our risk over into our fiat
and let our government have a chance to bugger it up for us instead of a non.
But actually, my thing, we have released our King airdrop. So that's nice. We have been waiting
for Juno Tools all this time. And I'm pretty excited about its release. It was incredibly easy to use
and super smooth. So congratulations to the team who built that. And yeah, so that's
my thing that I'm pretty happy about. If anyone wants to do an airdrop, recommend using Juno
Tools with the Merkle route. It's cheap on gas. It's very easy to use. You can put tens of thousands
of addresses into it. So it's very good. Nice. Yeah. Asaf, what are you most excited about in the
coming week in the course of our launch? Obviously seeing how the Lunar UST situation evolves.
And I'm hopeful that the Lunar community can come together and
interested to see what's going to happen with Lunar. I'm pretty sure that UST will repack, but
pretty interested to see what's going to happen with Lunar.
And yes, super excited to be done with Shockwave after edge and started to work on new things
like Cosmos and V1 and IBC contract and stuff like that.
Must be a huge load off having the work completed on Shockwave.
Yeah. So actually it was the first time that we used the upgrade module up until now we exported
and imported. And it was pretty painful every time and now it was super smooth. It took like
nine and a half minutes to upgrade, which was really nice compared to other times.
Quick question. Do you have a repo that we can do PRs to fix some of the documentation around
Cosmovisor having now implemented it? We've got a few fixes to those docs.
Yeah, I can also just say. Yeah, good. That's cool.
We're happy to go through and fix that up Asaf.
I think that I fixed your comments. But you've already hit it.
Happy if you can make more PRs and make it better. Yeah.
Yeah, might just have another look at it and see if there's any improvements we can make anyway.
To anybody listening, cool people do PRs, cool people fix docs. Be more like Null, fix documentation.
Be more like everyone here. We all hack on documentation in our spare time.
I think that's true. Yeah, as soon as the upgrade finished, the first thing I do is start modifying
the documents, making more accurate. So yeah, people love to have a pedantic streak.
People love. I love to be able to make docs that people can just copy paste.
We should do a whole segment on copy pasting docs and whether or not that's a good thing or a bad
thing. Couldn't we? I think we've talked about it previously about the copy paste glory.
Yeah. It's the new Nomes Blackboard. It's just copy paste question marks and shit in the middle.
I think every copy paste documentation should have rf slash in it and just
to see who's awake. Hold on a second. I'm not going to do that.
One of the most insane things. I'll tell this story and then I think that's pretty much all
the time we have. But one of the most insane things I've ever seen on the internet was back in
over 10 years ago on IRC, I saw on a chat room somebody looking for tech support. They just
logged it. They just started, I think this was pre Docker. They were running on bare metal. They
were trying to redeploy a Ruby on Rails application. They didn't really know what they were doing in
terms of running on Linux and they were having a hard time. A couple of people tried to help and
they'd been kind of an ass about it. The helpful people had gone, okay, you can get on with it.
Essentially a troll had started helping them and just been telling them garbage. Everybody has
been like, well, when you root the people trying to help you, whatever, let's just see how this
plays out. Then it suddenly escalated where this troll was literally like rmr slash kind of thing.
Then the person on IRC who was obviously there was like, it's asking for my password. What should I do?
We're all like, scrabbling for the keyboard being like, don't do it. Then they just connected from
IRC or whatever. It was like, what the hell happened there? I really want to know what the
hell happened there. It's like, was that it? Did they just accidentally trash their laptop? Were
they not even SSHed onto them? What was going on? Then everybody was just sat around and be like,
hey, did that guy just wipe his hard drive or something? I think he said he was on a server.
Can anybody remember what he said 10 minutes ago? This troll also just disconnects as well. It was
like, wow, that was really weird. It was like the UK Ruby or UK Rails IRC a long time ago. I was like,
wow. Maybe he was just trolling himself for shits and giggles. I mean, it might have been a piece
of kind of performance comedy via IRC. That would be a lot of effort again to do that whole kind
of thing. Do a kind of skit. I mean, if it was very well executed, I believed it. I was bought in.
I felt the emotion and I very much enjoyed it. 10 out of 10. We would go and see that
Edinburgh Fringe show again. Do you think that would be quite good? You could do it. You could
do a very avant-garde Edinburgh Fringe comedy festival thing where you did the entire show via
IRC or something like that. But then nobody would get it because who even knows what IRC is anymore?
Hello. Hi. Nice to meet you. That's Discord, right? Basically.
Discord is basically just like IRC without any of the freedom. That's Slack.
Slack was IRC for business, which I thought of literally 25 years ago, but sorry. I only
missed out a few billion there. Yeah, but did you make it? No. There you go. Too early.
Yeah, too early. What's the quote? There's nothing more infuriating than being
10 years too late, except being one year too early. Something like that, right?
So what is your most excited about in the cosmos this coming week?
Shit. Yeah, come back just in time for the final question. Come back just in time.
Come back to me. Let me give me a second. Okay. Well, you got a little bit of time.
Well, I can't come back to you.
My last one because it's me. So I'm going to go, I'm planning on, well, we are planning on
setting up on a cache in the coming weeks and we're going to go because we're obviously moving to using
like Null has and like King Nodes have. We're moving to remote silence across the board.
We're also going to look at moving some of our infrared to a cache. So that's been pretty exciting.
And I think we'll probably do the first, the first one we'll do in that structure will be
the a cache validator. So assuming I have time to do it in between stuff and obviously Prague
is next week, which was very exciting. But I'm more excited about the a cache validator than Prague.
Yeah, it's pretty cool, man. Maybe. Maybe I am.
Are you going to attempt the cache on a cache?
Yeah. Yeah. Yeah, we're going to do it. We're going to do it. We're going to do it. We're going to
we're going to live in glory or we're going to double sign.
Are you going to? Are you going to? Sorry, what was that? Is that a thing? And I keep,
can you run like in a cache validator or in a cache info?
Oh my god. Yeah, I think can now as of, especially as of the most recent upgrade where
they've the storage story has got a lot better. Well, you have been able to do it for a very
long time now. It's just that you had to. Sorry. The way
God, that was going to be interesting. I know. It's interesting.
You're back. He's frozen. Nope. He's frozen again. So you set up your times up.
Oh, I'll go. I'll go up. Yeah. So we're, we're, I was just actually rebuilding
some of the structure this week. So we're also moving to away from just validator signing nodes.
We're, we're going to do a Horcrux based swarm. That swarm is going to be across three vendors
in the U.S. So that's going to be an OVH on AWS. And we have a colo that we run and that we run a
ton of hardware in our own colo. So that's a third note. And so those are all connected. We were,
I was doing a lot of work with like wire garden, some different types of easy management tools of
that like net maker and things like that actually went away from that. And we're going to use Nebula,
which is a, if you guys have heard of Nebula, it's like a kind of like a peer to peer mesh
network built by some of the, some of the groups at Slack initially, because I think they use that
a tremendous amount to be able to do kind of internal routing across a lot of cloud providers.
And then now it's being spawned off. I think some of the either engineering team left or that's a
business with the business or something called defined. So that's, I'll put links in the show
notes around that. It's still pretty early, but man, is it stable. So I had a lot of issues with
wire guard appearing out, starting to do a lot of things where I would drop ports and other types
of things to make sure that those, that swarm would reconnect and everything else and, and just
kind of struggling with that. But so that'll be on three different networks, all 20 milliseconds
away. And so I kind of done some testing to be able to say, I can have a major outage on AWS
or somewhere else and, and still be able to sign. So once I do that, and I need to build some
ansible structures around that for the signing piece of things, then that allows the nodes to
really do anything. And so I think the Akash structure is something that after talking about
that is, is really, really interesting. We still run a ton of Colo, you know, we have a lot of
nodes in our Colo, but, but to be able to spread that out a little bit and reduce that need and then
those types of things. So my goal is that, that away from like some massively something horrible
at messing it happening in the North American East, away from that, that there could be a massive
vendor outage on any one of three or four really large, you know, bare metal hosting providers
and not miss a block. That's really my goal. And so, so anyway, it's just, you know, a lot of testing
kind of structure around that, but that's, I'm getting ready to be able to at least move some,
some stuff over and get that going, just to be able to do some testing and things like that. And
then, then hopefully we'll move some of the more important chains over and yeah, so that's kind of
what I'm working on. Cause more so many bus man is what I was, I was going to say before, but
you've been able to, to use it for validators for a very long time. It's just that you sort of,
the way it's set up, you have to put your keys in.