WEBVTT

00:00:01.954 --> 00:00:02.415
What are we doing?

00:00:02.455 --> 00:00:03.376
I'm fucking tired.

00:00:04.339 --> 00:00:06.083
Hey, that's good.

00:00:26.859 --> 00:00:27.679
welcome to game and knows a

00:00:27.719 --> 00:00:29.260
weekly podcast from

00:00:29.841 --> 00:00:31.041
independent validated teams

00:00:31.120 --> 00:00:33.201
and I always forget when I

00:00:33.261 --> 00:00:34.942
put my headphones on for

00:00:35.002 --> 00:00:37.621
the stream how loud that

00:00:37.722 --> 00:00:40.162
intro is when you're not

00:00:40.682 --> 00:00:41.423
when you've not got it

00:00:41.463 --> 00:00:42.543
coming out of your speakers

00:00:42.624 --> 00:00:43.804
at like a manageable room

00:00:43.823 --> 00:00:44.603
volume but you got it

00:00:44.624 --> 00:00:46.984
cranked and you just go bam

00:00:48.405 --> 00:00:51.566
you uh ends up being yeah

00:00:51.585 --> 00:00:53.426
ends up being a little bit like the um

00:00:55.296 --> 00:00:57.579
The drill tweet where he's like,

00:00:57.738 --> 00:00:58.500
I'm not owned.

00:00:58.619 --> 00:00:59.340
I'm not owned,

00:00:59.500 --> 00:01:01.341
I scream as I shrink down

00:01:01.381 --> 00:01:02.883
and turn into a little corncob.

00:01:04.424 --> 00:01:05.745
That's kind of what I felt like then.

00:01:09.128 --> 00:01:11.269
So, no, you're looking tired there, man.

00:01:11.310 --> 00:01:13.370
What's up?

00:01:13.411 --> 00:01:14.792
Seven AM is becoming more

00:01:14.831 --> 00:01:16.894
and more inconvenient for this podcast.

00:01:18.375 --> 00:01:19.456
Do you want to go to six and

00:01:19.475 --> 00:01:20.075
move it up a little bit?

00:01:20.296 --> 00:01:20.516
Yeah,

00:01:20.537 --> 00:01:22.378
we could just get you up a little bit

00:01:22.418 --> 00:01:22.737
earlier.

00:01:22.798 --> 00:01:23.858
I'm sure your wee one can

00:01:23.899 --> 00:01:24.819
help with that as well.

00:01:25.623 --> 00:01:27.323
Yeah, well, that's the thing because, like,

00:01:29.325 --> 00:01:31.548
just get up a lot in the

00:01:31.587 --> 00:01:32.649
middle of the night and

00:01:32.668 --> 00:01:33.750
then to get up early as

00:01:33.790 --> 00:01:34.650
well when it's not

00:01:34.709 --> 00:01:36.671
necessary is kind of fucking annoying.

00:01:36.691 --> 00:01:40.034
Oh, man.

00:01:40.155 --> 00:01:41.596
Yep.

00:01:41.695 --> 00:01:44.957
Yeah, so, but, you know, other than that,

00:01:45.019 --> 00:01:46.420
mate, I'm fighting.

00:01:47.641 --> 00:01:49.701
Fighting, fiddle, fighting, fit, fit,

00:01:49.721 --> 00:01:50.082
fighting.

00:01:50.963 --> 00:01:52.243
I'm great.

00:01:52.304 --> 00:01:53.484
It's also pissing with rain.

00:01:53.504 --> 00:01:55.105
I was a little bit late here

00:01:55.144 --> 00:01:57.066
because I was sort of looking at my pool,

00:01:57.085 --> 00:01:58.626
wondering whether I should go and fucking,

00:01:58.647 --> 00:02:00.527
I don't know, open the knob or some shit.

00:02:00.706 --> 00:02:01.748
I've never had a pool before.

00:02:02.668 --> 00:02:04.349
Is it overflowing or something?

00:02:04.429 --> 00:02:05.629
It is brimming.

00:02:06.328 --> 00:02:07.709
Yeah, it's been pissing with rain.

00:02:07.730 --> 00:02:10.570
And I was like, if it goes over,

00:02:10.610 --> 00:02:11.531
where's it going to go?

00:02:11.591 --> 00:02:12.091
The lawn?

00:02:12.270 --> 00:02:13.252
Is that bad for the lawn?

00:02:13.352 --> 00:02:13.812
Probably.

00:02:13.831 --> 00:02:15.893
I don't know.

00:02:15.913 --> 00:02:16.092
I mean,

00:02:16.133 --> 00:02:17.353
it's probably half fresh water by

00:02:17.372 --> 00:02:17.712
now anyway.

00:02:18.453 --> 00:02:19.334
Is it above your house or

00:02:19.373 --> 00:02:19.933
below your house?

00:02:20.818 --> 00:02:23.879
Oh, no, it's fucking right down the back.

00:02:24.319 --> 00:02:25.560
Don't those things have some

00:02:25.599 --> 00:02:26.539
sort of automatic way of

00:02:26.580 --> 00:02:27.061
doing that or no?

00:02:27.080 --> 00:02:27.860
I thought there was like an

00:02:27.901 --> 00:02:28.640
overflow in the pump

00:02:28.681 --> 00:02:29.501
structure or something that

00:02:29.520 --> 00:02:31.062
would like... There is a

00:02:31.141 --> 00:02:33.901
bung like right up near the top.

00:02:33.921 --> 00:02:34.581
That's funny.

00:02:34.622 --> 00:02:36.402
But I have it closed because

00:02:36.902 --> 00:02:38.282
when I fill it up from the tap,

00:02:38.342 --> 00:02:39.764
I like to fill it up more

00:02:39.783 --> 00:02:40.984
than it likes to be filled up,

00:02:42.843 --> 00:02:44.004
which now I filled it up a

00:02:44.044 --> 00:02:44.865
couple of days ago and now

00:02:44.884 --> 00:02:45.444
it's pissing rain.

00:02:45.705 --> 00:02:46.504
Maybe I should look at the

00:02:46.544 --> 00:02:48.126
weather forecast before I fill it up.

00:02:48.145 --> 00:02:48.966
No, that's fine.

00:02:48.985 --> 00:02:49.246
Anyway.

00:02:49.548 --> 00:02:52.290
These are real bull market

00:02:52.330 --> 00:02:53.831
problems you've got, Dalvin.

00:02:54.271 --> 00:02:55.432
It must be good times over

00:02:55.473 --> 00:02:57.895
there at Big Bull's blockchain towers.

00:02:58.635 --> 00:03:00.538
Well, the thing is, in Brisbane,

00:03:01.179 --> 00:03:02.120
I don't think you can buy a

00:03:02.159 --> 00:03:04.301
house without a pool.

00:03:04.361 --> 00:03:04.801
Oh, really?

00:03:05.141 --> 00:03:05.983
It must be like fucking

00:03:06.022 --> 00:03:06.804
mandated or something.

00:03:07.163 --> 00:03:09.126
There's like... When we were

00:03:09.145 --> 00:03:10.007
looking for houses,

00:03:10.026 --> 00:03:10.828
we looked at a lot of

00:03:10.867 --> 00:03:12.389
houses and not one of them

00:03:12.429 --> 00:03:13.009
didn't have a pool.

00:03:13.050 --> 00:03:13.870
So...

00:03:15.218 --> 00:03:16.360
We're going to need a fact

00:03:16.379 --> 00:03:18.300
check on that from our

00:03:18.401 --> 00:03:20.543
Australian fact checker.

00:03:22.163 --> 00:03:24.846
You get on Google Maps and look, man,

00:03:24.925 --> 00:03:26.106
and it's just fucking blue

00:03:26.146 --> 00:03:26.747
dots everywhere.

00:03:26.867 --> 00:03:29.050
Even the plebs have got pools around it.

00:03:29.069 --> 00:03:29.969
Do you find yourself using

00:03:29.990 --> 00:03:30.971
it or is it just being ignored?

00:03:31.912 --> 00:03:32.151
Hey.

00:03:32.671 --> 00:03:33.712
Do you use it a lot or no?

00:03:34.052 --> 00:03:34.574
Never swam in it.

00:03:35.054 --> 00:03:35.193
Yeah.

00:03:36.074 --> 00:03:36.895
The only time I went in

00:03:36.936 --> 00:03:38.516
there was to get the fucking leaves out.

00:03:42.052 --> 00:03:42.834
Yeah.

00:03:43.213 --> 00:03:43.675
So far,

00:03:43.694 --> 00:03:45.216
it's cost me a lot of money and I

00:03:45.235 --> 00:03:46.717
haven't used it.

00:03:46.856 --> 00:03:47.457
It's amazing.

00:03:49.118 --> 00:03:50.259
Well, I guess it's starting to warm up too,

00:03:50.319 --> 00:03:50.500
right?

00:03:50.960 --> 00:03:52.461
It's starting to get to pool weather.

00:03:52.542 --> 00:03:53.581
Probably wasn't pool weather at all.

00:03:54.402 --> 00:03:55.723
I think it's like benefit

00:03:55.743 --> 00:03:57.205
would be when we have

00:03:58.026 --> 00:03:59.546
family over and stuff and

00:04:00.627 --> 00:04:01.348
friends with kids.

00:04:01.367 --> 00:04:02.609
You got any music out there

00:04:02.628 --> 00:04:02.930
and everything?

00:04:03.069 --> 00:04:05.532
Is it rocking?

00:04:05.551 --> 00:04:06.312
Do we have music?

00:04:06.932 --> 00:04:07.532
Like out there?

00:04:07.573 --> 00:04:08.894
Like do you have speakers out there?

00:04:08.993 --> 00:04:09.615
You got like little...

00:04:10.425 --> 00:04:10.724
No,

00:04:10.885 --> 00:04:13.688
but since the advent of the Bluetooth

00:04:13.709 --> 00:04:14.088
speaker,

00:04:14.149 --> 00:04:15.430
I feel like we can make it happen.

00:04:15.651 --> 00:04:17.072
You can make it happen.

00:04:18.093 --> 00:04:19.033
That was kind of like

00:04:19.173 --> 00:04:21.237
semi-permanent audiophile

00:04:21.276 --> 00:04:23.218
setup that's got him in his head.

00:04:24.720 --> 00:04:26.182
I don't want to put the

00:04:26.362 --> 00:04:27.903
ceiling speakers through the house,

00:04:27.983 --> 00:04:30.386
but it seems like an unnecessary expense.

00:04:30.665 --> 00:04:31.908
Might have to wait a few years for that.

00:04:32.401 --> 00:04:32.521
Well,

00:04:32.540 --> 00:04:34.463
we have... I use a... Wait for the

00:04:34.483 --> 00:04:35.283
osmosis to come back.

00:04:35.543 --> 00:04:35.824
Yeah, right.

00:04:35.843 --> 00:04:36.463
You got to wait a little bit.

00:04:36.865 --> 00:04:37.125
Bam.

00:04:37.185 --> 00:04:40.307
Wait for my wah-wah to become valuable.

00:04:40.648 --> 00:04:41.487
That router protocol is

00:04:41.507 --> 00:04:42.329
going to start turning into

00:04:42.389 --> 00:04:43.250
real dollars any second.

00:04:43.269 --> 00:04:44.951
Oh, any minute now.

00:04:45.091 --> 00:04:45.610
Yeah.

00:04:45.651 --> 00:04:46.052
Yeah.

00:04:46.351 --> 00:04:47.273
Well, we just did our backyard,

00:04:47.293 --> 00:04:49.434
and so I put wall speakers in.

00:04:49.494 --> 00:04:52.416
I have some stuff in the...

00:04:52.437 --> 00:04:52.997
On the other side,

00:04:53.036 --> 00:04:54.699
it's like... Wires everywhere.

00:04:54.718 --> 00:04:55.639
It's coming to an amp.

00:04:56.379 --> 00:04:57.240
I have a Raspberry Pi

00:04:57.261 --> 00:04:59.002
driving it because I use this thing here.

00:04:59.021 --> 00:05:00.663
I can move in multiple different...

00:05:02.117 --> 00:05:03.218
um, what do you call it?

00:05:03.237 --> 00:05:04.418
Like, you know, like different regions.

00:05:04.439 --> 00:05:05.158
You can have stuff in

00:05:05.238 --> 00:05:06.240
upstairs also be on the,

00:05:06.560 --> 00:05:07.860
in the outside and like all that shit,

00:05:07.880 --> 00:05:08.420
which is great.

00:05:08.500 --> 00:05:08.620
Right.

00:05:08.641 --> 00:05:08.901
I love it.

00:05:09.382 --> 00:05:09.822
And somebody's like,

00:05:09.841 --> 00:05:10.322
why don't you just get a

00:05:10.341 --> 00:05:10.862
Bluetooth speaker?

00:05:10.882 --> 00:05:13.384
I was like, I have like, why,

00:05:14.043 --> 00:05:15.204
why did I do all this

00:05:15.245 --> 00:05:16.505
fucking work when it's kind

00:05:16.526 --> 00:05:17.586
of literally just taken a

00:05:17.646 --> 00:05:18.586
small Bluetooth speaker

00:05:18.627 --> 00:05:19.567
with me and brought it outside.

00:05:20.345 --> 00:05:20.545
I mean,

00:05:20.584 --> 00:05:22.548
this is literally cloud versus colo,

00:05:22.747 --> 00:05:23.468
isn't it?

00:05:23.488 --> 00:05:24.170
That's all age.

00:05:24.490 --> 00:05:24.971
That's all age.

00:05:24.992 --> 00:05:26.213
It's like, oh, yeah, I need music outside.

00:05:26.252 --> 00:05:26.934
Of course I need it.

00:05:26.954 --> 00:05:28.896
Of course I need a thousand

00:05:28.937 --> 00:05:29.538
dollars worth of fucking

00:05:29.577 --> 00:05:30.418
speakers outside and

00:05:30.478 --> 00:05:32.100
running twelve gauge wire everywhere.

00:05:32.341 --> 00:05:33.002
Well, that's what you do.

00:05:34.146 --> 00:05:39.490
So we have like this intercom, right?

00:05:39.870 --> 00:05:39.990
Yeah.

00:05:40.029 --> 00:05:41.730
But it's like a fucking boomer intercom.

00:05:41.790 --> 00:05:42.651
It's like, you know,

00:05:42.711 --> 00:05:44.331
the screen's kind of fadey

00:05:44.372 --> 00:05:48.494
and the fucking, you know, the sound.

00:05:48.613 --> 00:05:50.795
I was imagining like a bake

00:05:50.834 --> 00:05:52.956
light field telephone, you know,

00:05:52.995 --> 00:05:54.057
like from an old war film.

00:05:54.396 --> 00:05:55.358
It basically is.

00:05:55.637 --> 00:05:56.298
It basically is.

00:05:56.478 --> 00:05:57.038
Because, you know,

00:05:57.057 --> 00:05:58.439
we're on a battle axe block, right?

00:05:58.459 --> 00:06:00.139
So there's two gates up to

00:06:00.160 --> 00:06:02.021
the front gate and like,

00:06:02.540 --> 00:06:04.122
the the delivery man comes

00:06:04.141 --> 00:06:05.141
and he's like oh look go

00:06:05.322 --> 00:06:09.204
the off and um when people

00:06:09.244 --> 00:06:10.103
do ring the thing you're

00:06:10.124 --> 00:06:11.324
like it's hard to talk to

00:06:11.345 --> 00:06:12.665
them because they they like

00:06:13.105 --> 00:06:16.247
yeah right yeah does it

00:06:16.266 --> 00:06:17.447
have like an old school you

00:06:17.468 --> 00:06:18.327
have to hit hold a button

00:06:18.348 --> 00:06:19.067
just talk and everything

00:06:19.168 --> 00:06:20.048
like is it one of those no

00:06:20.069 --> 00:06:20.788
you don't have to hold the

00:06:20.889 --> 00:06:24.011
button but like you I can't

00:06:24.050 --> 00:06:25.011
hear what they're saying

00:06:26.130 --> 00:06:28.132
and and so I'm just like

00:06:28.291 --> 00:06:29.052
well maybe they do have to

00:06:29.072 --> 00:06:29.733
hold a button there's only

00:06:29.833 --> 00:06:31.754
one button so I I don't know

00:06:32.182 --> 00:06:33.242
I feel like then I'd be going,

00:06:33.682 --> 00:06:34.584
hold the button.

00:06:35.423 --> 00:06:37.805
Yeah.

00:06:38.887 --> 00:06:39.007
Um,

00:06:39.086 --> 00:06:42.288
so I do desperately want to fuck it off

00:06:42.309 --> 00:06:43.370
because I'm sick of going

00:06:43.389 --> 00:06:45.891
to the post office to get packages.

00:06:46.290 --> 00:06:47.471
Um, cause they just won't,

00:06:47.492 --> 00:06:48.833
they just fucking,

00:06:49.213 --> 00:06:50.954
they look at the thing and go, fuck off.

00:06:51.834 --> 00:06:52.755
So what?

00:06:52.776 --> 00:06:53.196
Cause they have to, Oh,

00:06:53.336 --> 00:06:54.297
you have to let them in the gate.

00:06:54.317 --> 00:06:55.458
Is that what it is?

00:06:55.497 --> 00:06:55.757
Yeah.

00:06:55.858 --> 00:06:56.757
Cause they won't just leave

00:06:56.778 --> 00:06:57.418
them out in the front.

00:06:57.951 --> 00:06:59.932
can you do they have like

00:06:59.952 --> 00:07:01.173
those package boxes by you

00:07:01.293 --> 00:07:01.913
we're seeing those like

00:07:01.934 --> 00:07:02.694
show up a little bit more

00:07:03.314 --> 00:07:04.355
yeah they do but they're

00:07:04.375 --> 00:07:05.437
still like I might as well

00:07:05.456 --> 00:07:06.637
go to the post office you

00:07:06.677 --> 00:07:08.117
know oh I think you just

00:07:08.137 --> 00:07:08.899
take it outside the gate

00:07:08.959 --> 00:07:09.879
and just shove it into a

00:07:09.899 --> 00:07:12.420
something that locks yeah I

00:07:12.461 --> 00:07:14.682
feel like I could bolt

00:07:14.721 --> 00:07:17.343
something to the post for

00:07:17.363 --> 00:07:18.343
the gates attached anyway

00:07:18.663 --> 00:07:21.605
I'm not going to do that um

00:07:21.625 --> 00:07:22.666
complicated stuff

00:07:23.809 --> 00:07:24.269
Anyway,

00:07:24.629 --> 00:07:26.271
I want to get where this was going

00:07:26.331 --> 00:07:28.233
was I want to get one of those, you know,

00:07:28.293 --> 00:07:32.639
like ring doorbells or like a decent,

00:07:32.718 --> 00:07:33.178
decent one.

00:07:33.579 --> 00:07:34.160
Yeah.

00:07:34.180 --> 00:07:35.901
Um, I think ubiquity have some,

00:07:36.362 --> 00:07:41.028
but like a lot of them use POE and, um,

00:07:42.314 --> 00:07:43.656
The cable up there at the

00:07:43.696 --> 00:07:45.237
moment is an Ethernet cable,

00:07:45.978 --> 00:07:47.298
but it's only sort of using

00:07:47.338 --> 00:07:48.338
that for the power.

00:07:48.879 --> 00:07:49.841
And then, you know,

00:07:49.860 --> 00:07:51.302
it's using individual wires

00:07:51.382 --> 00:07:53.122
out of it for this fucking intercom.

00:07:53.242 --> 00:07:55.845
It doesn't use it as a normal PoE device.

00:07:57.446 --> 00:07:58.607
And it's broken somewhere

00:07:58.627 --> 00:08:00.487
along the way and joined

00:08:00.507 --> 00:08:01.809
together with like, I don't know,

00:08:01.848 --> 00:08:02.730
fucking duct tape.

00:08:03.411 --> 00:08:03.651
So...

00:08:05.973 --> 00:08:07.574
I need to get a bunch of data cables run,

00:08:07.954 --> 00:08:08.975
but that thing is right up

00:08:08.995 --> 00:08:10.117
the front of the house and

00:08:10.156 --> 00:08:11.757
it fucking snakes through so much shit.

00:08:11.778 --> 00:08:13.519
Is it like,

00:08:13.579 --> 00:08:14.721
is it buried or is it a conduit?

00:08:14.761 --> 00:08:15.120
You don't know.

00:08:15.141 --> 00:08:15.781
It's just out there.

00:08:15.802 --> 00:08:17.744
It's like in a conduit somewhere,

00:08:18.324 --> 00:08:18.764
I presume.

00:08:21.326 --> 00:08:22.507
But yeah, anyway,

00:08:22.548 --> 00:08:23.749
that's my problem at the moment.

00:08:24.288 --> 00:08:25.730
Fucking do it.

00:08:25.971 --> 00:08:26.411
Yeah.

00:08:26.672 --> 00:08:27.932
Start pulling on cables and shit.

00:08:29.053 --> 00:08:29.274
Yeah.

00:08:29.293 --> 00:08:29.434
Yeah.

00:08:31.112 --> 00:08:33.572
Getting it through, I don't understand.

00:08:33.613 --> 00:08:35.533
Just do like in a cartoon

00:08:35.553 --> 00:08:37.994
where you pull it up and it

00:08:38.014 --> 00:08:39.394
comes up out the ground and

00:08:39.433 --> 00:08:40.315
then like somebody's house

00:08:40.355 --> 00:08:41.674
falls over in the background.

00:08:42.235 --> 00:08:42.514
Yeah,

00:08:42.534 --> 00:08:44.315
like China pulls in from the underside.

00:08:47.096 --> 00:08:48.936
I want to get like some data

00:08:48.956 --> 00:08:49.956
cabling in the house for

00:08:49.975 --> 00:08:51.476
the Wi-Fi points and for

00:08:51.496 --> 00:08:52.856
the office and that type of stuff.

00:08:53.517 --> 00:08:55.956
But I have walked around this thing, man.

00:08:56.037 --> 00:08:57.298
I have no idea how anyone

00:08:57.317 --> 00:08:59.457
would get between the levels.

00:09:02.291 --> 00:09:02.933
Oh, uh-oh.

00:09:02.953 --> 00:09:03.154
Yeah,

00:09:03.174 --> 00:09:04.379
how do you get a fucking employer in

00:09:04.400 --> 00:09:04.561
there?

00:09:04.581 --> 00:09:04.903
I don't know.

00:09:05.889 --> 00:09:07.129
There might be a riser somewhere too.

00:09:07.328 --> 00:09:08.049
Like all of our stuff kind

00:09:08.070 --> 00:09:08.669
of goes into a couple of

00:09:08.690 --> 00:09:11.570
different places that go up, but yeah,

00:09:11.590 --> 00:09:12.431
you might find something or

00:09:12.451 --> 00:09:13.331
there's like one wall that

00:09:13.350 --> 00:09:16.171
there's been cavied out or cavity cavity,

00:09:16.270 --> 00:09:16.792
cavity,

00:09:17.131 --> 00:09:20.451
whatever out cavity that might have a,

00:09:20.591 --> 00:09:20.831
you know,

00:09:21.052 --> 00:09:22.552
that might follow electricity or

00:09:22.572 --> 00:09:23.692
it might follow something else in there.

00:09:23.812 --> 00:09:25.173
Or sometimes we, or honestly,

00:09:25.192 --> 00:09:27.614
if it's just POE, just get a plan, get a,

00:09:27.933 --> 00:09:28.533
there's like plenum and

00:09:28.553 --> 00:09:29.734
there's also riser cables.

00:09:29.833 --> 00:09:30.575
Riser are the, are the,

00:09:30.815 --> 00:09:31.735
are the ones that are more,

00:09:32.375 --> 00:09:33.316
fire resistance because they

00:09:33.355 --> 00:09:34.677
go vertically plenum or

00:09:34.716 --> 00:09:36.519
meant to go horizontally um

00:09:36.619 --> 00:09:37.480
and you can get plenum

00:09:37.539 --> 00:09:38.520
ethernet cables and you can

00:09:38.541 --> 00:09:39.100
just stick them in your

00:09:39.240 --> 00:09:40.842
hvac like it's that's

00:09:40.903 --> 00:09:43.284
totally allowed um and like

00:09:43.325 --> 00:09:44.966
a return so usually returns

00:09:46.006 --> 00:09:47.869
um come like from upstairs

00:09:47.889 --> 00:09:48.570
and you don't have a return

00:09:48.590 --> 00:09:49.509
it's at the base of a floor

00:09:49.529 --> 00:09:50.110
and you can bring that

00:09:50.130 --> 00:09:50.871
downstairs and bring it

00:09:50.892 --> 00:09:52.033
like into your utility room

00:09:52.332 --> 00:09:53.153
and just just cut hole in

00:09:53.173 --> 00:09:54.294
the metal and just snake it

00:09:54.315 --> 00:09:55.135
through and bring it in but

00:09:55.155 --> 00:09:56.297
it has to be a plenum cable

00:09:56.616 --> 00:09:57.398
but you can do that it's

00:09:57.557 --> 00:09:58.719
that's totally fine to do

00:10:00.033 --> 00:10:00.774
So there's some options

00:10:00.793 --> 00:10:01.654
there if you want to.

00:10:01.695 --> 00:10:03.154
I feel like your HVAC works

00:10:03.215 --> 00:10:06.856
different to ours.

00:10:06.917 --> 00:10:07.356
Why?

00:10:07.756 --> 00:10:08.496
Well,

00:10:08.677 --> 00:10:12.097
our heat exchanger is actually in the

00:10:12.138 --> 00:10:13.479
cavity itself.

00:10:13.519 --> 00:10:14.198
Oh, yeah.

00:10:14.318 --> 00:10:17.419
You guys probably mostly use mini splits,

00:10:17.480 --> 00:10:17.620
right?

00:10:17.639 --> 00:10:19.821
Well, it's not a mini split,

00:10:19.880 --> 00:10:21.701
but the system is split.

00:10:21.741 --> 00:10:23.222
You've got the condenser outside,

00:10:24.082 --> 00:10:25.763
and then the heat exchanger

00:10:25.943 --> 00:10:26.964
is in the roof somewhere.

00:10:27.437 --> 00:10:28.197
Oh, okay.

00:10:28.217 --> 00:10:28.658
That's different.

00:10:28.839 --> 00:10:29.078
Yeah.

00:10:29.458 --> 00:10:29.599
Yeah.

00:10:29.619 --> 00:10:30.440
We've connected outside and

00:10:30.460 --> 00:10:31.240
then heat exchangers

00:10:31.360 --> 00:10:32.280
actually in the unit.

00:10:32.360 --> 00:10:33.160
And then there's a,

00:10:33.182 --> 00:10:34.422
there's a fee and a return, right?

00:10:34.461 --> 00:10:35.342
Cause the return gets pulled

00:10:35.363 --> 00:10:36.764
from multiple places and there's a fee.

00:10:36.823 --> 00:10:38.544
Obviously that goes out to registers.

00:10:39.264 --> 00:10:39.725
Fuck man.

00:10:39.745 --> 00:10:40.485
That sounds horrible.

00:10:42.129 --> 00:10:43.110
Well, it doesn't work as well.

00:10:43.889 --> 00:10:45.030
But you guys usually have, like,

00:10:45.051 --> 00:10:46.410
a drop into the basement or some shit,

00:10:46.431 --> 00:10:46.610
don't you?

00:10:46.671 --> 00:10:48.730
Yeah, yeah, yeah.

00:10:48.850 --> 00:10:49.971
Or wherever those things are, right?

00:10:50.611 --> 00:10:52.172
Like, the return front goes everywhere.

00:10:52.231 --> 00:10:53.312
Like, it picks up from, like,

00:10:53.731 --> 00:10:54.451
three different rooms or

00:10:54.471 --> 00:10:55.032
four different rooms.

00:10:55.152 --> 00:10:56.753
And then there's, obviously,

00:10:56.773 --> 00:10:57.712
there's a feed that goes

00:10:57.873 --> 00:10:58.633
everywhere in the house.

00:10:58.732 --> 00:10:59.633
Like, every room has, like,

00:10:59.692 --> 00:11:00.773
I have them above me, right?

00:11:00.913 --> 00:11:01.514
It just goes there.

00:11:01.533 --> 00:11:02.293
So you can use those a

00:11:02.313 --> 00:11:03.874
little bit here and there.

00:11:03.894 --> 00:11:04.673
I think you guys do it

00:11:04.734 --> 00:11:05.374
different because it's

00:11:05.394 --> 00:11:06.534
fucking cold as shit over there.

00:11:06.554 --> 00:11:06.634
Yeah.

00:11:06.654 --> 00:11:07.215
It could be.

00:11:07.235 --> 00:11:07.335
Yeah.

00:11:09.335 --> 00:11:10.855
yeah so we have just like an

00:11:10.914 --> 00:11:12.535
outdoor unit with gas lines

00:11:12.556 --> 00:11:14.216
that come into the ceiling

00:11:15.716 --> 00:11:17.356
with the refrigerant in gas

00:11:17.596 --> 00:11:18.736
it goes to the exchanger in

00:11:18.756 --> 00:11:19.918
the ceiling and then

00:11:20.118 --> 00:11:21.077
basically we only have one

00:11:21.118 --> 00:11:22.217
return because we're

00:11:22.258 --> 00:11:23.597
usually using ac and it

00:11:23.618 --> 00:11:24.759
just to get it to go all

00:11:24.798 --> 00:11:26.818
through the house it drops

00:11:26.859 --> 00:11:27.639
it out in the rooms and

00:11:27.658 --> 00:11:28.639
then sucks it all back to a

00:11:28.700 --> 00:11:29.659
central point in the house

00:11:30.019 --> 00:11:32.100
yeah so it spreads the you

00:11:32.120 --> 00:11:33.100
know makes the house uniform

00:11:34.400 --> 00:11:35.422
So there's only one return

00:11:36.123 --> 00:11:36.984
straight into the unit.

00:11:37.484 --> 00:11:38.804
Got it.

00:11:38.825 --> 00:11:40.726
Got it.

00:11:40.986 --> 00:11:42.668
Frothing on this conversation.

00:11:42.708 --> 00:11:42.828
Well,

00:11:42.869 --> 00:11:44.510
his house is built out of like mud

00:11:44.571 --> 00:11:45.331
and sticks and shit.

00:11:45.350 --> 00:11:49.975
No, it's like, what's air conditioning?

00:11:50.035 --> 00:11:52.937
My, my house, my house is a, is a,

00:11:53.097 --> 00:11:55.159
is an Edwardian, a sensible,

00:11:55.900 --> 00:11:57.542
boring Edwardian house.

00:11:57.581 --> 00:11:58.744
It's only a hundred years old.

00:11:59.224 --> 00:12:00.484
that's it it's only a

00:12:00.524 --> 00:12:01.563
hundred square feet too

00:12:01.945 --> 00:12:04.284
still oh yeah still the new

00:12:04.304 --> 00:12:05.965
house on the street yeah

00:12:05.985 --> 00:12:07.306
when you're like um when

00:12:07.326 --> 00:12:08.265
you guys are talking about

00:12:08.306 --> 00:12:09.105
like oh yeah you've got to

00:12:09.125 --> 00:12:10.025
go down the bottom of the

00:12:10.066 --> 00:12:10.905
whatever and you're just

00:12:10.946 --> 00:12:13.706
like yeah this is where

00:12:13.826 --> 00:12:14.626
this country's a little bit

00:12:14.667 --> 00:12:15.706
more densely packed than

00:12:15.726 --> 00:12:19.687
that you're like I don't

00:12:19.727 --> 00:12:20.687
know well I guess I guess

00:12:20.707 --> 00:12:21.427
the thing is like when you

00:12:21.467 --> 00:12:23.948
live within I don't know

00:12:24.889 --> 00:12:26.028
striking distance of any

00:12:26.089 --> 00:12:27.330
big city like there's just

00:12:27.370 --> 00:12:28.690
not really any space but

00:12:29.553 --> 00:12:30.693
I suppose if you were, like,

00:12:31.214 --> 00:12:32.075
in the arse end, yeah, sure,

00:12:32.095 --> 00:12:33.774
you could have a big garden.

00:12:33.794 --> 00:12:35.635
But... Yeah.

00:12:35.696 --> 00:12:37.916
It's just the food sucks, man.

00:12:38.417 --> 00:12:39.876
Like... Wait,

00:12:39.976 --> 00:12:42.138
have you ever thought of cooking at home?

00:12:42.298 --> 00:12:44.118
Like... Well, no, no, no.

00:12:44.138 --> 00:12:45.619
It's like your lunch break or whatever.

00:12:46.678 --> 00:12:47.538
Like, so if you're, like,

00:12:47.558 --> 00:12:48.558
in the arse end and you

00:12:48.578 --> 00:12:49.980
just have an office in the

00:12:50.080 --> 00:12:51.519
arse end or you go to work

00:12:51.559 --> 00:12:52.921
somewhere in the arse end, like...

00:12:54.301 --> 00:12:55.601
basically gonna get a pie

00:12:55.642 --> 00:12:57.082
for lunch and I like pie

00:12:57.163 --> 00:12:58.043
but I can't do that

00:12:58.063 --> 00:12:59.524
everywhere every day I

00:12:59.544 --> 00:13:00.525
gotta have like different

00:13:00.546 --> 00:13:03.567
lunch options so being near

00:13:03.628 --> 00:13:06.551
a city where I'm like half

00:13:06.591 --> 00:13:08.392
an hour away from like any

00:13:08.631 --> 00:13:09.832
cuisine I can really think

00:13:09.913 --> 00:13:14.897
of that's is is this

00:13:15.057 --> 00:13:16.379
episode one twenty nine it

00:13:16.798 --> 00:13:21.383
is yeah agenda items news items

00:13:22.784 --> 00:13:26.326
So we've got some... We've

00:13:26.365 --> 00:13:31.586
got... Has fucking... Has Rama quit?

00:13:31.606 --> 00:13:33.327
I mean, we haven't paid him in a while.

00:13:33.368 --> 00:13:34.307
There's stuff in there.

00:13:34.668 --> 00:13:35.307
Have we not paid him?

00:13:36.408 --> 00:13:37.028
Not for a while, no.

00:13:37.048 --> 00:13:37.948
There's money in the Dow.

00:13:38.729 --> 00:13:39.249
Yeah, I know.

00:13:39.369 --> 00:13:42.090
I'm lazy now that I have a baby.

00:13:42.330 --> 00:13:44.431
I just prioritize doing

00:13:44.490 --> 00:13:48.471
things that I like.

00:13:48.491 --> 00:13:49.111
Paying people isn't one.

00:13:49.669 --> 00:13:50.850
got the ledger here all

00:13:50.890 --> 00:13:53.412
ready to pay so you know oh

00:13:53.432 --> 00:13:55.812
yeah just saying I can't

00:13:55.832 --> 00:13:56.594
believe you have a ledger

00:13:56.614 --> 00:13:58.193
for that fucking yeah you

00:13:58.214 --> 00:14:00.315
crumb it's it's not the

00:14:00.416 --> 00:14:01.216
only thing that's on that

00:14:01.255 --> 00:14:04.798
ledger but it is wait it is

00:14:04.817 --> 00:14:05.457
the only thing that's on

00:14:05.477 --> 00:14:06.298
that ledger because core

00:14:06.399 --> 00:14:07.558
one doesn't exist anymore

00:14:09.340 --> 00:14:10.240
rama how long has it been

00:14:10.280 --> 00:14:11.000
since we've paid you

00:14:13.510 --> 00:14:14.751
He's in the comments for those.

00:14:17.533 --> 00:14:18.274
He's like, what?

00:14:18.554 --> 00:14:19.014
You pay me?

00:14:19.355 --> 00:14:20.336
Where's that been going?

00:14:21.017 --> 00:14:21.557
It's been going to the

00:14:21.596 --> 00:14:23.197
secret Nell fucking treasury.

00:14:24.779 --> 00:14:25.240
Yeah,

00:14:25.320 --> 00:14:28.903
that's why Nell's got a swimming pool.

00:14:28.962 --> 00:14:33.866
He's been dipping into the pension pots.

00:14:34.826 --> 00:14:36.028
I got a bucket full of water.

00:14:36.087 --> 00:14:38.809
I wouldn't swim in it, though.

00:14:39.530 --> 00:14:40.491
I don't know what that water

00:14:40.511 --> 00:14:41.091
quality is like.

00:14:42.585 --> 00:14:45.166
Tough times.

00:14:45.265 --> 00:14:45.505
Actually,

00:14:45.525 --> 00:14:46.485
we have a lot of shit to talk about.

00:14:46.826 --> 00:14:47.525
We had outers this week.

00:14:47.546 --> 00:14:48.206
We talked about that.

00:14:48.225 --> 00:14:51.106
We have... What else do we have?

00:14:51.126 --> 00:14:52.047
Obviously, some news items.

00:14:52.667 --> 00:14:53.768
Obviously, I'm Satoshi.

00:14:53.807 --> 00:14:54.687
That's now come out.

00:14:54.707 --> 00:14:55.587
It's come out.

00:14:55.607 --> 00:14:56.467
It's finally out.

00:14:56.488 --> 00:14:57.769
It's inevitably going to come out.

00:14:57.969 --> 00:15:01.149
I think we're all Satoshi at heart.

00:15:01.169 --> 00:15:02.009
I'm Satoshi.

00:15:04.509 --> 00:15:07.730
And... Yeah.

00:15:07.750 --> 00:15:10.692
The big one, actually, which I note...

00:15:11.629 --> 00:15:14.732
is in the news thank you

00:15:14.773 --> 00:15:16.495
rama is you know the

00:15:16.514 --> 00:15:19.216
informal thing yeah yeah I

00:15:19.236 --> 00:15:20.477
think that's I think that's

00:15:20.518 --> 00:15:21.899
quite newsworthy why don't

00:15:21.960 --> 00:15:24.841
you tell us what it is yeah

00:15:24.883 --> 00:15:26.604
right okay okay all right

00:15:27.184 --> 00:15:28.966
okay maybe I will so

00:15:30.102 --> 00:15:31.384
Because the audience can't

00:15:31.464 --> 00:15:32.585
read the fucking spreadsheet.

00:15:32.605 --> 00:15:36.649
Yeah, I'm aware of how radio works now.

00:15:37.830 --> 00:15:39.672
So no,

00:15:39.751 --> 00:15:41.673
I was just tutting to myself because

00:15:41.994 --> 00:15:43.995
I like the spicy tone of

00:15:44.015 --> 00:15:45.817
voice that Rama has adopted

00:15:46.638 --> 00:15:47.318
in his phrasing.

00:15:47.879 --> 00:15:49.880
So informal announcers who

00:15:49.900 --> 00:15:51.903
would no longer plunder the

00:15:52.043 --> 00:15:53.625
Atom community pool for

00:15:53.684 --> 00:15:55.606
salaries in twenty twenty five.

00:15:56.249 --> 00:15:57.230
Announcing it's completed

00:15:57.269 --> 00:15:58.211
all development required

00:15:58.230 --> 00:15:59.231
for the future of the hub

00:15:59.371 --> 00:16:00.812
in twenty twenty four with

00:16:00.852 --> 00:16:03.895
forge and hydro Now the

00:16:03.916 --> 00:16:04.916
team will be supporting the

00:16:04.975 --> 00:16:09.921
new team bracket s Coming

00:16:09.980 --> 00:16:16.125
in to nor on the remains So,

00:16:16.166 --> 00:16:17.226
I mean that that is that is

00:16:17.386 --> 00:16:18.687
one take but I don't have

00:16:18.727 --> 00:16:20.408
you read the J tram back

00:16:22.350 --> 00:16:24.153
Tweet about about this I mean it

00:16:28.613 --> 00:16:30.793
The informal post is a

00:16:30.833 --> 00:16:33.375
little bit tag your it in tone.

00:16:33.875 --> 00:16:34.774
So, I mean,

00:16:34.995 --> 00:16:37.135
given that Rama is more

00:16:37.176 --> 00:16:38.375
facetious and also not

00:16:38.416 --> 00:16:42.017
being paid for that work on the hub,

00:16:42.037 --> 00:16:42.996
you can easily see why

00:16:43.057 --> 00:16:46.057
somebody would interpret that post as,

00:16:46.398 --> 00:16:46.597
yeah.

00:16:49.614 --> 00:16:50.634
let some other folks have a

00:16:50.674 --> 00:16:51.575
chew if they want to

00:16:52.254 --> 00:16:55.317
basically it's like after

00:16:55.356 --> 00:16:56.957
the after your dog's like

00:16:56.998 --> 00:16:58.118
taking off the most of the

00:16:58.158 --> 00:17:00.058
meat from the from the

00:17:00.099 --> 00:17:01.679
knuckle and there's just

00:17:01.759 --> 00:17:03.181
like you know slobber and

00:17:03.260 --> 00:17:04.500
sinew left on there and

00:17:04.520 --> 00:17:05.701
then like the other dog

00:17:06.201 --> 00:17:07.143
that's a little bit more

00:17:07.222 --> 00:17:09.523
fucking raggedy looking you

00:17:09.544 --> 00:17:10.943
know slinkers up when the

00:17:10.983 --> 00:17:12.045
other dog goes to take a

00:17:12.065 --> 00:17:13.965
shit in the corner and then like licks it

00:17:14.574 --> 00:17:15.375
I think that's basically

00:17:15.394 --> 00:17:16.955
where we are with the

00:17:16.996 --> 00:17:19.598
cosmos hub would be my,

00:17:19.679 --> 00:17:21.000
the bone is the cosmos hub.

00:17:21.820 --> 00:17:22.141
Yes.

00:17:22.560 --> 00:17:27.326
In this analogy.

00:17:27.365 --> 00:17:30.969
So yeah, the, you just like,

00:17:32.615 --> 00:17:32.875
I mean,

00:17:32.915 --> 00:17:34.017
like my takeaway from it was they

00:17:34.037 --> 00:17:36.318
were basically like, okay, what we did,

00:17:36.338 --> 00:17:38.240
they kind of say something about like, oh,

00:17:38.339 --> 00:17:39.421
for an organization like us,

00:17:39.441 --> 00:17:40.320
we didn't realize how much

00:17:40.361 --> 00:17:41.761
community work could be involved.

00:17:41.781 --> 00:17:43.163
And it's like, well, you know,

00:17:43.222 --> 00:17:44.163
there's money coming from a

00:17:44.183 --> 00:17:44.784
community pool.

00:17:44.864 --> 00:17:45.204
Of course,

00:17:45.224 --> 00:17:46.325
you're going to get a bunch of

00:17:46.345 --> 00:17:47.306
rage and degens.

00:17:47.346 --> 00:17:48.046
That's literally the

00:17:48.086 --> 00:17:49.307
ecosystem you're in is the

00:17:49.346 --> 00:17:51.669
ecosystem that you guys helped kickstart.

00:17:53.527 --> 00:17:54.828
How are you not aware of this,

00:17:54.868 --> 00:17:56.691
how it works by now?

00:17:56.711 --> 00:17:57.151
But that's fine.

00:17:57.250 --> 00:17:57.791
Whatever, man.

00:17:57.832 --> 00:17:59.534
Like also totally cool.

00:18:00.034 --> 00:18:00.674
But they're like, you know,

00:18:00.714 --> 00:18:02.156
we only have this amount of

00:18:02.217 --> 00:18:03.818
hours or people assigned to this.

00:18:03.898 --> 00:18:05.980
And so it can't be managed or whatever.

00:18:06.000 --> 00:18:07.041
And it's like, okay, cool.

00:18:08.501 --> 00:18:09.342
which is another way of

00:18:09.382 --> 00:18:10.363
saying we've been paid a

00:18:10.403 --> 00:18:11.123
lot of money and we've

00:18:11.163 --> 00:18:12.544
decided to not hire anybody,

00:18:12.703 --> 00:18:14.084
even on a short-term contract,

00:18:14.164 --> 00:18:15.365
which is another way of saying,

00:18:15.684 --> 00:18:17.306
we don't believe there is

00:18:18.145 --> 00:18:19.967
life in this or it's sustainable,

00:18:20.287 --> 00:18:22.248
or we don't see our future in it.

00:18:22.807 --> 00:18:24.308
And, you know,

00:18:24.328 --> 00:18:26.009
I surprised that then down the lines,

00:18:26.230 --> 00:18:27.170
they don't see a future in it.

00:18:27.269 --> 00:18:30.951
And they, uh, I guess, uh,

00:18:31.031 --> 00:18:32.853
I guess they are breaking up with the hub,

00:18:32.913 --> 00:18:34.393
right?

00:18:34.493 --> 00:18:35.614
It's not, it's not you.

00:18:35.693 --> 00:18:36.433
It's us, right?

00:18:36.513 --> 00:18:37.775
That's kind of what's going on here.

00:18:38.228 --> 00:18:38.928
At least publicly.

00:18:39.949 --> 00:18:41.309
Yeah, I mean, behind the scenes,

00:18:41.329 --> 00:18:42.170
he fucking knows, right?

00:18:42.190 --> 00:18:43.471
Behind the scenes, he fucking knows,

00:18:43.510 --> 00:18:43.691
but yeah.

00:18:43.711 --> 00:18:47.313
Do you reckon that Informal

00:18:47.333 --> 00:18:48.594
must have a few bucks

00:18:48.773 --> 00:18:50.255
squirreled away at this point, right?

00:18:52.375 --> 00:18:53.076
Yeah, I would think so.

00:18:54.856 --> 00:18:56.157
Are they working on anything else?

00:18:56.657 --> 00:18:56.817
Like,

00:18:56.857 --> 00:18:58.638
what's... I know Rahm has said that

00:18:59.819 --> 00:19:00.980
Informal is building cycles

00:19:01.000 --> 00:19:01.621
and done with the hub.

00:19:02.573 --> 00:19:04.273
They've been trying to do a

00:19:04.314 --> 00:19:05.315
Rust implementation of

00:19:05.375 --> 00:19:06.875
Tendermint forever.

00:19:06.935 --> 00:19:07.135
I mean,

00:19:07.175 --> 00:19:08.916
I literally spoke to some people

00:19:08.957 --> 00:19:09.417
about it.

00:19:09.637 --> 00:19:11.939
I mean,

00:19:11.959 --> 00:19:13.579
they have smart engineers and whatnot,

00:19:16.721 --> 00:19:17.482
but they've been trying to

00:19:18.022 --> 00:19:19.544
ship that Rust Tendermint for like...

00:19:21.791 --> 00:19:23.712
Yeah, it sounds complicated.

00:19:24.032 --> 00:19:27.453
Like, like, like, I mean,

00:19:27.473 --> 00:19:27.993
I suppose somebody in

00:19:28.013 --> 00:19:29.015
twenty two about it.

00:19:29.075 --> 00:19:29.275
Right.

00:19:29.295 --> 00:19:30.515
That was two years ago.

00:19:30.535 --> 00:19:31.175
And you're like, well,

00:19:31.635 --> 00:19:32.756
ten minutes out of date, man.

00:19:32.816 --> 00:19:35.156
So just build something else.

00:19:36.096 --> 00:19:36.297
Yeah.

00:19:36.498 --> 00:19:37.837
Well, maybe maybe that's what this is.

00:19:37.897 --> 00:19:38.837
Well, maybe that's what they're doing.

00:19:39.179 --> 00:19:39.419
I mean,

00:19:39.439 --> 00:19:41.880
I suppose when when all you've got

00:19:41.920 --> 00:19:42.640
is a blockchain,

00:19:42.700 --> 00:19:43.799
everything looks like a.

00:19:46.141 --> 00:19:47.862
Distributed compute platform, you know.

00:19:48.876 --> 00:19:49.897
So I guess, I guess, I mean,

00:19:49.978 --> 00:19:50.917
ultimately it sounds like

00:19:51.057 --> 00:19:53.920
this is something that the

00:19:53.940 --> 00:19:55.059
juice isn't worth the squeeze,

00:19:55.279 --> 00:19:56.921
or at least the,

00:19:56.941 --> 00:19:58.021
when it comes to either the

00:19:58.061 --> 00:19:59.041
revenue associated to the

00:19:59.321 --> 00:20:00.282
cost and the time and

00:20:00.343 --> 00:20:01.222
effort associated to it,

00:20:01.403 --> 00:20:02.604
or just the political

00:20:02.663 --> 00:20:03.865
aspect of it and want to move on,

00:20:04.585 --> 00:20:06.346
or maybe they're just tired, right?

00:20:06.365 --> 00:20:07.906
It could be, it could still be profitable.

00:20:08.027 --> 00:20:09.126
It's just, it's just exhausting.

00:20:09.166 --> 00:20:09.686
And so fuck it.

00:20:10.407 --> 00:20:13.429
Or maybe they built it, tested it,

00:20:14.470 --> 00:20:15.109
still shit,

00:20:16.730 --> 00:20:18.412
decided to just leave it as an enigma.

00:20:21.124 --> 00:20:22.183
Can't fail if you never ship.

00:20:22.865 --> 00:20:23.664
Can't fail if you never ship.

00:20:23.724 --> 00:20:24.684
Well, to be fair to them,

00:20:24.924 --> 00:20:25.625
for the stuff that they

00:20:25.644 --> 00:20:26.365
said they were going to pay

00:20:26.385 --> 00:20:28.066
the community pool, sorry,

00:20:28.105 --> 00:20:28.846
they were being paid by the

00:20:28.865 --> 00:20:30.166
community pool to ship.

00:20:30.247 --> 00:20:31.467
They have shipped it.

00:20:31.487 --> 00:20:35.087
It's just, you know, what is the, I mean,

00:20:35.107 --> 00:20:35.728
I'm literally going like,

00:20:35.948 --> 00:20:36.689
what is the other stuff

00:20:36.729 --> 00:20:37.749
that Inform was working on?

00:20:37.828 --> 00:20:39.028
It's probably literally on their blog.

00:20:39.169 --> 00:20:39.869
Give me a second.

00:20:40.309 --> 00:20:42.829
Yeah.

00:20:42.849 --> 00:20:44.310
I do like the headline of the next one.

00:20:45.951 --> 00:20:47.912
UAE rugs influencer immigrants.

00:20:52.679 --> 00:20:53.200
Continue.

00:20:54.423 --> 00:20:55.243
Oh, fucking.

00:20:55.344 --> 00:20:55.625
Fuck is that?

00:20:55.644 --> 00:20:56.826
Well, I was expecting it coming.

00:20:56.926 --> 00:20:57.709
Oh, geez, that's witty.

00:20:57.729 --> 00:20:57.808
Yeah.

00:20:57.828 --> 00:20:57.909
Fuck.

00:20:57.929 --> 00:20:58.810
Yeah.

00:21:02.257 --> 00:21:02.717
Continue.

00:21:03.417 --> 00:21:05.638
This is in the words of Rama now.

00:21:05.858 --> 00:21:07.720
Crypto Cito et al.

00:21:08.740 --> 00:21:10.442
are in shambles after the

00:21:10.582 --> 00:21:12.143
UAE introduces new laws

00:21:12.182 --> 00:21:13.983
prohibiting residents from

00:21:14.124 --> 00:21:15.144
pump and dumping their

00:21:15.265 --> 00:21:16.965
followers slash viewers.

00:21:17.626 --> 00:21:18.227
Not really.

00:21:18.567 --> 00:21:20.208
They just introduced new

00:21:20.228 --> 00:21:21.328
disclaimer laws for

00:21:21.368 --> 00:21:23.109
companies promoting digital assets.

00:21:23.589 --> 00:21:25.912
Kind of funny though,

00:21:26.711 --> 00:21:28.252
after so many crypto people

00:21:28.292 --> 00:21:29.134
moved there recently.

00:21:29.866 --> 00:21:30.567
Now we will know when

00:21:30.587 --> 00:21:33.309
they're getting paid to shill.

00:21:33.329 --> 00:21:34.911
But it's not paid to shill, right?

00:21:35.030 --> 00:21:36.152
Is it paid for marketing or

00:21:36.172 --> 00:21:36.912
is it just if they're

00:21:36.971 --> 00:21:37.833
marketing in any sort of

00:21:37.873 --> 00:21:39.054
way associated to crypto assets?

00:21:39.074 --> 00:21:39.574
I got it.

00:21:39.634 --> 00:21:40.555
It's more of the latter.

00:21:40.615 --> 00:21:42.836
Like they can't just market

00:21:43.717 --> 00:21:44.478
whether for themselves or

00:21:44.498 --> 00:21:45.659
for anybody else or just in

00:21:45.759 --> 00:21:49.741
general without that kind of... Yeah,

00:21:49.801 --> 00:21:53.304
I presume it would be laws

00:21:53.384 --> 00:21:55.346
similar to like insider trading.

00:21:58.252 --> 00:22:00.193
yeah yeah and actually the

00:22:00.233 --> 00:22:03.094
the um the uh what do you

00:22:03.114 --> 00:22:03.993
call it I guess it's a

00:22:04.034 --> 00:22:05.035
disclaimer disclaimer

00:22:05.134 --> 00:22:06.154
probably the word it didn't

00:22:06.174 --> 00:22:06.954
seem that bad it was just

00:22:06.974 --> 00:22:08.714
like hey this is all this

00:22:08.734 --> 00:22:09.695
could go away any possible

00:22:09.736 --> 00:22:10.855
second right is basically

00:22:10.875 --> 00:22:11.655
all it was it wasn't too

00:22:11.695 --> 00:22:14.376
much beyond that yeah so I

00:22:14.416 --> 00:22:15.656
guess well from what my

00:22:15.737 --> 00:22:17.596
aram is saying it's uh they

00:22:17.616 --> 00:22:18.257
have to just put a

00:22:18.297 --> 00:22:19.218
disclaimer saying that

00:22:19.238 --> 00:22:20.698
they've been paid to market this

00:22:23.396 --> 00:22:25.459
Or they own a lot and are

00:22:25.479 --> 00:22:26.318
going to dump on your ass.

00:22:26.338 --> 00:22:30.843
We're planning on dumping on your ass.

00:22:31.042 --> 00:22:32.344
You are the exit liquidity.

00:22:35.685 --> 00:22:39.969
So I have the answer to what is Cycles.

00:22:40.229 --> 00:22:41.711
Cycles is the product

00:22:41.750 --> 00:22:43.192
they're currently focusing on,

00:22:43.612 --> 00:22:45.794
which is interbank clearing.

00:22:46.773 --> 00:22:48.494
essentially but they say

00:22:48.515 --> 00:22:50.435
it's an open protocol for

00:22:50.816 --> 00:22:52.676
doing the same job as

00:22:52.876 --> 00:22:54.978
back-to-back clearing and

00:22:55.018 --> 00:22:56.298
it's built on the cosmos

00:22:56.338 --> 00:22:58.680
stack and they have a

00:22:58.720 --> 00:23:00.060
slightly flashy website for

00:23:00.141 --> 00:23:06.483
it but no other info what is it

00:23:07.794 --> 00:23:12.336
It is a way of doing debt settlement,

00:23:12.415 --> 00:23:12.715
I guess,

00:23:12.756 --> 00:23:14.476
because clearing is just debt settlement,

00:23:14.516 --> 00:23:14.756
right?

00:23:14.955 --> 00:23:17.297
It's just denominated somehow.

00:23:17.957 --> 00:23:22.419
So instead of doing debt

00:23:22.439 --> 00:23:23.618
settlement via a bank,

00:23:23.699 --> 00:23:25.339
you can instead use cycles.

00:23:26.859 --> 00:23:27.400
Potentially,

00:23:27.420 --> 00:23:29.181
because it's not implemented yet,

00:23:29.240 --> 00:23:30.101
judging by the fact there's

00:23:30.122 --> 00:23:31.301
just a splash page for it,

00:23:32.542 --> 00:23:34.644
which means you can now use

00:23:34.824 --> 00:23:38.565
any IBC bridged asset to

00:23:39.405 --> 00:23:42.126
pay somebody and dox that you've done it.

00:23:44.715 --> 00:23:47.458
you can okay you can pay

00:23:47.478 --> 00:23:48.979
somebody then all right

00:23:49.038 --> 00:23:50.779
yeah basically that's I

00:23:50.799 --> 00:23:51.800
don't know I'm being super

00:23:51.820 --> 00:23:52.861
fucking facetious obviously

00:23:52.881 --> 00:23:56.604
it would be private but um

00:23:57.163 --> 00:23:59.065
but but but like you know

00:23:59.346 --> 00:24:00.185
it's used as the cosmo

00:24:00.205 --> 00:24:02.248
stack so whatever that's a

00:24:02.468 --> 00:24:05.148
pretty big ask to work back from um

00:24:06.150 --> 00:24:07.911
So, yeah.

00:24:07.971 --> 00:24:10.192
So, yeah, Rama says Rama's non-facetious.

00:24:10.531 --> 00:24:12.313
Weirdly, I'm now being the facetious one.

00:24:12.452 --> 00:24:15.154
Rama is now being the informative one.

00:24:15.194 --> 00:24:16.134
Rama says in the chat,

00:24:16.275 --> 00:24:17.776
P to P debt settlement.

00:24:18.115 --> 00:24:19.497
That's a good way of looking at it.

00:24:21.137 --> 00:24:21.458
You know,

00:24:24.000 --> 00:24:26.382
a thought would be what is debt

00:24:26.742 --> 00:24:29.845
and a debt with a capital D

00:24:29.904 --> 00:24:34.209
has a lot of definitions and, you know,

00:24:34.808 --> 00:24:35.950
there's credits and debts

00:24:36.009 --> 00:24:37.230
and there are simplified

00:24:37.270 --> 00:24:38.352
models of these things.

00:24:38.571 --> 00:24:39.972
And a lot of it ends up

00:24:40.012 --> 00:24:42.275
being transactions and debt.

00:24:42.478 --> 00:24:44.380
obligations and whatnot and

00:24:45.320 --> 00:24:47.021
I don't know I don't really

00:24:47.061 --> 00:24:48.784
see how the kind of

00:24:48.824 --> 00:24:50.025
simplified model for this

00:24:50.045 --> 00:24:51.546
stuff that's that's being

00:24:51.586 --> 00:24:52.467
discussed is actually

00:24:52.487 --> 00:24:53.247
really that different from

00:24:53.288 --> 00:24:54.268
any of the stuff that smart

00:24:54.288 --> 00:24:55.609
contracts we do or the

00:24:55.630 --> 00:24:56.490
transactions we currently

00:24:56.509 --> 00:24:57.611
issue on open blockchains

00:24:57.711 --> 00:25:02.635
are um so so to bubble it

00:25:02.695 --> 00:25:03.656
up they're moving away from

00:25:03.717 --> 00:25:05.377
building platforms and

00:25:05.397 --> 00:25:06.338
they're building apps

00:25:07.990 --> 00:25:10.231
Yeah, I guess they, Rama says,

00:25:10.412 --> 00:25:11.492
many people can pay for

00:25:11.554 --> 00:25:12.555
someone else's debt.

00:25:14.076 --> 00:25:19.101
Yeah, I mean, yeah, cool.

00:25:19.500 --> 00:25:20.942
They're going to need a

00:25:20.982 --> 00:25:22.223
better stack than Cosmos if

00:25:22.243 --> 00:25:22.924
they want to do that,

00:25:22.944 --> 00:25:23.704
but good luck to them.

00:25:25.207 --> 00:25:25.487
Would they?

00:25:25.586 --> 00:25:26.607
Why?

00:25:26.647 --> 00:25:27.348
Seems like it'd be fine.

00:25:28.086 --> 00:25:30.287
Because Cosmos is like, well,

00:25:30.307 --> 00:25:30.968
it depends which bits of

00:25:30.988 --> 00:25:32.087
the Cosmos stack they want to use,

00:25:32.107 --> 00:25:33.368
but Tendermint ain't quick enough.

00:25:34.368 --> 00:25:35.329
Can't be optimized to be

00:25:35.369 --> 00:25:37.590
quick enough without major rewrite.

00:25:38.030 --> 00:25:38.571
There's a whole bunch of

00:25:38.592 --> 00:25:40.113
problems with its implementation.

00:25:40.192 --> 00:25:41.573
It's written in the wrong language,

00:25:41.814 --> 00:25:42.453
I would say.

00:25:43.594 --> 00:25:46.556
And the SDK is a fucking disaster.

00:25:46.635 --> 00:25:47.876
So obviously you don't want

00:25:49.045 --> 00:25:50.365
core banking infrastructure

00:25:50.385 --> 00:25:51.907
written with the cosmos sdk

00:25:51.928 --> 00:25:56.092
um so that's that and and

00:25:56.112 --> 00:25:57.173
also actually there's a

00:25:57.232 --> 00:25:58.874
broader point right which

00:25:58.913 --> 00:26:00.955
is that um multi-party

00:26:00.996 --> 00:26:02.917
credit is really

00:26:02.978 --> 00:26:04.740
complicated from a game

00:26:04.779 --> 00:26:05.800
theory perspective right

00:26:05.881 --> 00:26:07.041
and this has been um

00:26:07.142 --> 00:26:09.044
studied and modeled mathematically

00:26:10.263 --> 00:26:10.845
in the past,

00:26:12.065 --> 00:26:13.645
both in real world systems and

00:26:13.726 --> 00:26:14.645
in e-commerce.

00:26:14.685 --> 00:26:16.406
So e-commerce fraud and

00:26:16.446 --> 00:26:17.968
credit fraud are two really,

00:26:17.988 --> 00:26:18.827
really big areas,

00:26:19.949 --> 00:26:21.169
especially credit fraud.

00:26:21.368 --> 00:26:23.750
And it's like an area where

00:26:24.330 --> 00:26:25.671
I gather at various points,

00:26:25.711 --> 00:26:26.692
you've been able to very

00:26:26.771 --> 00:26:27.832
easily get some funding for

00:26:27.852 --> 00:26:29.932
a PhD if you want to go and

00:26:30.073 --> 00:26:31.713
do a lot of research into credit fraud,

00:26:31.753 --> 00:26:31.933
right?

00:26:33.714 --> 00:26:34.955
And so there's kind of the

00:26:35.036 --> 00:26:36.657
incentives to like, you know,

00:26:36.698 --> 00:26:37.939
cheat defect and whatever

00:26:38.038 --> 00:26:39.740
in those are quite well studied.

00:26:39.920 --> 00:26:42.702
And, you know, like a lot of these,

00:26:44.403 --> 00:26:45.305
like a lot of these systems,

00:26:45.505 --> 00:26:46.705
like Cosmos itself,

00:26:46.766 --> 00:26:47.506
like all the stuff we've

00:26:47.526 --> 00:26:48.727
talked about with upgrades

00:26:49.067 --> 00:26:51.690
and governance and whatnot,

00:26:53.070 --> 00:26:54.132
it turns out the hard bit

00:26:54.531 --> 00:26:55.593
isn't necessarily like

00:26:55.653 --> 00:26:56.693
stopping a double spend.

00:26:57.375 --> 00:26:57.575
It's,

00:26:58.839 --> 00:27:00.161
all of the social dynamics

00:27:00.201 --> 00:27:01.461
of everything else that is

00:27:01.520 --> 00:27:03.182
involved in the network

00:27:03.521 --> 00:27:05.564
still functioning um

00:27:05.884 --> 00:27:06.604
amongst other things

00:27:06.644 --> 00:27:07.304
there's a whole bunch of

00:27:07.324 --> 00:27:08.164
other things right because

00:27:08.184 --> 00:27:10.405
obviously like you can just

00:27:11.885 --> 00:27:13.366
you can just take cash off people and,

00:27:13.487 --> 00:27:14.267
you know, whatever.

00:27:14.287 --> 00:27:15.268
There's a whole bunch of stuff to it.

00:27:15.428 --> 00:27:15.548
It's,

00:27:15.647 --> 00:27:17.608
it's complex and I'm an idiot and I'm not,

00:27:17.888 --> 00:27:19.769
I'm way out of my lane here, but anyway,

00:27:19.809 --> 00:27:20.710
whatever.

00:27:20.930 --> 00:27:21.849
Um,

00:27:21.890 --> 00:27:24.791
so the funniest thing I will say about it,

00:27:24.811 --> 00:27:26.251
by the way, I should also add informal,

00:27:26.512 --> 00:27:28.292
like no particular beef with informal.

00:27:28.313 --> 00:27:29.013
They're clever people.

00:27:29.074 --> 00:27:29.534
Like I say,

00:27:29.933 --> 00:27:30.834
from every conversation I've

00:27:30.855 --> 00:27:32.134
actually had with informal engineers,

00:27:32.515 --> 00:27:34.516
they're clever, lovely people.

00:27:34.596 --> 00:27:37.457
Sorry, but all of that disclaimer aside,

00:27:37.857 --> 00:27:38.998
their website is worded in

00:27:39.018 --> 00:27:39.818
a really funny way.

00:27:40.578 --> 00:27:40.679
Um,

00:27:42.133 --> 00:27:43.013
I'm just going to read you

00:27:43.034 --> 00:27:44.275
the headline of their website.

00:27:44.394 --> 00:27:46.817
Okay.

00:27:46.837 --> 00:27:49.339
Informal your partner across

00:27:49.359 --> 00:27:51.521
the interchange.

00:27:51.541 --> 00:27:52.903
Our mission is to foster

00:27:52.963 --> 00:27:58.429
trust in software and money.

00:27:58.449 --> 00:27:58.789
That's it.

00:27:59.009 --> 00:27:59.069
Our,

00:27:59.390 --> 00:28:01.632
our mission is to foster trust in

00:28:01.731 --> 00:28:03.894
software and money.

00:28:03.974 --> 00:28:04.174
Right.

00:28:05.730 --> 00:28:07.192
But is that sentence,

00:28:07.612 --> 00:28:08.673
our mission is to foster

00:28:08.732 --> 00:28:10.855
trusted software and money?

00:28:11.115 --> 00:28:11.434
And money.

00:28:11.795 --> 00:28:13.096
Or is it,

00:28:13.115 --> 00:28:14.457
our mission is to foster trust in

00:28:14.497 --> 00:28:15.397
software and money?

00:28:16.479 --> 00:28:20.501
They basically, I think, missed a comma.

00:28:21.041 --> 00:28:22.623
So it's ambiguous how you're

00:28:22.643 --> 00:28:24.305
supposed to read that sentence.

00:28:24.444 --> 00:28:26.746
And I think I choose to read it,

00:28:27.846 --> 00:28:28.887
our mission is to foster

00:28:28.928 --> 00:28:31.390
trust in software and money.

00:28:31.450 --> 00:28:31.569
Yeah.

00:28:34.431 --> 00:28:36.213
Hey, look, all of our business is money.

00:28:36.314 --> 00:28:37.976
We're just more, we're just validators.

00:28:38.056 --> 00:28:39.198
We're just, we're the DJs.

00:28:39.218 --> 00:28:40.519
We know our business is to

00:28:40.559 --> 00:28:42.843
foster trust in money entering

00:28:43.641 --> 00:28:44.721
our validator wallets

00:28:44.761 --> 00:28:45.902
because you know we need to

00:28:45.961 --> 00:28:48.202
dump it to tank the price

00:28:48.242 --> 00:28:49.064
that's what we've seen from

00:28:49.104 --> 00:28:50.644
network analysis right yeah

00:28:50.684 --> 00:28:52.086
of course that's that's

00:28:52.105 --> 00:28:53.105
what we're here for right

00:28:53.145 --> 00:28:56.307
job one that's why we got

00:28:56.367 --> 00:28:57.848
into this space it wasn't

00:28:57.989 --> 00:28:59.650
idealism it was because uh

00:28:59.690 --> 00:29:01.210
we wanted to tank the price

00:29:01.309 --> 00:29:02.810
of random coins that were

00:29:02.851 --> 00:29:04.491
already tanking because

00:29:04.511 --> 00:29:07.814
their founders were selling anyway

00:29:10.094 --> 00:29:10.213
Hey,

00:29:10.233 --> 00:29:11.035
do you remember when there was that

00:29:11.075 --> 00:29:12.277
whole wild thing where it

00:29:12.336 --> 00:29:14.019
turned out that Wolf was

00:29:14.059 --> 00:29:15.221
just selling the whole time

00:29:15.481 --> 00:29:19.428
on the way up and all the way down?

00:29:20.108 --> 00:29:20.568
That means, I guess,

00:29:20.588 --> 00:29:22.051
Inform will launch a token with that too,

00:29:22.092 --> 00:29:22.271
right?

00:29:23.843 --> 00:29:24.683
Does it now?

00:29:24.703 --> 00:29:26.305
Yeah, that's where the money's made.

00:29:26.345 --> 00:29:27.746
I hear that's where you make

00:29:27.786 --> 00:29:28.405
the money now.

00:29:29.046 --> 00:29:30.067
I've heard that this is how

00:29:30.086 --> 00:29:34.349
you make the money.

00:29:34.390 --> 00:29:34.809
Not in this.

00:29:35.009 --> 00:29:36.951
I'm not going to shit talk.

00:29:37.672 --> 00:29:38.932
I nearly started shit

00:29:38.952 --> 00:29:39.532
talking about something

00:29:39.553 --> 00:29:43.596
that is currently private then.

00:29:43.635 --> 00:29:43.935
Next week.

00:29:44.943 --> 00:29:45.804
Yeah.

00:29:45.903 --> 00:29:46.104
Yeah.

00:29:46.124 --> 00:29:47.484
So they, yeah.

00:29:47.525 --> 00:29:48.244
Yeah.

00:29:48.265 --> 00:29:49.325
Printing your own money is a

00:29:49.444 --> 00:29:50.345
lot easier than taking

00:29:50.384 --> 00:29:51.285
grants from somebody else.

00:29:51.644 --> 00:29:51.965
Yes.

00:29:52.286 --> 00:29:52.925
I was going to say,

00:29:52.945 --> 00:29:53.506
I was going to say the

00:29:53.546 --> 00:29:55.165
regulation in the UK is no

00:29:55.205 --> 00:29:56.346
longer boys will be boys on

00:29:56.385 --> 00:29:57.346
printing your own money.

00:29:57.807 --> 00:29:57.866
Um,

00:29:57.926 --> 00:29:59.406
and it's no longer boys will be boys on

00:29:59.547 --> 00:30:00.906
advertising your shit coins,

00:30:01.186 --> 00:30:02.288
kind of like the UAE.

00:30:02.948 --> 00:30:05.107
Um, and, uh, you know,

00:30:05.228 --> 00:30:06.328
I'm speaking about you, sir,

00:30:06.348 --> 00:30:08.669
but there was a, a private, uh,

00:30:08.709 --> 00:30:09.888
situation in the last

00:30:10.548 --> 00:30:12.589
couple of weeks that was a bit, um,

00:30:14.030 --> 00:30:14.892
what have you guys done for

00:30:14.951 --> 00:30:17.433
us lately that involves something?

00:30:17.513 --> 00:30:19.375
And there was one particular

00:30:19.776 --> 00:30:21.678
box on this thing.

00:30:21.998 --> 00:30:23.839
And my reaction was,

00:30:23.880 --> 00:30:25.300
I'm pretty sure if we did

00:30:25.361 --> 00:30:26.261
anything around that,

00:30:26.321 --> 00:30:28.002
it would be flat illegal in

00:30:28.063 --> 00:30:29.183
our jurisdiction.

00:30:30.605 --> 00:30:31.125
Yeah.

00:30:32.166 --> 00:30:33.147
Kind of shill tokens.

00:30:33.347 --> 00:30:34.888
We just run servers, man.

00:30:34.989 --> 00:30:37.791
We don't know what the servers do.

00:30:37.811 --> 00:30:39.272
I don't know.

00:30:39.292 --> 00:30:40.614
I don't even know where these came from.

00:30:42.665 --> 00:30:44.186
Yeah, exactly.

00:30:44.247 --> 00:30:45.846
Your Null's going to have that problem.

00:30:46.126 --> 00:30:47.188
Please raid his house.

00:30:47.307 --> 00:30:50.088
He's like, I have no idea.

00:30:50.108 --> 00:30:51.068
I'm pretty sure we don't

00:30:51.088 --> 00:30:51.709
even have any servers.

00:30:51.769 --> 00:30:52.548
And they're like, sir,

00:30:52.568 --> 00:30:54.049
you have fifteen hundred

00:30:54.109 --> 00:30:55.470
hard drives in your garage?

00:30:57.250 --> 00:30:58.171
They're meant to all be

00:30:58.250 --> 00:30:59.432
second-hand hard drives

00:30:59.451 --> 00:31:01.592
with very heavy amounts of wear on them.

00:31:01.612 --> 00:31:02.653
They wouldn't even be able

00:31:02.673 --> 00:31:03.373
to get in the gate.

00:31:03.972 --> 00:31:07.513
Fucking bell doesn't work.

00:31:07.534 --> 00:31:08.134
Guess he's not here.

00:31:08.193 --> 00:31:09.115
Yeah, they're not getting in here.

00:31:10.759 --> 00:31:11.559
I guess he's not here.

00:31:11.720 --> 00:31:13.381
Turn around.

00:31:13.401 --> 00:31:14.422
You know,

00:31:15.781 --> 00:31:18.223
if my remote broke while I was away,

00:31:18.384 --> 00:31:19.265
I don't know how the fuck

00:31:19.285 --> 00:31:20.405
we would get into this place.

00:31:20.925 --> 00:31:22.666
We'd have to like... Yeah, scale it.

00:31:22.686 --> 00:31:24.788
We'd have to get fucking

00:31:25.028 --> 00:31:26.130
helicoptered in or something.

00:31:26.529 --> 00:31:28.590
Don't you have a secret pad

00:31:28.631 --> 00:31:29.912
out there or anything?

00:31:29.991 --> 00:31:31.353
Or just like a keyhole or something?

00:31:31.973 --> 00:31:32.973
You can't get in with a key.

00:31:33.134 --> 00:31:33.394
No.

00:31:33.914 --> 00:31:35.236
There's a keypad that doesn't work.

00:31:36.701 --> 00:31:39.303
Maybe that's something to look at.

00:31:39.323 --> 00:31:45.164
Except to fucking take the title.

00:31:45.224 --> 00:31:48.326
This is classic validated problem solving.

00:31:48.767 --> 00:31:49.826
It's like, well,

00:31:49.846 --> 00:31:51.008
there's only this one line

00:31:51.028 --> 00:31:53.249
of defense before literally

00:31:53.388 --> 00:31:54.930
everything is ruined forever.

00:31:55.794 --> 00:31:57.394
What happens if I lose this SSH key?

00:31:57.515 --> 00:31:57.615
Huh.

00:31:57.634 --> 00:31:59.875
What happens if that happens?

00:31:59.996 --> 00:32:00.796
That would be bad.

00:32:00.816 --> 00:32:01.497
I'm fucked in that situation.

00:32:01.656 --> 00:32:04.657
Oh, please, God, no, not the SSH key.

00:32:05.479 --> 00:32:06.858
Should we do something about that?

00:32:06.919 --> 00:32:09.401
Nah, it'll probably be all right.

00:32:09.421 --> 00:32:11.842
Nah, it'll be fine.

00:32:12.561 --> 00:32:14.242
That's fine.

00:32:14.262 --> 00:32:15.243
This one point of failure

00:32:15.263 --> 00:32:16.003
will never fail you.

00:32:16.443 --> 00:32:17.424
This one point of failure

00:32:17.444 --> 00:32:19.526
would be very unfortunate if it failed.

00:32:19.905 --> 00:32:21.366
Highly improbable, some would say.

00:32:22.454 --> 00:32:24.056
it'd be nice if Schultz was here.

00:32:24.175 --> 00:32:26.896
Cause I would like to ask him if they,

00:32:27.277 --> 00:32:28.416
cause they have like, you know,

00:32:28.717 --> 00:32:29.778
more people in the team.

00:32:30.057 --> 00:32:31.238
I'm wondering like what they

00:32:31.357 --> 00:32:36.240
use for their SSH key management and like,

00:32:36.259 --> 00:32:36.941
they might distribute them.

00:32:36.961 --> 00:32:37.921
It's not groups or something.

00:32:39.192 --> 00:32:42.354
Well, there's software for managing that.

00:32:42.375 --> 00:32:44.536
That way, you can have teams,

00:32:44.576 --> 00:32:46.397
and if you ditch a team member,

00:32:46.417 --> 00:32:48.098
you can just press the button,

00:32:48.138 --> 00:32:50.381
and they no longer can SSH.

00:32:50.500 --> 00:32:54.284
Yeah, so we can just use Ansible for that.

00:32:54.304 --> 00:32:54.943
You can definitely do that.

00:32:54.963 --> 00:32:55.825
Yeah, but what if they're quicker?

00:32:58.067 --> 00:32:59.008
They lock you out.

00:32:59.067 --> 00:33:00.048
Depends on... But also,

00:33:00.568 --> 00:33:01.388
in those situations...

00:33:02.315 --> 00:33:02.515
there,

00:33:02.615 --> 00:33:03.516
there probably is some stuff that's

00:33:03.536 --> 00:33:04.836
sensitive, but for the most part,

00:33:04.916 --> 00:33:05.477
like if you wanted to,

00:33:05.916 --> 00:33:06.837
like if you had that case,

00:33:06.857 --> 00:33:07.917
like you would have it

00:33:07.938 --> 00:33:08.358
where you might have

00:33:08.398 --> 00:33:09.138
Horcrux nodes or those

00:33:09.159 --> 00:33:10.259
types of things that only you have access,

00:33:10.839 --> 00:33:12.060
excuse me, that you have access to,

00:33:12.201 --> 00:33:14.662
but you might distribute, you know,

00:33:14.682 --> 00:33:15.462
like if you had a,

00:33:15.803 --> 00:33:17.403
if you had a little minion minion,

00:33:17.423 --> 00:33:18.624
you have access to all the nodes,

00:33:18.664 --> 00:33:19.944
but you can't do any damage with that.

00:33:20.085 --> 00:33:20.506
You know what I mean?

00:33:20.586 --> 00:33:21.625
Other than.

00:33:21.766 --> 00:33:21.885
Yeah.

00:33:22.026 --> 00:33:22.307
Yeah.

00:33:23.490 --> 00:33:25.030
fuck your nodes up yeah but

00:33:25.070 --> 00:33:26.132
I mean nobody's gonna be

00:33:26.172 --> 00:33:27.873
you can't you can't if you

00:33:27.893 --> 00:33:28.693
want if you if you had

00:33:28.713 --> 00:33:29.673
somebody in the house who

00:33:29.714 --> 00:33:30.494
wanted to be malicious

00:33:30.515 --> 00:33:31.455
there's really no way to

00:33:31.476 --> 00:33:32.557
stop those things yeah

00:33:32.596 --> 00:33:33.717
right you you really you're

00:33:33.737 --> 00:33:35.097
really trying to protect

00:33:35.137 --> 00:33:36.479
against mistakes and

00:33:36.499 --> 00:33:37.740
stupidity I mean like a

00:33:37.779 --> 00:33:39.481
very simple really hard

00:33:39.501 --> 00:33:40.082
like that's really

00:33:40.122 --> 00:33:40.962
difficult to do something

00:33:40.982 --> 00:33:42.683
with I mean a very simple

00:33:42.903 --> 00:33:43.744
approach that's worked in

00:33:43.765 --> 00:33:45.385
the past is just you have a bastion

00:33:47.196 --> 00:33:48.597
You have a bastion and

00:33:49.178 --> 00:33:51.338
that's where you have your configuration.

00:33:51.940 --> 00:33:53.319
So whether or not you want

00:33:53.339 --> 00:33:54.260
to do it with Ansible,

00:33:54.461 --> 00:33:55.642
it's essentially a single

00:33:55.682 --> 00:33:56.961
change on the bastion to

00:33:57.022 --> 00:33:57.883
lock somebody out.

00:33:59.104 --> 00:33:59.824
And then obviously if you're

00:33:59.844 --> 00:34:00.964
using any cloud provider,

00:34:01.144 --> 00:34:03.726
it's always a one-click to revoke like...

00:34:06.848 --> 00:34:08.871
You can use IAM groups.

00:34:09.112 --> 00:34:09.431
To be fair,

00:34:09.452 --> 00:34:12.876
you can also actually SSO your

00:34:12.896 --> 00:34:14.518
key management from a cloud

00:34:14.539 --> 00:34:15.659
provider and then extend

00:34:15.679 --> 00:34:18.384
that even to bare metal quite trivially.

00:34:18.503 --> 00:34:18.824
Again,

00:34:19.706 --> 00:34:21.407
a good way of potentially doing that

00:34:21.467 --> 00:34:23.690
is a bastion host that is within...

00:34:27.106 --> 00:34:28.847
If your physical

00:34:28.907 --> 00:34:29.847
infrastructure is not in a

00:34:29.889 --> 00:34:32.329
situation where you would

00:34:32.369 --> 00:34:34.231
be prevented from putting

00:34:34.311 --> 00:34:35.672
it in a virtual network

00:34:35.873 --> 00:34:37.134
with something on a cloud host,

00:34:37.173 --> 00:34:38.715
you could basically have a really,

00:34:38.735 --> 00:34:41.277
really tiny VM in a cloud

00:34:41.317 --> 00:34:42.858
host managed by key

00:34:42.898 --> 00:34:43.918
management from the cloud

00:34:43.958 --> 00:34:46.181
host and then do a similar thing.

00:34:46.221 --> 00:34:46.360
I mean,

00:34:46.380 --> 00:34:47.362
there's a whole bunch of ways to do this,

00:34:47.402 --> 00:34:47.521
right?

00:34:49.288 --> 00:34:50.048
Yeah, which I mean,

00:34:50.088 --> 00:34:51.150
it's basically what we do.

00:34:51.469 --> 00:34:52.150
You just shut the bash

00:34:52.210 --> 00:34:53.291
because we don't have any

00:34:53.391 --> 00:34:54.632
SSH ports exposed.

00:34:54.652 --> 00:34:56.014
So you'd have to be you have

00:34:56.034 --> 00:34:56.914
to be in the front.

00:34:56.934 --> 00:34:57.735
You have to be through your

00:34:57.775 --> 00:34:59.416
gate before you could

00:34:59.456 --> 00:35:00.657
actually even get to the

00:35:00.697 --> 00:35:01.617
point of using the key.

00:35:02.739 --> 00:35:03.719
I looked at what I did look

00:35:03.760 --> 00:35:04.320
at a while ago.

00:35:04.340 --> 00:35:05.400
I don't know how around this topic,

00:35:05.440 --> 00:35:07.503
but I looked at which I

00:35:07.563 --> 00:35:08.523
might do depending on what

00:35:08.543 --> 00:35:09.264
we do with our ISO.

00:35:09.304 --> 00:35:10.385
But when I looked at like

00:35:10.425 --> 00:35:11.867
JumpCloud or something similar to that,

00:35:11.907 --> 00:35:12.106
where.

00:35:13.021 --> 00:35:13.922
Where number one,

00:35:13.961 --> 00:35:15.543
there's an auth layer that

00:35:15.563 --> 00:35:17.364
you have to get through number one first.

00:35:17.465 --> 00:35:18.264
And then there's also two

00:35:18.304 --> 00:35:19.646
factor associated that auth layer.

00:35:20.007 --> 00:35:20.766
And you can build in like

00:35:20.806 --> 00:35:21.387
timeouts of that.

00:35:21.427 --> 00:35:22.369
And then from there,

00:35:22.688 --> 00:35:23.670
you basically have a token

00:35:23.690 --> 00:35:24.891
that has a limited use that

00:35:24.911 --> 00:35:25.931
you could use it associated

00:35:25.951 --> 00:35:26.731
to specific hosts.

00:35:27.132 --> 00:35:28.032
And you can build multiple

00:35:28.072 --> 00:35:29.153
different role-based access

00:35:29.173 --> 00:35:31.094
controls to say with this user,

00:35:31.135 --> 00:35:32.155
this user has access to

00:35:32.175 --> 00:35:33.277
this IP space or these

00:35:33.317 --> 00:35:34.157
hosts or whatever else.

00:35:34.677 --> 00:35:35.577
And then you have like a,

00:35:35.597 --> 00:35:36.657
it's basically like an LDAP

00:35:36.797 --> 00:35:37.318
type structure.

00:35:37.358 --> 00:35:38.918
And so you, you sign with LDAP,

00:35:38.958 --> 00:35:40.398
you use like duo or a different,

00:35:40.659 --> 00:35:41.898
like two factory, you get a push.

00:35:42.260 --> 00:35:43.219
And then from there you have a,

00:35:43.340 --> 00:35:44.119
you have a token that you

00:35:44.139 --> 00:35:46.360
can use that's on, on, on servers.

00:35:47.320 --> 00:35:48.340
Now, do you need, like I said,

00:35:48.360 --> 00:35:49.001
do you need on every

00:35:49.021 --> 00:35:49.721
fucking server though?

00:35:50.300 --> 00:35:51.181
I don't know.

00:35:51.201 --> 00:35:51.780
It's quite high.

00:35:51.880 --> 00:35:52.581
There's some stuff,

00:35:52.782 --> 00:35:53.601
there's some stuff like I

00:35:53.621 --> 00:35:54.862
would like Horcrux nodes or

00:35:55.382 --> 00:35:57.061
fricking Aptos and things like that where,

00:35:57.262 --> 00:35:59.362
where you don't have like

00:35:59.463 --> 00:36:00.603
there's keys in the wild and,

00:36:00.922 --> 00:36:02.885
So it's a pure, pure,

00:36:03.065 --> 00:36:04.786
pure bull market vibes.

00:36:05.407 --> 00:36:07.150
This is going back a little while now.

00:36:10.472 --> 00:36:10.632
Oh no,

00:36:10.672 --> 00:36:11.853
we have one host where this is still

00:36:11.873 --> 00:36:12.235
true of,

00:36:12.295 --> 00:36:14.396
but back in the day we used to

00:36:14.436 --> 00:36:16.898
have every single network

00:36:17.420 --> 00:36:19.101
isolated from each other.

00:36:20.070 --> 00:36:24.231
all with essentially

00:36:24.271 --> 00:36:25.012
different two-factor

00:36:25.032 --> 00:36:26.793
authentication configured

00:36:26.833 --> 00:36:27.833
when the hosts were brought up.

00:36:28.413 --> 00:36:30.733
So you had to basically,

00:36:30.873 --> 00:36:32.094
as part of the bootstrapping,

00:36:32.114 --> 00:36:34.414
it would fall out of the automatic bit,

00:36:34.954 --> 00:36:36.054
give you the recovery,

00:36:36.155 --> 00:36:37.295
and then re-enter it.

00:36:41.956 --> 00:36:44.396
And then, obviously, any additional user,

00:36:44.416 --> 00:36:45.237
because it was using a

00:36:45.557 --> 00:36:46.458
different daemon for that.

00:36:47.614 --> 00:36:48.934
you'd have to add them and

00:36:48.954 --> 00:36:51.574
then also give them a

00:36:51.614 --> 00:36:53.076
what's it called a two fa

00:36:53.835 --> 00:36:54.576
thing and then you'd have

00:36:54.615 --> 00:36:57.137
to add that back in yeah so

00:36:57.177 --> 00:36:58.356
that was pretty good and

00:36:58.396 --> 00:36:59.436
but then it was it was one

00:36:59.476 --> 00:37:00.418
of those where I was like

00:37:00.538 --> 00:37:02.077
ah this is this is amazing

00:37:02.097 --> 00:37:03.478
if we hire and then we

00:37:03.539 --> 00:37:05.559
ended up covering

00:37:05.639 --> 00:37:06.699
everything we needed to do

00:37:07.420 --> 00:37:09.139
with the odd freelance bit

00:37:09.159 --> 00:37:10.141
of extra work here there

00:37:10.161 --> 00:37:11.121
and everywhere and

00:37:13.596 --> 00:37:15.257
number of times these boxes

00:37:15.277 --> 00:37:19.282
needed to be SSH'd into was... Yeah.

00:37:19.583 --> 00:37:23.387
I don't really go into too many notes.

00:37:23.407 --> 00:37:24.668
What I would like to do, though,

00:37:25.148 --> 00:37:27.731
is I'd like to figure out a

00:37:27.771 --> 00:37:29.635
decent vault structure that... I mean,

00:37:29.655 --> 00:37:30.235
we use vaults like

00:37:30.275 --> 00:37:31.076
Enhanceable and things like that,

00:37:31.096 --> 00:37:32.597
but I'd really like to do a

00:37:33.097 --> 00:37:33.858
public cloud vault

00:37:34.740 --> 00:37:36.782
piece to pull down specific

00:37:36.822 --> 00:37:39.923
keys at runtime or a

00:37:39.963 --> 00:37:41.143
specific credit runtime.

00:37:41.764 --> 00:37:42.885
And then like, like,

00:37:43.184 --> 00:37:45.085
like we're doing stuff with SSV, right?

00:37:45.726 --> 00:37:48.126
Like those SSV node keys are

00:37:48.288 --> 00:37:48.827
on the machine.

00:37:49.268 --> 00:37:50.889
And so it would be great to

00:37:50.949 --> 00:37:52.489
have some of these tools

00:37:52.530 --> 00:37:54.110
allow it where I can either

00:37:54.150 --> 00:37:55.530
call a vault directly off that tool,

00:37:55.590 --> 00:37:56.711
or I can pre-run that

00:37:56.791 --> 00:37:57.692
through like through Docker,

00:37:58.132 --> 00:37:59.193
grab that structure

00:37:59.833 --> 00:38:00.893
and then store it and then

00:38:00.972 --> 00:38:02.213
read it basically off of memory.

00:38:02.253 --> 00:38:03.134
And then once, if the,

00:38:03.233 --> 00:38:06.054
if somebody gets in the machine or if,

00:38:06.755 --> 00:38:07.675
if it reboots or if

00:38:07.735 --> 00:38:08.715
somebody takes a drive out

00:38:08.735 --> 00:38:09.876
of it or something else like that,

00:38:09.896 --> 00:38:10.956
which if it's not,

00:38:10.976 --> 00:38:12.496
whether it's in our custody or not,

00:38:13.097 --> 00:38:13.617
it's not there,

00:38:14.056 --> 00:38:15.056
like it's off the machine.

00:38:15.097 --> 00:38:17.318
So Aptos is a good example of that too,

00:38:17.338 --> 00:38:18.398
because those keys are literally,

00:38:18.557 --> 00:38:19.699
you have to have them on the machine.

00:38:20.398 --> 00:38:23.099
And then, so having that be, you know,

00:38:23.139 --> 00:38:23.500
have that be,

00:38:24.405 --> 00:38:26.246
in memory and carry that

00:38:26.286 --> 00:38:27.246
forward and then be able to

00:38:27.286 --> 00:38:28.527
pull that down and even,

00:38:28.606 --> 00:38:29.728
even like pull it out of a vault.

00:38:30.228 --> 00:38:31.128
And then if the machine

00:38:31.148 --> 00:38:32.947
reboots or if I have to run it, then it,

00:38:33.027 --> 00:38:33.648
it does like a push

00:38:33.688 --> 00:38:35.369
notification to me and I can approve that,

00:38:35.989 --> 00:38:36.949
um, or something similar to that.

00:38:36.969 --> 00:38:38.409
It's like this machine is requesting this,

00:38:38.909 --> 00:38:39.429
the secret.

00:38:39.849 --> 00:38:41.090
Um, I want to get to that point.

00:38:41.130 --> 00:38:41.951
So I've been looking at it a

00:38:41.971 --> 00:38:42.610
little bit here and there.

00:38:42.630 --> 00:38:43.391
I just don't have the time

00:38:43.431 --> 00:38:45.112
since everything else is

00:38:45.152 --> 00:38:46.052
exploding at the same time,

00:38:46.692 --> 00:38:47.293
but that would be cool.

00:38:47.393 --> 00:38:48.012
I'd like to do that.

00:38:48.773 --> 00:38:48.913
Um,

00:38:49.943 --> 00:38:50.824
it'd be great to get some

00:38:50.844 --> 00:38:53.047
things like out of, out of repos,

00:38:53.086 --> 00:38:54.829
even if they're encrypted and, and get,

00:38:55.570 --> 00:38:56.150
get them a different

00:38:56.190 --> 00:38:56.971
structure where I can,

00:38:57.110 --> 00:38:59.273
I can call into that vault with a key.

00:38:59.313 --> 00:38:59.775
Well, I think it's,

00:39:00.014 --> 00:39:01.056
but it's also like anything else,

00:39:01.096 --> 00:39:01.376
isn't it?

00:39:01.396 --> 00:39:03.518
There's like anything that

00:39:03.559 --> 00:39:04.260
involves a certain amount

00:39:04.280 --> 00:39:05.460
of time has to be a profit

00:39:05.481 --> 00:39:06.242
motive or whatever.

00:39:06.481 --> 00:39:07.804
And like the, yeah.

00:39:08.226 --> 00:39:09.827
the bull market energy when, when,

00:39:09.847 --> 00:39:11.047
you know, we were like, oh yeah,

00:39:11.086 --> 00:39:12.286
everything has to be two to

00:39:12.327 --> 00:39:13.588
a fade or whatever and blah, blah, blah,

00:39:13.608 --> 00:39:13.927
blah.

00:39:13.947 --> 00:39:14.188
You know,

00:39:14.208 --> 00:39:16.849
like we're making money every month,

00:39:17.188 --> 00:39:19.148
you know, in cosmos, you know,

00:39:19.168 --> 00:39:20.170
we're losing money in cosmos now.

00:39:20.210 --> 00:39:20.989
That's why we're coming off

00:39:21.070 --> 00:39:22.889
on coming off of our last networks.

00:39:22.931 --> 00:39:23.170
Right.

00:39:23.811 --> 00:39:24.331
And, um,

00:39:26.523 --> 00:39:27.945
Yeah,

00:39:27.985 --> 00:39:30.327
I think if we were doing more in the

00:39:30.387 --> 00:39:31.369
custodial space,

00:39:31.409 --> 00:39:34.652
then I think my... Oh yeah,

00:39:34.672 --> 00:39:35.353
that'd be totally different.

00:39:35.472 --> 00:39:35.913
Yeah, exactly.

00:39:36.132 --> 00:39:38.175
My threat profiling and all

00:39:38.195 --> 00:39:39.436
that stuff would be

00:39:39.476 --> 00:39:41.438
completely out of here.

00:39:42.259 --> 00:39:44.880
So like there are, yeah, exactly.

00:39:44.900 --> 00:39:45.840
Like this kind of structures

00:39:45.860 --> 00:39:47.581
you're talking about are

00:39:47.621 --> 00:39:49.503
much more appropriate in terms of... Yeah,

00:39:49.523 --> 00:39:50.202
there's not much loss of

00:39:50.242 --> 00:39:51.744
funds issues that we have

00:39:51.764 --> 00:39:52.264
with that stuff.

00:39:52.284 --> 00:39:52.764
You know what I mean?

00:39:52.943 --> 00:39:54.005
Like double, I mean,

00:39:54.025 --> 00:39:54.664
the big thing would be like

00:39:54.824 --> 00:39:55.726
keys are supposed to do a

00:39:55.746 --> 00:39:56.726
val on double signing and

00:39:56.786 --> 00:39:57.527
us getting slashed.

00:39:57.567 --> 00:39:58.706
But now most of these

00:39:58.726 --> 00:39:59.648
networks are even reducing

00:39:59.668 --> 00:40:00.527
that slashing amount.

00:40:01.748 --> 00:40:03.028
Yeah, and I just,

00:40:03.568 --> 00:40:06.831
I'm not sure that... It's difficult,

00:40:06.891 --> 00:40:07.070
right?

00:40:07.110 --> 00:40:10.452
Because, you know, slashing is...

00:40:15.202 --> 00:40:17.623
the current leading solution

00:40:17.864 --> 00:40:19.204
to the nothing at stake problem,

00:40:22.007 --> 00:40:24.329
but it's not really,

00:40:24.829 --> 00:40:26.532
it's not really clear that it solves it,

00:40:27.251 --> 00:40:28.934
but it is clear that it, well,

00:40:29.034 --> 00:40:30.215
it is likely that if you

00:40:30.335 --> 00:40:31.275
don't do anything,

00:40:31.856 --> 00:40:32.916
eventually you will get

00:40:32.956 --> 00:40:34.097
Byzantine behavior.

00:40:34.157 --> 00:40:35.699
And the only other competing

00:40:36.219 --> 00:40:38.942
sort of design for solving

00:40:39.021 --> 00:40:42.405
nothing at stake is simply

00:40:42.465 --> 00:40:43.545
to make your blockchain

00:40:45.699 --> 00:40:48.802
not decentralized right okay

00:40:49.063 --> 00:40:50.583
um so what am I talking

00:40:50.603 --> 00:40:51.565
about here right so nothing

00:40:51.605 --> 00:40:52.585
I think at stake is where

00:40:52.726 --> 00:40:53.646
if there's two versions of

00:40:53.666 --> 00:40:54.768
the truth you just sign

00:40:54.807 --> 00:40:55.628
both of them and you move

00:40:55.708 --> 00:40:57.349
on right because if there's

00:40:57.389 --> 00:40:58.490
no slashing who gives a

00:40:58.530 --> 00:40:59.711
right that's that's what

00:40:59.751 --> 00:41:00.472
slashing is meant to

00:41:00.492 --> 00:41:03.355
prevent right um in the

00:41:03.394 --> 00:41:05.376
event where you just say we

00:41:05.396 --> 00:41:07.097
are the foundation we control the chain

00:41:08.126 --> 00:41:09.146
you don't need to worry

00:41:09.327 --> 00:41:11.449
about nothing to stake, right?

00:41:11.489 --> 00:41:13.210
Because if a validator is malicious,

00:41:13.311 --> 00:41:15.432
does anything by Zantina, anybody notices,

00:41:15.532 --> 00:41:16.454
you just take away the

00:41:16.494 --> 00:41:17.554
foundation delegation and

00:41:17.594 --> 00:41:18.675
they fall out set and

00:41:18.695 --> 00:41:21.518
basically nobody gives a shit.

00:41:21.559 --> 00:41:21.719
Right?

00:41:22.119 --> 00:41:22.239
So,

00:41:24.713 --> 00:41:26.994
In any other situation other than that,

00:41:27.956 --> 00:41:29.797
the economic incentives say

00:41:29.878 --> 00:41:30.858
that if there is a

00:41:30.938 --> 00:41:32.380
technical means by which a

00:41:32.400 --> 00:41:34.322
validator would be able to

00:41:34.402 --> 00:41:35.463
potentially sign two

00:41:35.483 --> 00:41:36.182
versions of the truth,

00:41:37.364 --> 00:41:40.047
they'll be incentivized to

00:41:40.067 --> 00:41:42.248
do so and obviously potentially

00:41:43.429 --> 00:41:45.251
Byzantine Lee attack the chain.

00:41:45.291 --> 00:41:46.472
If there were some profit,

00:41:46.492 --> 00:41:49.813
profit motive for altering, you know, the,

00:41:49.873 --> 00:41:50.253
the truth,

00:41:50.333 --> 00:41:51.434
if they would say the proposer

00:41:51.454 --> 00:41:52.054
or something like that,

00:41:52.094 --> 00:41:52.914
obviously it depends on the

00:41:52.936 --> 00:41:55.336
P to P mechanism being used.

00:41:55.356 --> 00:41:57.018
Cause they're all slightly different.

00:41:57.057 --> 00:41:58.559
Like the vector by which you

00:41:58.599 --> 00:42:00.480
could attack the chain is

00:42:00.519 --> 00:42:02.501
different in cosmos versus Ethereum.

00:42:02.541 --> 00:42:04.621
And a lot of the theory,

00:42:04.681 --> 00:42:05.742
if you like of this came

00:42:05.822 --> 00:42:07.123
out of the Ethereum space

00:42:07.143 --> 00:42:07.963
is my understanding.

00:42:07.983 --> 00:42:08.704
So yeah,

00:42:10.858 --> 00:42:11.199
Yeah,

00:42:11.619 --> 00:42:14.059
the only solutions to this that don't

00:42:14.119 --> 00:42:15.539
involve any kind of

00:42:15.820 --> 00:42:20.942
slashing seem to have come

00:42:21.242 --> 00:42:22.422
in situations where it's

00:42:22.442 --> 00:42:24.202
simply not been implemented.

00:42:24.802 --> 00:42:27.164
And in most of those, as far as I can tell,

00:42:27.304 --> 00:42:30.324
that's because either the

00:42:30.443 --> 00:42:32.304
amount you need to bond to

00:42:32.364 --> 00:42:33.925
put a validator into the

00:42:33.965 --> 00:42:37.266
set is so high that

00:42:37.326 --> 00:42:38.967
Byzantine behavior is...

00:42:39.974 --> 00:42:42.755
is like it is by definition

00:42:43.597 --> 00:42:46.838
um disincentivized very

00:42:46.878 --> 00:42:52.041
very heavily um or or and

00:42:52.101 --> 00:42:53.960
additionally the foundation

00:42:54.001 --> 00:42:56.762
can actually pull levers to

00:42:56.862 --> 00:42:57.963
get validates out of the

00:42:57.983 --> 00:42:59.364
set because they control

00:43:00.204 --> 00:43:01.605
the majority of the tokens still

00:43:03.902 --> 00:43:05.282
Because if you look at actually,

00:43:05.782 --> 00:43:07.164
like Bitcoin's a good example, right,

00:43:07.204 --> 00:43:08.264
where even though it's

00:43:08.304 --> 00:43:09.105
really expensive to

00:43:09.164 --> 00:43:10.764
actually run a validator, sorry,

00:43:10.804 --> 00:43:12.126
run a miner that can still

00:43:12.326 --> 00:43:13.686
keep up and actually be

00:43:13.726 --> 00:43:16.228
part of a pool that makes money,

00:43:17.389 --> 00:43:18.188
those mining pools are

00:43:18.208 --> 00:43:20.010
still incentivized to do

00:43:20.170 --> 00:43:23.211
everything under their power to mess with

00:43:25.326 --> 00:43:27.288
you know, the truth essentially, right?

00:43:27.768 --> 00:43:30.811
So in every situation where

00:43:30.831 --> 00:43:31.893
you don't have a foundation

00:43:31.954 --> 00:43:34.277
able to coordinate and pull stake,

00:43:35.898 --> 00:43:36.920
essentially like defeating

00:43:36.960 --> 00:43:38.320
decentralization that way,

00:43:39.762 --> 00:43:40.864
you need an additional

00:43:40.945 --> 00:43:42.606
economic incentive to stop

00:43:44.105 --> 00:43:44.525
um,

00:43:44.565 --> 00:43:48.146
validators behaving Byzantine Lee is the,

00:43:48.186 --> 00:43:49.706
the theory of nothing at stake thing.

00:43:50.047 --> 00:43:51.088
And I'm not a hundred

00:43:51.108 --> 00:43:51.987
percent aware that there is

00:43:52.007 --> 00:43:53.748
another good solution to it,

00:43:54.128 --> 00:43:55.989
but I do know that quite a

00:43:56.009 --> 00:43:57.429
number of networks seem to just be like,

00:43:57.489 --> 00:43:57.969
ah, Yolo.

00:43:57.989 --> 00:43:58.110
Yeah.

00:43:59.590 --> 00:44:01.411
That's what most do, right?

00:44:02.271 --> 00:44:03.492
Yeah.

00:44:03.572 --> 00:44:07.273
I mean, that's exactly what it is.

00:44:07.353 --> 00:44:08.153
Yeah, it is what it is.

00:44:08.173 --> 00:44:08.893
Um, yeah.

00:44:10.659 --> 00:44:11.802
but hey um it would be

00:44:11.822 --> 00:44:13.163
interesting I mean it's one

00:44:13.182 --> 00:44:13.784
of those things where

00:44:13.804 --> 00:44:14.824
you're like it would be

00:44:14.905 --> 00:44:15.666
interesting if somebody

00:44:15.706 --> 00:44:16.668
comes up with an elegant

00:44:16.708 --> 00:44:19.150
solution to that um be

00:44:19.271 --> 00:44:19.811
interesting from an

00:44:19.851 --> 00:44:20.793
economics perspective

00:44:20.952 --> 00:44:21.474
interesting from a

00:44:21.514 --> 00:44:23.036
technical perspective um

00:44:24.871 --> 00:44:27.813
It doesn't sound like

00:44:27.833 --> 00:44:28.634
Informal's working on it,

00:44:28.653 --> 00:44:29.894
so it might be something I

00:44:29.914 --> 00:44:30.355
have to step in.

00:44:30.375 --> 00:44:34.378
Yeah, they're just like, oh,

00:44:34.398 --> 00:44:35.217
that sounds tough.

00:44:38.280 --> 00:44:38.760
That's a tough one.

00:44:38.800 --> 00:44:39.742
We'll give that one a miss, lads.

00:44:39.762 --> 00:44:41.382
Let's do this dead thing.

00:44:42.708 --> 00:44:44.809
So I'm just noticing we obviously,

00:44:44.829 --> 00:44:45.728
we had some comments that

00:44:45.748 --> 00:44:47.909
we skipped over when we

00:44:47.929 --> 00:44:49.630
were talking about cycles

00:44:49.791 --> 00:44:53.492
and informal Rama said

00:44:54.331 --> 00:44:56.353
cycles is Ethan socialism

00:44:56.373 --> 00:44:57.532
for blockchain project.

00:44:58.974 --> 00:44:59.673
Great hot take.

00:45:01.235 --> 00:45:03.615
Lil D says cycles doesn't

00:45:03.635 --> 00:45:04.376
make sense to me.

00:45:04.416 --> 00:45:05.675
The idea of applying graph

00:45:05.715 --> 00:45:07.036
theory to payments on

00:45:07.137 --> 00:45:08.797
network to figure out

00:45:08.876 --> 00:45:10.358
settlement patterns is cool,

00:45:11.596 --> 00:45:12.797
but does any chain have the

00:45:12.856 --> 00:45:14.838
necessary utilization for it?

00:45:15.458 --> 00:45:15.978
Oof.

00:45:17.719 --> 00:45:19.159
I've just, it's a murder.

00:45:22.820 --> 00:45:24.202
And also, actually, to be fair,

00:45:24.222 --> 00:45:25.922
that will be also said later.

00:45:26.742 --> 00:45:27.364
I think there's probably

00:45:27.384 --> 00:45:28.284
some important reasons why

00:45:28.344 --> 00:45:29.664
mutual credit systems

00:45:29.804 --> 00:45:31.166
haven't really worked.

00:45:31.746 --> 00:45:32.505
There probably are.

00:45:33.226 --> 00:45:35.407
Yeah, there's some important ones.

00:45:36.688 --> 00:45:38.748
Technology is not necessarily one of those,

00:45:38.809 --> 00:45:40.030
but we'll find out, I guess.

00:45:41.380 --> 00:45:41.619
Yeah.

00:45:41.679 --> 00:45:42.161
I mean, well,

00:45:42.240 --> 00:45:43.902
I suppose the flip side of that was,

00:45:44.021 --> 00:45:44.202
you know,

00:45:44.222 --> 00:45:45.943
we talked about it literally

00:45:46.402 --> 00:45:47.844
eight months ago on the pod.

00:45:47.903 --> 00:45:49.125
Oh, like, um, you know,

00:45:49.184 --> 00:45:53.847
in Graber's book on debt, debt, um,

00:45:53.867 --> 00:45:56.449
there's a whole section on, uh, how, um,

00:45:56.710 --> 00:45:58.351
in medieval England and

00:45:58.431 --> 00:46:00.773
elsewhere tallies were used, um,

00:46:01.612 --> 00:46:03.434
for debt recording.

00:46:03.715 --> 00:46:04.175
And, um,

00:46:05.722 --> 00:46:06.884
I guess it comes down to a

00:46:06.923 --> 00:46:08.164
philosophical question, right?

00:46:08.244 --> 00:46:10.164
When people accept the tally

00:46:10.204 --> 00:46:11.346
of a debt as a

00:46:11.405 --> 00:46:12.766
representation of a debt

00:46:12.786 --> 00:46:14.306
that can be passed on as a

00:46:14.427 --> 00:46:17.748
multi-party debt or a multi-party credit,

00:46:17.768 --> 00:46:19.550
I suppose, as much as a multi-party debt,

00:46:21.710 --> 00:46:26.514
is that credit, what's being,

00:46:26.733 --> 00:46:28.074
what's actually happening, right?

00:46:28.434 --> 00:46:29.195
Is that credit,

00:46:29.295 --> 00:46:30.414
is there a credit in a debt

00:46:30.434 --> 00:46:32.936
there or is it the token that's the,

00:46:33.237 --> 00:46:34.336
is it the representation

00:46:34.376 --> 00:46:35.137
that's the important?

00:46:36.414 --> 00:46:38.155
thing because the debt itself is, is a,

00:46:38.536 --> 00:46:41.338
is a, um, intangible obligation.

00:46:42.798 --> 00:46:45.099
Um,

00:46:45.159 --> 00:46:46.440
and I realized there's probably some

00:46:46.481 --> 00:46:47.521
connection between chosen

00:46:47.561 --> 00:46:49.643
possession and chosen action here, but I,

00:46:51.125 --> 00:46:51.425
I it's,

00:46:51.525 --> 00:46:52.706
it's late here and I lack the

00:46:52.746 --> 00:46:54.226
brain power to make the connection.

00:46:54.246 --> 00:46:57.148
If there is one, um,

00:46:57.208 --> 00:46:57.869
I guess it's like the

00:46:57.909 --> 00:46:58.951
ability to action it.

00:46:59.010 --> 00:46:59.271
Right.

00:46:59.371 --> 00:47:00.731
And the, the manifest,

00:47:00.751 --> 00:47:01.813
the physical manifestation

00:47:01.833 --> 00:47:03.434
is about being able to

00:47:03.773 --> 00:47:04.715
action it in the future.

00:47:10.284 --> 00:47:10.643
I don't know.

00:47:10.684 --> 00:47:12.065
Man, payments is complicated.

00:47:12.144 --> 00:47:13.326
As soon as you start asking

00:47:13.525 --> 00:47:15.186
what are payments and what are debts,

00:47:15.887 --> 00:47:16.748
you know, it's

00:47:18.177 --> 00:47:22.739
You should do that for your doctorate.

00:47:22.759 --> 00:47:23.960
I tried to write the word

00:47:24.019 --> 00:47:25.081
debt at one point,

00:47:25.380 --> 00:47:28.983
and it made a whole... So I

00:47:29.023 --> 00:47:31.403
made the mistake, right?

00:47:31.704 --> 00:47:32.684
I did a presentation,

00:47:32.925 --> 00:47:34.065
and I made the mistake of

00:47:34.146 --> 00:47:35.505
putting credit with a

00:47:35.545 --> 00:47:38.668
capital C on a slide.

00:47:39.527 --> 00:47:39.789
Yeah.

00:47:39.969 --> 00:47:40.168
And...

00:47:42.030 --> 00:47:44.733
And then I got hit with, I,

00:47:44.753 --> 00:47:45.873
in the question section,

00:47:46.014 --> 00:47:46.855
I got hit with the

00:47:46.914 --> 00:47:50.637
questions about how credit

00:47:50.697 --> 00:47:52.679
would work in the situation

00:47:52.699 --> 00:47:54.039
that I had outlined.

00:47:54.239 --> 00:47:55.900
And I was like, oh no, no, no.

00:47:55.920 --> 00:47:57.081
Like we're talking about a

00:47:57.141 --> 00:48:00.184
really simple model for, for multi-party

00:48:01.596 --> 00:48:02.797
multi-party credit like a

00:48:02.976 --> 00:48:04.117
really really simple model

00:48:04.737 --> 00:48:06.079
yeah like we put it with a

00:48:06.119 --> 00:48:07.659
capital c that means it has

00:48:07.739 --> 00:48:09.442
the implications of and I

00:48:09.461 --> 00:48:11.003
was like oh damn okay well

00:48:11.043 --> 00:48:11.782
what's the question and

00:48:11.802 --> 00:48:12.784
then they just hit me with

00:48:12.804 --> 00:48:15.286
this really clever like

00:48:15.726 --> 00:48:17.246
well worked out detailed

00:48:17.306 --> 00:48:18.228
just like just off the top

00:48:18.248 --> 00:48:18.708
of their head they were

00:48:18.728 --> 00:48:19.327
like well because this

00:48:19.347 --> 00:48:20.268
would then mean that and

00:48:20.309 --> 00:48:22.230
then that so then I mean

00:48:22.250 --> 00:48:23.972
that's really like I just

00:48:23.992 --> 00:48:24.771
don't see how it work and

00:48:24.791 --> 00:48:26.373
I'm just like oh my god

00:48:26.592 --> 00:48:28.614
like stop stop stop hitting

00:48:28.635 --> 00:48:30.076
me that's awful

00:48:31.226 --> 00:48:33.186
Yeah, I was just like,

00:48:33.246 --> 00:48:33.907
I don't even think they

00:48:33.927 --> 00:48:35.708
were trying to be like, oh, actually,

00:48:35.827 --> 00:48:36.728
I don't even think they

00:48:36.748 --> 00:48:37.548
were trying to own me.

00:48:37.568 --> 00:48:38.809
They just were asking like a

00:48:38.849 --> 00:48:41.130
legitimate question,

00:48:41.170 --> 00:48:42.731
but just knew way more than me.

00:48:42.831 --> 00:48:44.391
And I just got owned in

00:48:44.411 --> 00:48:45.512
front of a room of people.

00:48:45.632 --> 00:48:46.311
It was just like,

00:48:47.152 --> 00:48:48.813
I literally have no idea

00:48:48.853 --> 00:48:50.733
how to even begin answering

00:48:50.773 --> 00:48:51.775
what you just said.

00:48:51.934 --> 00:48:52.135
Yeah.

00:48:56.911 --> 00:48:59.134
Yeah, but the capital C thing.

00:48:59.594 --> 00:49:01.315
But maybe I should have.

00:49:01.416 --> 00:49:02.936
I think literally my answer

00:49:02.976 --> 00:49:04.797
was I should probably go

00:49:04.838 --> 00:49:06.059
back and change the slide

00:49:06.079 --> 00:49:07.260
to be a small C. Oh, shit.

00:49:07.380 --> 00:49:09.081
Yeah, exactly.

00:49:10.702 --> 00:49:11.202
Oh, God.

00:49:11.382 --> 00:49:11.724
Yeah.

00:49:11.764 --> 00:49:12.945
This is probably why people

00:49:12.965 --> 00:49:15.306
don't talk to me in the

00:49:15.327 --> 00:49:15.907
department anymore.

00:49:15.927 --> 00:49:20.090
Oh, man.

00:49:20.871 --> 00:49:21.130
Yeah.

00:49:21.190 --> 00:49:22.172
So what else is up?

00:49:23.655 --> 00:49:24.155
What's happening?

00:49:24.556 --> 00:49:26.759
Well, you had a big outage, didn't you?

00:49:26.778 --> 00:49:27.820
We did, yeah.

00:49:27.840 --> 00:49:28.481
Your tie boy?

00:49:29.141 --> 00:49:31.264
Yeah, we did.

00:49:31.324 --> 00:49:34.268
So we have a couple data centers in the US,

00:49:34.570 --> 00:49:39.155
and then our largest one is like,

00:49:39.295 --> 00:49:40.496
it's a waste for me.

00:49:40.556 --> 00:49:40.697
It's

00:49:41.583 --> 00:49:43.826
Is it just downhill from a

00:49:43.925 --> 00:49:45.387
swimming pool somewhere in Britain?

00:49:45.407 --> 00:49:48.090
Just go in your car immediately.

00:49:49.291 --> 00:49:50.132
Dude, that sounds awful.

00:49:50.534 --> 00:49:50.833
It's like a

00:49:50.914 --> 00:49:53.077
five-and-a-half-hour drive for me.

00:49:53.876 --> 00:49:54.378
Oh, nice.

00:49:54.458 --> 00:49:54.978
Something like that,

00:49:55.039 --> 00:49:55.920
which is like your backyard.

00:49:55.980 --> 00:49:56.159
I know,

00:49:56.199 --> 00:49:58.543
but... The fucking drive to the

00:49:58.583 --> 00:49:59.603
mailbox around here.

00:49:59.623 --> 00:50:00.264
I was going to say,

00:50:00.284 --> 00:50:01.425
not on Niles' property.

00:50:02.686 --> 00:50:02.867
Yeah.

00:50:02.887 --> 00:50:04.068
Just passing the swimming pool.

00:50:04.628 --> 00:50:05.588
So Monday, it was about now.

00:50:05.608 --> 00:50:06.489
It was like six o'clock or

00:50:06.528 --> 00:50:06.969
something Monday.

00:50:06.989 --> 00:50:08.369
I'm just hanging out with my feet up,

00:50:08.650 --> 00:50:09.451
you know, watching shit.

00:50:09.471 --> 00:50:10.391
I was watching some logs and

00:50:10.411 --> 00:50:10.931
some stuff and then

00:50:10.951 --> 00:50:11.851
everything just halted.

00:50:12.771 --> 00:50:13.552
And all the,

00:50:16.233 --> 00:50:18.815
like I lost my VPN into there

00:50:18.835 --> 00:50:20.755
and like all our monitoring

00:50:20.775 --> 00:50:21.577
is in there as well.

00:50:21.677 --> 00:50:24.117
So I got some alerts from

00:50:24.217 --> 00:50:25.418
external monitors that we have,

00:50:25.438 --> 00:50:26.298
but all our internal

00:50:26.338 --> 00:50:27.639
monitors were not firing.

00:50:27.659 --> 00:50:29.440
And then I realized that basically, yeah,

00:50:30.356 --> 00:50:32.157
that the data center was down.

00:50:33.697 --> 00:50:36.458
And basic, like it just started to,

00:50:36.478 --> 00:50:37.518
we started to get a good

00:50:37.538 --> 00:50:38.659
understanding what the fuck was going on.

00:50:39.398 --> 00:50:42.179
And so my partner who is

00:50:42.260 --> 00:50:43.000
about twenty minutes from

00:50:43.019 --> 00:50:44.079
that data center was on a

00:50:44.119 --> 00:50:46.561
flight from from there to San Jose,

00:50:46.601 --> 00:50:48.521
California, which is not great.

00:50:48.561 --> 00:50:50.702
So but Delta has Wi-Fi.

00:50:50.722 --> 00:50:51.722
So I'm talking back and

00:50:51.762 --> 00:50:53.063
forth and trying to figure

00:50:53.083 --> 00:50:54.384
out exactly what the fuck is going on.

00:50:54.403 --> 00:50:55.204
And we think it might be.

00:50:56.242 --> 00:50:57.143
We were really sure what it was.

00:50:57.184 --> 00:50:58.184
It was acting very weird

00:50:59.045 --> 00:50:59.846
because we could see that

00:51:00.126 --> 00:51:01.286
that the inbound traffic

00:51:01.347 --> 00:51:02.989
went to eight hundred megabit inbound,

00:51:03.548 --> 00:51:04.469
which was really strange,

00:51:04.570 --> 00:51:05.931
even though there was nothing responding,

00:51:07.452 --> 00:51:08.693
which no, we should talk about, too,

00:51:08.733 --> 00:51:09.373
because I think I don't

00:51:09.393 --> 00:51:10.034
know what that would have been.

00:51:10.054 --> 00:51:10.735
It would have been literally

00:51:10.755 --> 00:51:12.155
like Zabbix things calling

00:51:12.195 --> 00:51:12.916
home and failing.

00:51:13.217 --> 00:51:13.918
It was really strange.

00:51:15.635 --> 00:51:16.215
So we went from like a

00:51:16.255 --> 00:51:17.818
hundred megabit inbound to eight hundred.

00:51:17.938 --> 00:51:19.159
So first thing I thought was like, Hey,

00:51:19.179 --> 00:51:20.061
we're getting DDoS or

00:51:20.101 --> 00:51:20.722
something similar to that,

00:51:20.742 --> 00:51:21.322
which it wasn't.

00:51:22.724 --> 00:51:23.485
Cause that would usually be

00:51:23.565 --> 00:51:24.047
more than that.

00:51:24.126 --> 00:51:25.208
Like eight, eight hundred megabits,

00:51:25.248 --> 00:51:27.291
not enough to saturate the

00:51:27.331 --> 00:51:28.492
gig line in this one example.

00:51:30.548 --> 00:51:31.989
So anyway, it took, it took, we were on,

00:51:32.010 --> 00:51:33.490
I was on calls with our provider.

00:51:33.530 --> 00:51:34.050
We're trying to figure out

00:51:34.110 --> 00:51:35.192
exactly what the fuck is going on.

00:51:35.231 --> 00:51:36.391
And then it basically took a

00:51:36.431 --> 00:51:37.333
few hours to realize

00:51:37.393 --> 00:51:38.233
exactly what the hell is

00:51:38.253 --> 00:51:39.914
going on because it was very difficult to,

00:51:40.514 --> 00:51:42.416
to troubleshoot remotely.

00:51:42.476 --> 00:51:44.077
And we do have an LTE backup here too.

00:51:44.117 --> 00:51:45.898
So meaning I can get into that,

00:51:46.277 --> 00:51:47.559
some parts of that stack

00:51:47.599 --> 00:51:49.300
through a cellular line

00:51:49.699 --> 00:51:50.639
that we have as a backup.

00:51:51.721 --> 00:51:52.481
And that's really for just

00:51:52.501 --> 00:51:53.481
from a management perspective,

00:51:53.501 --> 00:51:54.463
like if it's lights out,

00:51:54.963 --> 00:51:55.563
I might need to do

00:51:55.623 --> 00:51:56.884
something like I might need to,

00:51:57.523 --> 00:51:58.364
if for some reason, like,

00:51:59.440 --> 00:52:00.360
all the fibers cut to this

00:52:00.420 --> 00:52:01.641
place and I need to shut

00:52:01.681 --> 00:52:02.463
down a validator.

00:52:02.523 --> 00:52:04.403
I needed to grab keys or I

00:52:04.423 --> 00:52:05.826
need to do something that I

00:52:05.846 --> 00:52:06.525
want to make sure that when

00:52:06.565 --> 00:52:08.927
it came back up, that service was off,

00:52:09.409 --> 00:52:09.608
right?

00:52:09.688 --> 00:52:11.250
Like there's a lot of those examples,

00:52:11.291 --> 00:52:11.451
right?

00:52:11.490 --> 00:52:12.652
Like if you're not running

00:52:12.692 --> 00:52:13.492
Horcrux and things like that,

00:52:13.512 --> 00:52:14.853
but there are a lot of those situations.

00:52:14.873 --> 00:52:15.974
And so that's kind of a backup there.

00:52:15.994 --> 00:52:17.715
So we ended up what we start

00:52:17.896 --> 00:52:18.797
and nothing was routing and

00:52:18.817 --> 00:52:19.378
trying to figure out what

00:52:19.398 --> 00:52:19.878
the fuck was going on.

00:52:19.898 --> 00:52:21.378
So long story short, it took like,

00:52:21.940 --> 00:52:22.239
honestly,

00:52:22.260 --> 00:52:23.862
it took a few hours to figure

00:52:23.882 --> 00:52:25.242
out exactly what the fuck was going on.

00:52:25.262 --> 00:52:26.003
Cause we actually thought,

00:52:26.643 --> 00:52:27.384
we actually thought it was a

00:52:27.764 --> 00:52:29.244
burned out SPF port is

00:52:29.284 --> 00:52:29.945
really what we thought.

00:52:30.025 --> 00:52:32.344
We thought that we had inbound fiber,

00:52:32.724 --> 00:52:33.905
that fiber that was dark.

00:52:35.425 --> 00:52:36.606
Then we lost connectivity to

00:52:36.646 --> 00:52:38.367
the uplink switch to the host switch.

00:52:38.947 --> 00:52:40.226
And that went dark because

00:52:40.246 --> 00:52:40.987
that stopped paying,

00:52:41.007 --> 00:52:41.907
which means good thing out.

00:52:42.168 --> 00:52:43.108
They said there was no light.

00:52:43.588 --> 00:52:43.708
So,

00:52:43.748 --> 00:52:45.429
which means we know that that whole

00:52:45.449 --> 00:52:45.989
thing is dark.

00:52:46.349 --> 00:52:47.228
And so why would that be dark?

00:52:47.773 --> 00:52:48.373
because we could actually

00:52:48.414 --> 00:52:49.114
see that the router,

00:52:49.795 --> 00:52:50.876
one of the core switches was up,

00:52:50.916 --> 00:52:51.777
and so blah, blah, blah.

00:52:52.237 --> 00:52:54.519
So what we realized was,

00:52:55.280 --> 00:52:56.021
and for the most part, actually,

00:52:56.041 --> 00:52:56.681
everything was fine.

00:52:56.742 --> 00:52:57.663
I do have some shit chains

00:52:57.682 --> 00:52:59.063
that are not on Horcrux.

00:52:59.164 --> 00:53:00.184
And so those were down,

00:53:00.204 --> 00:53:02.126
because I run those off of cheap VMs,

00:53:02.527 --> 00:53:03.949
not to be named.

00:53:04.588 --> 00:53:04.688
Um,

00:53:04.728 --> 00:53:06.028
and some test net boxes and things like

00:53:06.048 --> 00:53:06.268
that.

00:53:06.449 --> 00:53:06.969
Um, but there are some,

00:53:07.130 --> 00:53:07.849
there are some more things,

00:53:07.989 --> 00:53:09.010
there are some other things that we run,

00:53:09.670 --> 00:53:10.750
uh, for, um,

00:53:10.769 --> 00:53:11.911
for Chainlink and others that,

00:53:12.110 --> 00:53:12.891
that are housed there

00:53:12.931 --> 00:53:13.411
because it's actually,

00:53:13.431 --> 00:53:14.411
it's the most resilient,

00:53:14.751 --> 00:53:15.431
one of the most resilient

00:53:15.451 --> 00:53:16.050
things that we have,

00:53:16.070 --> 00:53:17.012
because we have a pretty

00:53:17.052 --> 00:53:18.811
large vSphere infrastructure.

00:53:18.831 --> 00:53:19.672
And so we can move VMs

00:53:19.711 --> 00:53:20.791
around and not deal with

00:53:20.811 --> 00:53:21.932
downtime associated with those.

00:53:21.972 --> 00:53:23.612
And there's like, we're not gonna get,

00:53:23.693 --> 00:53:24.193
we're not gonna go down

00:53:24.213 --> 00:53:25.313
because of a drive issue in

00:53:25.333 --> 00:53:25.954
those situations.

00:53:26.653 --> 00:53:28.193
Um, so yeah.

00:53:28.498 --> 00:53:28.978
So what we ended up

00:53:29.119 --> 00:53:30.721
realizing was that we have

00:53:30.760 --> 00:53:31.981
four switches in this rack

00:53:32.422 --> 00:53:33.822
and one of them is like a

00:53:33.902 --> 00:53:35.063
twenty-four port all fiber

00:53:35.423 --> 00:53:36.965
and the other one is a

00:53:37.065 --> 00:53:38.585
forty-eight port with fiber

00:53:38.626 --> 00:53:40.467
up links and has all copper.

00:53:40.987 --> 00:53:43.010
And we realized that that copper switch

00:53:44.161 --> 00:53:45.181
which actually has the

00:53:45.282 --> 00:53:46.262
inbound link or one of the

00:53:46.282 --> 00:53:48.103
inbound links is dead.

00:53:48.623 --> 00:53:50.344
And so it's, they haven't been,

00:53:50.364 --> 00:53:51.244
what they call the stack,

00:53:51.264 --> 00:53:52.364
which means they're

00:53:52.385 --> 00:53:53.385
connected to each other via

00:53:53.405 --> 00:53:54.126
this umbilical,

00:53:54.286 --> 00:53:55.766
two umbilicals actually between switches.

00:53:56.347 --> 00:53:58.027
There's four power supplies among the four,

00:53:58.088 --> 00:53:58.568
two switches.

00:53:58.608 --> 00:54:00.009
So there's fucking tons of power.

00:54:00.028 --> 00:54:00.248
We have,

00:54:00.628 --> 00:54:01.429
they're on two different battery

00:54:01.449 --> 00:54:03.769
backup units and like, like we don't,

00:54:04.010 --> 00:54:04.731
so we knew that there was

00:54:04.791 --> 00:54:05.690
no power issues.

00:54:06.831 --> 00:54:07.672
And that switch just

00:54:07.692 --> 00:54:08.572
basically browned out.

00:54:08.813 --> 00:54:09.592
It just fucking died.

00:54:10.293 --> 00:54:10.793
That's the end of it.

00:54:11.034 --> 00:54:11.193
Like,

00:54:11.534 --> 00:54:12.434
We couldn't get it to work.

00:54:12.494 --> 00:54:14.554
We couldn't get it to come up.

00:54:14.614 --> 00:54:17.014
We could not get, we have console to it.

00:54:17.034 --> 00:54:17.494
It was dead.

00:54:17.875 --> 00:54:18.695
It was just a fucking mess.

00:54:18.715 --> 00:54:20.195
It really was.

00:54:20.255 --> 00:54:22.275
So the remote hands at that

00:54:22.315 --> 00:54:23.275
center is not great.

00:54:24.536 --> 00:54:26.356
And so we ended up, it took us a few hours,

00:54:26.376 --> 00:54:27.297
but we ended up moving,

00:54:27.717 --> 00:54:28.737
getting the inbound moved

00:54:28.757 --> 00:54:29.516
to a different switch,

00:54:29.896 --> 00:54:30.677
which basically just take

00:54:30.818 --> 00:54:31.518
literally unplugging

00:54:31.557 --> 00:54:32.217
something and plugging it

00:54:32.237 --> 00:54:34.237
someplace else from a fiber perspective,

00:54:35.038 --> 00:54:36.358
which gave us some outbound

00:54:36.878 --> 00:54:38.219
and routing on that VLAN.

00:54:38.358 --> 00:54:38.719
And then,

00:54:39.807 --> 00:54:42.130
Um, and then from there we ended up, um,

00:54:42.210 --> 00:54:43.271
hard booting that switch,

00:54:43.291 --> 00:54:44.393
like literally pulled four

00:54:44.713 --> 00:54:46.253
power supplies out and just

00:54:46.293 --> 00:54:46.974
letting it sit for a minute

00:54:46.994 --> 00:54:47.695
and putting it back in.

00:54:47.775 --> 00:54:49.297
And then that secondary came back,

00:54:49.416 --> 00:54:50.117
which was really weird.

00:54:50.137 --> 00:54:51.079
Cause if that didn't come back,

00:54:51.619 --> 00:54:52.340
we had some other issues.

00:54:52.360 --> 00:54:54.322
That means we knew we're going to have to,

00:54:54.342 --> 00:54:55.222
we do have extra hardware,

00:54:55.242 --> 00:54:56.143
but it just can become a

00:54:56.224 --> 00:54:57.445
bigger issue that we were

00:54:57.465 --> 00:54:58.405
concerned that some remote

00:54:58.445 --> 00:54:59.367
hands would not be able to do.

00:55:00.268 --> 00:55:00.487
Um,

00:55:00.527 --> 00:55:02.469
so we got through that and then it came

00:55:02.489 --> 00:55:02.670
back.

00:55:02.710 --> 00:55:03.990
So we were down for, fuck,

00:55:04.050 --> 00:55:04.552
it must've been.

00:55:05.961 --> 00:55:07.322
nine hours, something like that.

00:55:07.443 --> 00:55:08.443
I think I was awake for most of it,

00:55:11.405 --> 00:55:12.025
which is probably the

00:55:12.085 --> 00:55:12.965
longest that we've had in

00:55:13.005 --> 00:55:13.847
those types of situations,

00:55:13.867 --> 00:55:14.807
especially in those DCs

00:55:14.847 --> 00:55:17.329
where you can have a bunch of issues.

00:55:17.349 --> 00:55:18.670
We have multiple inbound lines.

00:55:18.710 --> 00:55:20.471
Multiple inbound lines would have helped,

00:55:21.010 --> 00:55:22.733
but just because we have

00:55:23.771 --> 00:55:25.152
four massive switches in there,

00:55:25.172 --> 00:55:25.751
and that would have had

00:55:25.791 --> 00:55:26.612
some redundancy in there,

00:55:26.632 --> 00:55:27.273
although you have to deal

00:55:27.293 --> 00:55:28.273
with outbound routes and

00:55:28.293 --> 00:55:28.813
things like that.

00:55:29.112 --> 00:55:30.072
BGP wouldn't have helped.

00:55:31.413 --> 00:55:32.132
There's a bunch of things

00:55:32.152 --> 00:55:33.373
that we dealt with that

00:55:33.414 --> 00:55:37.534
were very specific to just

00:55:37.554 --> 00:55:38.934
like a... And plus,

00:55:39.894 --> 00:55:41.554
Jeff being on a flight didn't help.

00:55:41.795 --> 00:55:42.695
That just slowed us down.

00:55:43.635 --> 00:55:44.215
Things like that where

00:55:44.235 --> 00:55:45.735
you're not... If he was in town,

00:55:45.795 --> 00:55:46.815
we would have been back up

00:55:46.835 --> 00:55:48.135
in like an hour or less, right?

00:55:48.155 --> 00:55:49.476
Because he would have been over there.

00:55:49.496 --> 00:55:50.217
We would have fixed it up.

00:55:50.376 --> 00:55:51.157
It would have been no problem.

00:55:51.197 --> 00:55:51.976
But when you deal with like...

00:55:52.797 --> 00:55:53.916
remote data centers are we

00:55:53.936 --> 00:55:54.597
dealing with that type of

00:55:54.617 --> 00:55:56.177
thing it becomes uh it

00:55:56.217 --> 00:55:57.797
becomes a challenge so it

00:55:57.818 --> 00:55:58.818
was stressful what kind of

00:55:58.898 --> 00:55:59.639
what kind of stuff was uh

00:56:00.119 --> 00:56:01.760
was affected because I know

00:56:01.780 --> 00:56:03.019
you're on like uh some some

00:56:03.139 --> 00:56:04.320
cosmos chains we got jailed

00:56:04.420 --> 00:56:05.740
on but again they're trying

00:56:05.780 --> 00:56:10.541
small things um we had um

00:56:10.601 --> 00:56:11.222
honestly I wasn't really

00:56:11.262 --> 00:56:12.001
worried about any of those

00:56:12.043 --> 00:56:13.382
if they were saying there

00:56:13.483 --> 00:56:14.322
once we had once I had a

00:56:14.362 --> 00:56:15.322
connectivity back up in the

00:56:15.362 --> 00:56:16.324
morning at least outbound

00:56:17.103 --> 00:56:18.204
um I I could have moved

00:56:18.244 --> 00:56:18.704
some of that stuff

00:56:18.744 --> 00:56:20.224
beforehand I talked to null

00:56:20.244 --> 00:56:20.724
about actually

00:56:21.755 --> 00:56:25.516
about just undelegating

00:56:25.556 --> 00:56:26.376
associated with those, right?

00:56:26.416 --> 00:56:27.217
And just taking all the

00:56:27.237 --> 00:56:28.057
stake away just so we get

00:56:28.097 --> 00:56:30.838
jailed and not get slashed.

00:56:31.599 --> 00:56:32.559
But the ones that we were dealing with,

00:56:32.619 --> 00:56:34.219
I mean, we're talking like, honestly,

00:56:34.239 --> 00:56:35.201
not to bad mouth change,

00:56:35.221 --> 00:56:35.920
but there were ones that we

00:56:35.940 --> 00:56:37.481
had less than like,

00:56:37.541 --> 00:56:39.021
ten thousand dollars staked

00:56:39.041 --> 00:56:40.302
to the machine, total.

00:56:40.443 --> 00:56:41.643
Like, total, everybody.

00:56:41.824 --> 00:56:42.784
Like, less than,

00:56:43.003 --> 00:56:43.804
or even some of those were

00:56:43.824 --> 00:56:44.585
probably less than that.

00:56:44.625 --> 00:56:45.744
So it was really no big deal.

00:56:45.764 --> 00:56:45.864
Like,

00:56:45.905 --> 00:56:46.925
I wasn't really too worried about that.

00:56:48.882 --> 00:56:49.902
And then OCR for Chainlink

00:56:49.922 --> 00:56:51.242
was in there and some CCIP

00:56:51.322 --> 00:56:52.782
or some stuff that we do there.

00:56:53.282 --> 00:56:55.222
Not the actual nodes are not there.

00:56:55.362 --> 00:56:56.463
Like the RPC is not there,

00:56:56.483 --> 00:56:58.463
but the actual core of what

00:56:58.483 --> 00:56:59.103
we do is there.

00:56:59.123 --> 00:57:00.963
And there's a couple other things too,

00:57:01.003 --> 00:57:01.585
actually, which I didn't,

00:57:01.605 --> 00:57:02.324
I wouldn't really think about,

00:57:02.344 --> 00:57:03.125
which I'm going to change.

00:57:03.824 --> 00:57:04.945
We run a pretty large load

00:57:04.985 --> 00:57:06.605
balancer out of there to be

00:57:06.625 --> 00:57:07.425
able to have a single

00:57:07.485 --> 00:57:10.606
endpoint for maybe thirty L

00:57:10.706 --> 00:57:11.365
one L two chains.

00:57:11.385 --> 00:57:13.027
And so if somebody needs an L one,

00:57:13.166 --> 00:57:14.666
like DYDX is a good example of that.

00:57:15.492 --> 00:57:16.693
A DYDX needs an L one.

00:57:16.793 --> 00:57:17.972
That L one is actually in

00:57:17.992 --> 00:57:18.713
that data center.

00:57:19.134 --> 00:57:20.835
And so it's calling back to that L one.

00:57:21.474 --> 00:57:22.255
And then there's a load

00:57:22.275 --> 00:57:23.815
balancer structure or, you know,

00:57:23.835 --> 00:57:24.737
it's in a failover piece

00:57:24.757 --> 00:57:26.398
where it calls a single Ethereum node.

00:57:26.418 --> 00:57:27.538
And then if that one is down

00:57:27.557 --> 00:57:27.978
or I've been doing

00:57:27.998 --> 00:57:28.998
maintenance and it goes to the next one.

00:57:29.887 --> 00:57:30.887
Um, and that is in there.

00:57:30.907 --> 00:57:31.748
So that's actually,

00:57:32.389 --> 00:57:33.630
that's actually should not be in there.

00:57:33.911 --> 00:57:34.211
So like,

00:57:34.231 --> 00:57:35.233
I'm going to move that to a cloud

00:57:35.273 --> 00:57:36.173
services type of thing.

00:57:36.193 --> 00:57:37.474
Cause that's not really that big of it.

00:57:37.494 --> 00:57:38.556
It doesn't move that much traffic.

00:57:39.297 --> 00:57:39.677
Um, and it,

00:57:39.737 --> 00:57:42.679
but it is important enough where, um,

00:57:43.221 --> 00:57:44.742
I might have multiple of those up.

00:57:44.762 --> 00:57:44.922
Actually,

00:57:44.943 --> 00:57:45.943
I'm just going to copy what I

00:57:45.983 --> 00:57:46.744
have and put it in a couple

00:57:46.764 --> 00:57:47.965
of different places just so

00:57:48.005 --> 00:57:48.987
if I need to redirect something,

00:57:49.047 --> 00:57:49.847
I could do that.

00:57:50.882 --> 00:57:52.244
Um, so it wasn't, it,

00:57:52.284 --> 00:57:53.286
it didn't cost us any, I mean,

00:57:54.086 --> 00:57:54.608
it didn't really,

00:57:54.927 --> 00:57:56.148
it wasn't financially painful.

00:57:56.610 --> 00:57:58.492
Uh, it was stressful, painful,

00:57:58.532 --> 00:57:59.213
and it's also a lot of

00:57:59.233 --> 00:58:00.153
communication because I was

00:58:00.173 --> 00:58:01.195
trying to keep teams

00:58:01.315 --> 00:58:02.456
involved with what the hell was going on.

00:58:03.358 --> 00:58:06.280
Um, but, uh, yeah, it was something, man,

00:58:06.320 --> 00:58:07.402
it was fucking pain in the ass.

00:58:08.409 --> 00:58:09.590
like just,

00:58:09.690 --> 00:58:11.391
just a perfect storm of stupid shit.

00:58:11.431 --> 00:58:12.672
So we have a long list of

00:58:12.713 --> 00:58:14.153
things we want to change to

00:58:14.173 --> 00:58:15.373
be able to help with that redundant,

00:58:15.574 --> 00:58:16.534
a separate redundant route

00:58:16.594 --> 00:58:18.255
outbound because what we

00:58:18.275 --> 00:58:20.197
thought was the issue wasn't the issue,

00:58:20.217 --> 00:58:21.418
but it could be an issue in the future.

00:58:23.079 --> 00:58:24.239
We have some ideas around

00:58:24.380 --> 00:58:26.960
switch segregation and we're going to,

00:58:26.981 --> 00:58:27.461
we're going to build a

00:58:27.501 --> 00:58:28.521
complete out of band network,

00:58:28.682 --> 00:58:30.023
which basically means we

00:58:30.043 --> 00:58:30.963
have a primary and then you

00:58:30.983 --> 00:58:31.844
have a total separate one

00:58:31.864 --> 00:58:32.824
that has no traffic from

00:58:32.864 --> 00:58:34.585
the outbound and it has a totally,

00:58:35.045 --> 00:58:36.686
a hundred percent cut across.

00:58:36.726 --> 00:58:36.927
And then,

00:58:37.668 --> 00:58:39.449
And then all of our nodes in

00:58:39.489 --> 00:58:40.608
there will be multi-homed,

00:58:40.708 --> 00:58:42.750
which means it has primary outbound.

00:58:42.789 --> 00:58:43.489
And if for some reason we

00:58:43.510 --> 00:58:44.030
need to switch it,

00:58:44.130 --> 00:58:45.130
I can go back and change.

00:58:45.150 --> 00:58:46.550
We can use something like

00:58:46.610 --> 00:58:47.710
IPSLA or other types of

00:58:47.751 --> 00:58:48.951
things that would just take

00:58:48.971 --> 00:58:50.092
specific nodes and redirect

00:58:50.112 --> 00:58:51.172
them to a different outbound.

00:58:51.192 --> 00:58:52.211
So all their IPs would change,

00:58:52.672 --> 00:58:53.972
but at least they would start talking.

00:58:54.193 --> 00:58:54.592
You know what I mean?

00:58:54.612 --> 00:58:56.074
At least we have some outside there.

00:58:57.673 --> 00:58:58.994
So there's a, which is,

00:58:59.173 --> 00:58:59.755
it just gets really

00:58:59.775 --> 00:59:00.954
expensive is really what it

00:59:01.014 --> 00:59:02.195
is because we're just

00:59:02.255 --> 00:59:03.596
adding costs and those types of things.

00:59:03.635 --> 00:59:04.596
But again,

00:59:04.675 --> 00:59:06.476
I was hoping to get rid of this

00:59:06.516 --> 00:59:07.036
whole stack.

00:59:09.621 --> 00:59:11.063
But the more that we start

00:59:11.083 --> 00:59:11.804
dealing with it and we

00:59:11.824 --> 00:59:13.864
start talking about like Zabbix HA,

00:59:14.545 --> 00:59:15.686
which I would do another

00:59:15.726 --> 00:59:16.507
one of these outside of

00:59:16.527 --> 00:59:19.088
that data center and things like that.

00:59:19.168 --> 00:59:20.309
Oh man, I was gonna mention that.

00:59:20.789 --> 00:59:23.152
V seven HA baby, it looks pretty good.

00:59:23.853 --> 00:59:25.373
Yeah, so that's something.

00:59:25.474 --> 00:59:25.934
And then now,

00:59:25.994 --> 00:59:27.255
but that helps me also think about,

00:59:27.275 --> 00:59:29.097
I was thinking about HA more from like,

00:59:29.949 --> 00:59:30.889
My first thing when I think

00:59:30.909 --> 00:59:33.452
about HA is like hardware failure, right?

00:59:33.472 --> 00:59:34.492
That's like my first thing.

00:59:35.032 --> 00:59:37.295
But the real issue with that

00:59:37.474 --> 00:59:39.635
is like when that was down,

00:59:40.456 --> 00:59:41.518
in the middle of all that shit,

00:59:41.878 --> 00:59:42.778
in the middle of all that stuff,

00:59:42.798 --> 00:59:44.559
we had an issue where on

00:59:44.579 --> 00:59:45.840
this latest say upgrade,

00:59:46.641 --> 00:59:47.521
we were getting some RPC

00:59:47.541 --> 00:59:48.563
traffic that is basically

00:59:48.782 --> 00:59:49.603
putting the CPUs to a

00:59:49.643 --> 00:59:51.184
hundred percent and they're

00:59:51.204 --> 00:59:51.826
starting to lag.

00:59:52.425 --> 00:59:54.668
And so at like four a.m.,

00:59:54.807 --> 00:59:56.108
that same fucking night,

00:59:56.668 --> 00:59:57.670
I have an external alert

00:59:57.690 --> 00:59:58.889
going that all our RPCs are

00:59:58.909 --> 00:59:59.811
down or there's nothing

00:59:59.831 --> 01:00:00.972
that's responding to requests.

01:00:01.452 --> 01:00:02.592
And I got down here after

01:00:02.612 --> 01:00:03.353
like thirty minutes of

01:00:03.373 --> 01:00:04.753
sleep and I realized that

01:00:05.034 --> 01:00:06.414
they're all like way high.

01:00:06.434 --> 01:00:07.835
And I just restart the

01:00:07.876 --> 01:00:08.836
process and it goes down to

01:00:08.876 --> 01:00:09.757
zero and it's fine for a

01:00:09.777 --> 01:00:10.858
while and it happens again.

01:00:11.318 --> 01:00:12.840
So there's a say issue with that.

01:00:13.340 --> 01:00:14.221
But I realized that, like,

01:00:14.360 --> 01:00:15.581
I'm so fucking blind.

01:00:16.641 --> 01:00:18.164
so much monitoring that I

01:00:18.184 --> 01:00:19.664
feel so blind without that.

01:00:20.786 --> 01:00:21.567
There's a ton of stuff like

01:00:21.806 --> 01:00:22.688
anything that we were running.

01:00:23.469 --> 01:00:24.250
We could have been jailed on

01:00:24.289 --> 01:00:25.590
things or you have outages

01:00:25.650 --> 01:00:29.695
or machine dies or fucking anything,

01:00:29.755 --> 01:00:29.896
right?

01:00:29.916 --> 01:00:30.195
There's a,

01:00:30.295 --> 01:00:32.418
there's so much that we are monitoring.

01:00:32.438 --> 01:00:32.617
Like,

01:00:33.438 --> 01:00:34.420
don't even know there's a

01:00:34.460 --> 01:00:36.721
lot of faith in that in

01:00:36.760 --> 01:00:38.181
that one instance right you

01:00:38.222 --> 01:00:39.422
gotta oh it's crazy it's

01:00:39.443 --> 01:00:40.923
time to like start looking

01:00:40.943 --> 01:00:42.465
at aj for that for sure not

01:00:42.505 --> 01:00:43.826
only that but just but I

01:00:43.846 --> 01:00:46.088
mean just like um I mean

01:00:46.128 --> 01:00:48.048
we're pulling I'm pulling

01:00:48.228 --> 01:00:50.891
different hosts of data in

01:00:50.931 --> 01:00:52.791
zabbix and then so it's on

01:00:52.851 --> 01:00:53.693
average we're pulling

01:00:55.313 --> 01:00:58.675
different values per second

01:00:58.715 --> 01:01:00.016
values per second right if

01:01:00.036 --> 01:01:01.177
you add them all this up right

01:01:01.918 --> 01:01:02.760
um that's all bare that

01:01:02.780 --> 01:01:03.322
could be bare metal it

01:01:03.342 --> 01:01:04.806
could be http type things it could be

01:01:05.563 --> 01:01:06.123
Missed blocks,

01:01:06.463 --> 01:01:07.222
there's a ton of things that

01:01:07.864 --> 01:01:09.643
we collectively measure, right?

01:01:09.664 --> 01:01:10.684
Seven hundred a second at

01:01:10.784 --> 01:01:12.264
nine hours is a fucking lot

01:01:12.284 --> 01:01:13.844
of visibility of things

01:01:13.864 --> 01:01:15.545
that are being checked that are,

01:01:16.905 --> 01:01:17.786
there's so many things that

01:01:17.806 --> 01:01:18.365
could go wrong.

01:01:18.385 --> 01:01:21.106
That's really what it is, right?

01:01:21.146 --> 01:01:22.827
And when you don't have that,

01:01:23.146 --> 01:01:25.208
when you don't have that safety net of,

01:01:25.387 --> 01:01:26.628
oh, if something does go wrong,

01:01:26.648 --> 01:01:27.367
I'm going to get this phone

01:01:27.387 --> 01:01:29.168
call with the Yakety Sax.

01:01:29.668 --> 01:01:31.108
When that is, and that you can,

01:01:31.228 --> 01:01:32.489
it's fun to make fun of that, right?

01:01:32.509 --> 01:01:34.230
And like, it's so fucking annoying.

01:01:34.710 --> 01:01:35.891
But when that's gone,

01:01:35.971 --> 01:01:37.552
like I'm thinking like, all right,

01:01:37.632 --> 01:01:39.032
what else could fail right now?

01:01:39.213 --> 01:01:40.373
And how the fuck would I know?

01:01:41.434 --> 01:01:41.635
Like,

01:01:41.735 --> 01:01:42.655
and we have a lot of external

01:01:42.675 --> 01:01:44.376
monitors too and like big services,

01:01:44.978 --> 01:01:45.978
but not everything.

01:01:46.177 --> 01:01:46.619
You know what I mean?

01:01:46.778 --> 01:01:47.559
Like if AppDos goes down,

01:01:47.599 --> 01:01:48.380
actually we have like four

01:01:48.400 --> 01:01:49.001
different monitors,

01:01:49.021 --> 01:01:50.021
both internally and externally.

01:01:50.041 --> 01:01:50.601
Like we have a bunch of

01:01:50.621 --> 01:01:51.742
those and like things like

01:01:51.762 --> 01:01:53.083
that for Chainlink or for

01:01:53.204 --> 01:01:54.403
our big RPC arrays that we

01:01:54.465 --> 01:01:56.045
run or those types of things.

01:01:56.085 --> 01:01:57.527
But there's so many others.

01:01:58.086 --> 01:02:00.128
Like there's so many others.

01:02:00.148 --> 01:02:00.509
Yeah.

01:02:00.909 --> 01:02:02.208
And that really sucks.

01:02:02.548 --> 01:02:04.550
So yeah.

01:02:04.590 --> 01:02:06.190
So we came up with a pretty good,

01:02:06.690 --> 01:02:08.070
we're good meeting again,

01:02:08.130 --> 01:02:09.210
but we came up with a good

01:02:09.250 --> 01:02:10.871
list of just hardening

01:02:10.911 --> 01:02:14.030
things that I think will be quite good.

01:02:14.110 --> 01:02:15.731
Even, even like, you know,

01:02:15.972 --> 01:02:17.012
we even track like what

01:02:17.072 --> 01:02:18.251
power supplies goes to

01:02:18.311 --> 01:02:19.791
exactly what port on what

01:02:19.972 --> 01:02:22.753
PDUs and go into what are, you know,

01:02:23.032 --> 01:02:23.773
backup units.

01:02:23.833 --> 01:02:26.032
And you know,

01:02:26.373 --> 01:02:27.934
like there's a ton of mapping

01:02:27.974 --> 01:02:29.014
that goes on there and we

01:02:29.054 --> 01:02:29.893
know exactly what

01:02:30.851 --> 01:02:32.012
ports on the ethernet side

01:02:32.032 --> 01:02:33.333
are going to which things

01:02:33.413 --> 01:02:36.135
and for what and so if

01:02:36.155 --> 01:02:37.255
there's a brownout port and

01:02:37.275 --> 01:02:38.115
there's a name on it we

01:02:38.155 --> 01:02:38.795
have a good understanding

01:02:38.815 --> 01:02:40.317
what that is like and it

01:02:40.336 --> 01:02:41.197
just becomes a data

01:02:41.257 --> 01:02:43.219
management nightmare of

01:02:43.239 --> 01:02:44.000
keeping track some of that

01:02:44.039 --> 01:02:45.860
stuff but and then so what

01:02:45.880 --> 01:02:47.061
that means is then we're

01:02:47.081 --> 01:02:47.862
talking about we do have a

01:02:47.882 --> 01:02:48.682
change management database

01:02:48.722 --> 01:02:49.304
but we're starting to get a

01:02:49.324 --> 01:02:49.983
little bit more advanced in

01:02:50.003 --> 01:02:50.885
terms of change management

01:02:51.717 --> 01:02:52.458
Um, and things like that,

01:02:52.498 --> 01:02:54.699
because also in those situations,

01:02:55.179 --> 01:02:56.121
what's really annoying is like,

01:02:56.181 --> 01:02:57.222
if you're trying to figure

01:02:57.242 --> 01:02:58.302
out the hell's going on and

01:02:58.322 --> 01:02:59.583
you need to like, oh,

01:02:59.603 --> 01:03:00.784
let's go see if this is happening.

01:03:00.804 --> 01:03:02.125
And then, oh, how do I find that out?

01:03:02.144 --> 01:03:03.945
And you gotta go look up this, you know,

01:03:03.965 --> 01:03:05.708
figure out like these,

01:03:05.807 --> 01:03:06.788
like these four switches,

01:03:07.248 --> 01:03:08.369
but like we have four power

01:03:08.389 --> 01:03:09.670
supplies and the two switches,

01:03:10.231 --> 01:03:11.090
two or two are going to two

01:03:11.110 --> 01:03:11.811
different battery backup

01:03:11.871 --> 01:03:12.711
units and two are going

01:03:12.751 --> 01:03:13.672
directly into PDUs.

01:03:14.574 --> 01:03:15.994
If I needed to shut those four down,

01:03:17.074 --> 01:03:18.076
how the fuck would we do that?

01:03:19.257 --> 01:03:19.416
Like.

01:03:19.793 --> 01:03:19.952
Like,

01:03:20.413 --> 01:03:21.853
which battery backup unit would I kill?

01:03:21.893 --> 01:03:23.335
And which port on this PDU?

01:03:23.715 --> 01:03:24.815
And how would I make sure

01:03:24.835 --> 01:03:25.976
that that thing was off if

01:03:25.996 --> 01:03:27.197
I don't have a camera facing at it?

01:03:27.237 --> 01:03:30.300
Like, those types of things, it starts,

01:03:31.000 --> 01:03:32.661
like, you get headaches really quick.

01:03:32.902 --> 01:03:34.862
So I was just trying to think of, like,

01:03:34.922 --> 01:03:36.864
how to, if this had happened, like,

01:03:36.884 --> 01:03:37.664
you couldn't shut down both,

01:03:38.204 --> 01:03:39.846
we couldn't shut down both APCs.

01:03:39.865 --> 01:03:40.806
Like, that would be a real problem.

01:03:40.827 --> 01:03:41.987
But I could shut down one.

01:03:42.847 --> 01:03:43.748
Um, and,

01:03:43.809 --> 01:03:45.429
but that APC is plugged into the

01:03:45.449 --> 01:03:46.650
same switch that's browned out.

01:03:46.751 --> 01:03:47.231
Oh fuck.

01:03:47.452 --> 01:03:48.211
So I can't do that.

01:03:48.251 --> 01:03:50.253
So then like, how the hell, like what,

01:03:50.293 --> 01:03:51.434
how else can I talk to this thing?

01:03:51.454 --> 01:03:52.815
And like all that type of shit.

01:03:52.835 --> 01:03:56.177
So it's, uh, and of course,

01:03:56.237 --> 01:03:57.498
which outages is probably

01:03:57.518 --> 01:03:58.139
the worst thing that can

01:03:58.179 --> 01:03:58.920
happen because you're just,

01:03:58.940 --> 01:03:59.860
you're just in the dark.

01:03:59.900 --> 01:04:00.081
Right.

01:04:00.121 --> 01:04:01.061
So we had all our fiber was

01:04:01.101 --> 01:04:01.842
working and we could see

01:04:01.862 --> 01:04:03.003
all the BMs and all that kind of stuff,

01:04:03.063 --> 01:04:05.324
but the actual, you know, um,

01:04:05.344 --> 01:04:07.827
fee sphere and, uh, IPMI and, um,

01:04:08.307 --> 01:04:08.748
you know,

01:04:08.768 --> 01:04:10.047
things that actually use copper

01:04:10.068 --> 01:04:11.528
were all dark and that,

01:04:11.768 --> 01:04:13.309
that becomes really,

01:04:13.369 --> 01:04:14.128
really painful when you're

01:04:14.148 --> 01:04:15.188
trying to troubleshoot from

01:04:15.630 --> 01:04:16.650
a couple thousand miles away.

01:04:17.510 --> 01:04:19.309
So, um, and,

01:04:19.409 --> 01:04:20.349
or having somebody standing

01:04:20.369 --> 01:04:21.550
from front of that rack who

01:04:21.570 --> 01:04:22.751
doesn't know any of that type of stuff,

01:04:22.771 --> 01:04:23.271
who's trying to,

01:04:23.311 --> 01:04:24.751
or trying to guide and say, okay,

01:04:24.771 --> 01:04:25.190
try this.

01:04:25.210 --> 01:04:26.911
And, you know, doing a FaceTime call,

01:04:26.931 --> 01:04:27.731
they'll see what the hell's

01:04:27.771 --> 01:04:28.711
blinking and what's not in

01:04:28.751 --> 01:04:29.512
certain situations.

01:04:29.552 --> 01:04:32.393
Like, uh, so yeah, anyway, that was good.

01:04:32.632 --> 01:04:33.092
I enjoyed it.

01:04:33.112 --> 01:04:33.793
It was fun.

01:04:35.193 --> 01:04:36.414
And troubleshooting with a

01:04:36.494 --> 01:04:37.434
remote hands could be,

01:04:38.626 --> 01:04:39.987
quite annoying.

01:04:40.567 --> 01:04:41.987
Well, I mean, even for you,

01:04:42.047 --> 01:04:43.108
like you have stuff now too, like,

01:04:43.708 --> 01:04:44.369
like we're starting to

01:04:44.389 --> 01:04:45.929
write run books and those

01:04:45.969 --> 01:04:50.152
run books are like, again, what ports,

01:04:50.391 --> 01:04:53.273
what goes to what diagrams, what PDUs,

01:04:53.773 --> 01:04:54.914
how are those PDUs controlled?

01:04:55.175 --> 01:04:56.355
What are the logins for those?

01:04:56.394 --> 01:04:57.376
What are the IPs for those?

01:04:57.476 --> 01:04:59.237
So if a remote hands gets in

01:04:59.257 --> 01:04:59.896
there and they plug a

01:04:59.956 --> 01:05:00.838
laptop into something

01:05:00.858 --> 01:05:01.797
that's sitting there, that's working,

01:05:02.978 --> 01:05:03.938
like, where do they go?

01:05:04.018 --> 01:05:05.500
How do they get in?

01:05:05.539 --> 01:05:06.920
Is that password available easily?

01:05:08.235 --> 01:05:11.436
Um, what exact, um, you know, there's a,

01:05:11.456 --> 01:05:13.396
there's a million things in there to like,

01:05:13.416 --> 01:05:15.998
try to figure out how to, uh,

01:05:16.018 --> 01:05:17.918
how to troubleshoot remotely or,

01:05:18.159 --> 01:05:19.800
or you're, you're heading down there,

01:05:19.820 --> 01:05:20.500
which might be fine.

01:05:20.519 --> 01:05:21.400
But if you're in the U S.

01:05:22.329 --> 01:05:22.969
and you have an outage and

01:05:22.989 --> 01:05:23.630
you have somebody standing

01:05:23.650 --> 01:05:24.630
in front of that rack that

01:05:24.650 --> 01:05:27.052
you're paying and trying to

01:05:27.152 --> 01:05:28.552
guide them through these types of things,

01:05:28.592 --> 01:05:29.733
some of those things might be impossible.

01:05:30.014 --> 01:05:30.894
Like you might not be able to do it.

01:05:31.155 --> 01:05:31.335
Right.

01:05:31.454 --> 01:05:33.936
And so it's a good thing.

01:05:33.976 --> 01:05:34.356
So we started,

01:05:34.617 --> 01:05:36.197
we started talking about what, you know,

01:05:36.217 --> 01:05:37.099
we have run books for other stuff,

01:05:37.139 --> 01:05:37.378
but like,

01:05:37.958 --> 01:05:39.420
like what does that unrun book look like?

01:05:39.460 --> 01:05:39.740
And like,

01:05:39.880 --> 01:05:40.800
what are those types of items

01:05:40.820 --> 01:05:41.521
that have to be in there?

01:05:41.581 --> 01:05:46.284
And so it's it just makes you, you know,

01:05:46.304 --> 01:05:47.166
those types of things like

01:05:47.226 --> 01:05:49.206
really open your eyes to, Oh fuck.

01:05:49.407 --> 01:05:52.148
Like what about this situation?

01:05:53.130 --> 01:05:54.349
So AutoBand will be our first.

01:05:55.110 --> 01:05:56.751
Our LTE backup worked out great.

01:05:56.911 --> 01:05:57.532
If we didn't have that,

01:05:57.552 --> 01:05:58.193
we would have been fucked.

01:05:59.353 --> 01:06:00.193
That was really nice to be

01:06:00.213 --> 01:06:01.655
able to have a wireless

01:06:02.215 --> 01:06:03.436
option into that data center.

01:06:03.637 --> 01:06:05.097
We're going to add one to

01:06:05.117 --> 01:06:05.918
our other one as well,

01:06:05.958 --> 01:06:06.938
which we don't have right now,

01:06:06.958 --> 01:06:07.478
although the other one is

01:06:07.518 --> 01:06:08.320
not nearly as important.

01:06:08.880 --> 01:06:10.219
as this one, uh,

01:06:10.240 --> 01:06:12.001
we'll do a full out of band network, um,

01:06:12.021 --> 01:06:12.701
that will allow,

01:06:12.780 --> 01:06:15.242
we'll do IPS LA on specific things,

01:06:15.822 --> 01:06:17.742
whether that's automatic or even manual,

01:06:17.762 --> 01:06:19.423
it would just be ability to,

01:06:19.943 --> 01:06:24.344
to divert things from A to B. Um, and, uh,

01:06:24.385 --> 01:06:28.086
yeah, there's a, I mean, it's, it was,

01:06:28.206 --> 01:06:29.407
it was, uh, it was good.

01:06:29.427 --> 01:06:30.206
I mean, it was, you know,

01:06:30.547 --> 01:06:31.788
at the time it was miserable at night.

01:06:31.807 --> 01:06:32.527
You like look back and you're like,

01:06:32.547 --> 01:06:33.188
all right, that was good.

01:06:33.307 --> 01:06:35.389
Cause I think no matter what we do there,

01:06:35.768 --> 01:06:36.650
um, it'll come out stronger.

01:06:37.630 --> 01:06:37.670
Um,

01:06:39.152 --> 01:06:43.737
So my rack is pretty new.

01:06:43.898 --> 01:06:46.661
Well, the new rack is pretty new.

01:06:46.702 --> 01:06:48.784
But even looking at the

01:06:48.804 --> 01:06:49.605
hardware we're going to put

01:06:49.644 --> 01:06:52.548
in ours and learning off

01:06:52.809 --> 01:06:53.909
some of your... We were

01:06:53.929 --> 01:06:55.172
sort of thinking about this beforehand,

01:06:55.192 --> 01:06:56.373
but now hearing about...

01:06:57.054 --> 01:06:57.835
what happened with you,

01:06:57.894 --> 01:06:59.197
I think we're almost

01:06:59.237 --> 01:07:00.077
certainly going to do this

01:07:00.137 --> 01:07:02.780
now is multi-chassis link

01:07:02.820 --> 01:07:04.161
aggregation from the routers.

01:07:04.302 --> 01:07:05.264
So we have to get new routers,

01:07:05.344 --> 01:07:07.385
but basically like these,

01:07:07.626 --> 01:07:08.527
these routers all have

01:07:08.588 --> 01:07:09.969
twenty five gig ports in them.

01:07:10.590 --> 01:07:13.172
And then so they you have

01:07:13.213 --> 01:07:14.715
two Internet connections coming in.

01:07:15.766 --> 01:07:16.628
one to each router.

01:07:17.208 --> 01:07:18.690
And then the multi-chassis

01:07:18.730 --> 01:07:19.771
link aggregation is

01:07:19.811 --> 01:07:20.891
basically you take a twenty

01:07:20.931 --> 01:07:23.793
five gig from each router

01:07:24.554 --> 01:07:25.856
or switch at this stage and

01:07:25.876 --> 01:07:28.018
then you can plug two

01:07:28.057 --> 01:07:29.358
twenty fives into each chassis.

01:07:31.081 --> 01:07:32.161
And then basically if one

01:07:32.322 --> 01:07:33.583
entire router goes down,

01:07:33.663 --> 01:07:34.503
then you've still got a

01:07:34.543 --> 01:07:36.644
route from the Internet to the

01:07:37.865 --> 01:07:40.586
um to each chassis so you

01:07:40.626 --> 01:07:42.447
can't like you can lose an

01:07:42.487 --> 01:07:43.668
internet connection or you

01:07:43.688 --> 01:07:45.789
can lose an entire router

01:07:46.048 --> 01:07:46.889
and you should still be

01:07:46.929 --> 01:07:50.110
good um but the only like

01:07:50.130 --> 01:07:51.811
you say with the the ips on

01:07:51.851 --> 01:07:54.132
the outgoing can change but

01:07:54.172 --> 01:07:55.632
you can actually have like

01:07:55.972 --> 01:07:56.693
I think you mentioned that

01:07:56.713 --> 01:07:58.652
you can multi-home the the

01:07:58.753 --> 01:07:59.994
ip so you can actually

01:08:00.054 --> 01:08:00.713
route it through two

01:08:00.733 --> 01:08:02.215
different providers if you

01:08:02.275 --> 01:08:05.896
use bgp um and your own ips I think

01:08:07.331 --> 01:08:09.592
So you can, you don't want to,

01:08:09.612 --> 01:08:10.793
you don't want to manage BGP.

01:08:10.833 --> 01:08:11.972
So they would manage that.

01:08:12.233 --> 01:08:13.733
No, that our providers do that.

01:08:13.773 --> 01:08:15.373
That's also, that's also really,

01:08:16.154 --> 01:08:16.494
that's not,

01:08:16.715 --> 01:08:18.956
that's also challenging to be honest.

01:08:19.176 --> 01:08:20.376
So if you're managing it yourself,

01:08:20.416 --> 01:08:22.056
like because it's not,

01:08:22.256 --> 01:08:23.877
it's not like it's not magic.

01:08:24.137 --> 01:08:24.757
Like there's a lot of,

01:08:24.778 --> 01:08:26.198
especially if you're sink

01:08:26.238 --> 01:08:27.639
holding type things or like,

01:08:27.658 --> 01:08:28.439
there's a lot to that,

01:08:28.460 --> 01:08:29.199
that has to go in there.

01:08:29.239 --> 01:08:31.161
But so you don't necessarily

01:08:31.341 --> 01:08:32.180
need to do that,

01:08:32.440 --> 01:08:33.761
but it would be good to have.

01:08:34.898 --> 01:08:35.018
No,

01:08:35.297 --> 01:08:37.759
but if somebody figured that out and

01:08:37.779 --> 01:08:38.378
somebody's pissed at you

01:08:38.399 --> 01:08:40.538
and they're DDoSing that IP,

01:08:41.439 --> 01:08:42.378
then somebody's going to

01:08:42.439 --> 01:08:43.680
manually type of sinkhole

01:08:43.699 --> 01:08:45.000
that and be able to create

01:08:45.020 --> 01:08:46.779
a separate route to that,

01:08:46.880 --> 01:08:48.440
which will help but not

01:08:48.480 --> 01:08:49.701
necessarily solve the issue.

01:08:50.541 --> 01:08:53.341
Or we'll have two separate

01:08:53.360 --> 01:08:56.162
networks on two whole network spaces.

01:08:56.742 --> 01:08:58.082
One of those will probably

01:08:58.122 --> 01:09:00.063
be identifiable just through DNS.

01:09:00.082 --> 01:09:01.162
The other one would not.

01:09:01.912 --> 01:09:02.412
And so,

01:09:03.453 --> 01:09:04.552
and then we'll do something where

01:09:05.073 --> 01:09:06.113
we'll change default routes

01:09:06.154 --> 01:09:07.555
to a totally different IP space.

01:09:07.994 --> 01:09:09.234
Cause we talked about BGP as well.

01:09:09.255 --> 01:09:10.676
Like that's, that's, it's not a bad idea.

01:09:10.695 --> 01:09:12.936
It's just, it's just,

01:09:13.016 --> 01:09:14.457
it doesn't solve all those issues.

01:09:14.797 --> 01:09:17.118
And then two IP spaces creates new ones.

01:09:17.139 --> 01:09:17.918
So it's not like it's,

01:09:18.458 --> 01:09:19.380
it's a better situation.

01:09:19.399 --> 01:09:19.979
It's just different.

01:09:20.000 --> 01:09:22.621
And then if in those situations,

01:09:22.661 --> 01:09:23.742
if both those twenty five

01:09:23.761 --> 01:09:25.743
gig routers or if they're actual switches,

01:09:26.462 --> 01:09:27.222
if they have enough

01:09:27.262 --> 01:09:28.684
capacity for all of the

01:09:29.203 --> 01:09:30.685
hardware underneath them, obviously,

01:09:32.024 --> 01:09:33.525
you need connections to both of those,

01:09:33.565 --> 01:09:34.287
whether that's in the same

01:09:34.306 --> 01:09:35.448
VLAN or multiple VLANs.

01:09:36.569 --> 01:09:37.949
And then you have to figure

01:09:38.010 --> 01:09:39.832
out how to manage that as well.

01:09:41.774 --> 01:09:43.215
So these particular routers,

01:09:43.255 --> 01:09:45.957
you can basically just trunk the two,

01:09:46.057 --> 01:09:47.078
one hundred gig ports and

01:09:47.099 --> 01:09:48.279
then direct connect them

01:09:48.761 --> 01:09:49.662
with a two hundred gig link.

01:09:50.141 --> 01:09:50.402
Yeah.

01:09:51.422 --> 01:09:53.244
So basically all the traffic

01:09:53.284 --> 01:09:54.405
is just forwarded between

01:09:54.466 --> 01:09:56.068
both back and forward with the,

01:09:57.685 --> 01:09:58.985
multi-shazzy link aggregation.

01:09:59.506 --> 01:10:00.287
But usually only one of

01:10:00.306 --> 01:10:01.448
those impound links would

01:10:01.467 --> 01:10:02.908
be active to your provider.

01:10:03.489 --> 01:10:03.949
Not both.

01:10:04.029 --> 01:10:04.310
Yeah.

01:10:04.371 --> 01:10:04.551
Yeah.

01:10:04.591 --> 01:10:05.712
With the, with the internet.

01:10:05.731 --> 01:10:05.952
Yeah.

01:10:06.412 --> 01:10:06.632
Yeah.

01:10:07.412 --> 01:10:08.033
You have, well,

01:10:08.054 --> 01:10:09.135
you can either aggregate

01:10:09.154 --> 01:10:11.556
them or you can have like a, a backup.

01:10:12.219 --> 01:10:13.140
So you might send all the

01:10:13.279 --> 01:10:14.501
outgoing traffic through one,

01:10:14.541 --> 01:10:15.702
but then you kind of get

01:10:15.742 --> 01:10:18.684
problems with TLS when you do that.

01:10:19.125 --> 01:10:19.425
Yeah.

01:10:19.704 --> 01:10:20.024
Yeah.

01:10:20.865 --> 01:10:21.766
But anyway.

01:10:21.827 --> 01:10:23.247
Yeah.

01:10:23.307 --> 01:10:24.328
So I think we'll probably

01:10:24.349 --> 01:10:26.470
just test it separately and

01:10:26.529 --> 01:10:28.311
see if we can get it humming.

01:10:29.172 --> 01:10:29.372
Yeah.

01:10:30.073 --> 01:10:30.993
Yeah.

01:10:31.453 --> 01:10:31.613
Yeah.

01:10:31.675 --> 01:10:32.755
So I think that's,

01:10:33.315 --> 01:10:34.417
that's about it with that shit.

01:10:34.596 --> 01:10:35.077
That was enough.

01:10:35.938 --> 01:10:37.759
So now I'm behind everything else.

01:10:38.159 --> 01:10:38.920
I have so much shit to do.

01:10:39.180 --> 01:10:39.720
Is this like,

01:10:40.626 --> 01:10:41.707
You don't want that stress, man.

01:10:42.266 --> 01:10:42.766
No.

01:10:43.186 --> 01:10:45.068
This whole... I mean,

01:10:45.087 --> 01:10:46.309
we've done so much to...

01:10:47.349 --> 01:10:48.409
We've done collectively.

01:10:48.550 --> 01:10:49.591
I think you guys are in the same position.

01:10:49.631 --> 01:10:50.871
But I've done so much to not

01:10:51.411 --> 01:10:52.351
get woken up at three a.m.

01:10:52.532 --> 01:10:54.372
Like, I've done so much to do that.

01:10:55.734 --> 01:10:58.875
And... I sometimes wonder, like, I'm like,

01:10:58.994 --> 01:11:00.355
is all of the effort to not

01:11:00.395 --> 01:11:02.197
get woken up at three a.m.,

01:11:04.395 --> 01:11:07.118
worth it like maybe I just

01:11:07.137 --> 01:11:08.179
get working up at three a

01:11:08.238 --> 01:11:10.360
couple of times like no

01:11:10.421 --> 01:11:11.122
it's definitely it's

01:11:11.141 --> 01:11:12.603
definitely worth it since

01:11:12.983 --> 01:11:14.305
it's definitely worth it

01:11:14.604 --> 01:11:16.327
beforehand I was like yeah

01:11:16.667 --> 01:11:17.587
I don't know bear in mind

01:11:17.707 --> 01:11:18.769
bear in mind at the point

01:11:18.809 --> 01:11:21.351
that you're at was at my

01:11:21.492 --> 01:11:25.716
peak aws usage no

01:11:27.752 --> 01:11:29.154
when my one was at your your

01:11:29.293 --> 01:11:31.676
age we were pretty much a

01:11:31.716 --> 01:11:33.636
hundred percent aws for the

01:11:33.676 --> 01:11:35.559
first like it's stuff that

01:11:35.639 --> 01:11:37.079
wasn't on aws went in the

01:11:37.100 --> 01:11:38.980
direction of aws for that

01:11:39.001 --> 01:11:39.822
first couple of months

01:11:39.841 --> 01:11:41.182
because I was damned if I

01:11:41.203 --> 01:11:42.243
was being woken up in the

01:11:42.283 --> 01:11:43.824
middle of the night we were

01:11:43.864 --> 01:11:46.506
paying for it yeah there's

01:11:46.527 --> 01:11:47.387
a fleet of fucking

01:11:47.828 --> 01:11:49.909
yeah like there's there's

01:11:49.930 --> 01:11:51.350
some value in the in that

01:11:51.650 --> 01:11:52.770
knowing that that service

01:11:52.810 --> 01:11:53.912
is going to be active and

01:11:53.971 --> 01:11:55.693
you're not oh yeah there

01:11:55.733 --> 01:11:56.873
were definitely times when

01:11:56.974 --> 01:11:58.836
it was just like there is

01:11:58.916 --> 01:12:00.036
nothing left in the tank

01:12:00.115 --> 01:12:01.056
right now if something goes

01:12:01.117 --> 01:12:04.640
down it's yeah yeah and

01:12:04.659 --> 01:12:05.319
it's difficult right

01:12:05.340 --> 01:12:06.440
especially if you're I mean

01:12:06.480 --> 01:12:07.582
that's maybe a failure of

01:12:07.622 --> 01:12:08.862
our model right where we

01:12:09.743 --> 01:12:10.543
where we have needed

01:12:10.564 --> 01:12:11.684
additional help it's been

01:12:11.703 --> 01:12:13.284
freelancers and whatnot and

01:12:13.485 --> 01:12:14.706
it means it's harder to

01:12:14.746 --> 01:12:16.807
cover hands when there is a big

01:12:17.588 --> 01:12:18.588
there is a big out of hours

01:12:18.628 --> 01:12:21.550
problem you know there's

01:12:21.569 --> 01:12:22.270
something that you couldn't

01:12:22.291 --> 01:12:23.251
have scheduled for you

01:12:23.291 --> 01:12:25.371
couldn't have got an extra

01:12:25.390 --> 01:12:26.292
help ahead of time or

01:12:26.311 --> 01:12:28.072
planned for then you you

01:12:28.271 --> 01:12:31.073
have an inherent kind of um

01:12:31.113 --> 01:12:31.932
weakness that can only

01:12:31.974 --> 01:12:32.673
really be solved by

01:12:32.734 --> 01:12:35.454
automation or paying money

01:12:35.475 --> 01:12:36.274
to something else in this

01:12:36.295 --> 01:12:43.077
case jeff bezos so hey jeffy yeah

01:12:44.539 --> 01:12:45.820
I think I've already, like,

01:12:46.900 --> 01:12:48.322
our baby's four weeks old now.

01:12:48.481 --> 01:12:49.682
I think I've definitely,

01:12:50.222 --> 01:12:51.503
on multiple occasions,

01:12:52.003 --> 01:12:54.064
done one-handed laptop with

01:12:54.104 --> 01:12:55.905
the baby crying at, like, two a.m.

01:12:55.926 --> 01:12:56.126
For sure.

01:12:56.145 --> 01:12:56.225
Yeah.

01:12:57.886 --> 01:12:59.148
Because sometimes they just cry.

01:12:59.167 --> 01:12:59.787
They don't shut up.

01:13:00.007 --> 01:13:00.908
You just, like, got to hold them.

01:13:01.969 --> 01:13:04.489
But you also got to work.

01:13:04.510 --> 01:13:05.270
Shit's got to get done.

01:13:05.770 --> 01:13:05.930
Yeah.

01:13:06.070 --> 01:13:11.554
Yeah.

01:13:11.613 --> 01:13:11.694
Yeah.

01:13:11.713 --> 01:13:11.953
I don't know.

01:13:11.974 --> 01:13:13.015
That's just life, I suppose,

01:13:13.034 --> 01:13:13.454
at this point.

01:13:14.596 --> 01:13:15.838
Yeah, I think so.

01:13:15.859 --> 01:13:17.820
I think, I mean, and honestly it's, it's,

01:13:18.221 --> 01:13:20.064
uh, I mean, some of that,

01:13:20.083 --> 01:13:20.885
you just have to do what you do.

01:13:20.904 --> 01:13:22.947
I, I, what I found now is like,

01:13:23.087 --> 01:13:24.408
I also make dumb mistakes

01:13:24.750 --> 01:13:25.570
in those situations.

01:13:25.730 --> 01:13:26.011
That's,

01:13:26.231 --> 01:13:27.132
that's what I get more worried

01:13:27.153 --> 01:13:27.894
about is it's not

01:13:27.913 --> 01:13:28.854
necessarily being up at three,

01:13:28.895 --> 01:13:30.235
but like doing something

01:13:30.256 --> 01:13:31.398
stupid because I want the

01:13:31.597 --> 01:13:32.859
fucking alarm to go away.

01:13:33.404 --> 01:13:34.805
and you go ah I shouldn't

01:13:34.827 --> 01:13:37.971
have done that like yeah

01:13:38.011 --> 01:13:38.934
well also just like when

01:13:38.953 --> 01:13:40.136
you're really really so

01:13:40.155 --> 01:13:42.279
what we had um like I

01:13:42.340 --> 01:13:44.182
really really distinctly remember the

01:13:45.481 --> 01:13:50.546
we had an, uh, was it the aptos migration?

01:13:51.846 --> 01:13:52.426
I think we were the,

01:13:52.606 --> 01:13:53.547
I think it was actually that one.

01:13:53.587 --> 01:13:53.908
I think it was,

01:13:53.948 --> 01:13:55.009
we were the canary for the

01:13:55.069 --> 01:13:56.329
aptos migration, you know,

01:13:56.350 --> 01:13:57.270
when there was that big

01:13:57.310 --> 01:13:59.452
configuration change and they were like,

01:13:59.693 --> 01:13:59.832
oh,

01:14:00.172 --> 01:14:01.474
now is the time for everybody to move

01:14:01.493 --> 01:14:02.614
boxes as well.

01:14:03.775 --> 01:14:07.037
Um, so, uh, the spec changed.

01:14:07.177 --> 01:14:08.399
And so we took that

01:14:08.439 --> 01:14:10.560
opportunity to move off of AWS.

01:14:10.940 --> 01:14:13.502
That was when we moved the AWS boxes, um,

01:14:14.783 --> 01:14:16.645
So it had been, I don't know, August,

01:14:16.664 --> 01:14:18.444
September last year, something like that,

01:14:18.664 --> 01:14:19.564
because the Wii one was

01:14:19.585 --> 01:14:20.506
about six months old.

01:14:20.706 --> 01:14:22.246
So I felt more confident about that.

01:14:24.287 --> 01:14:25.587
And there was distinctly a

01:14:25.627 --> 01:14:27.426
point where I can't remember what it was.

01:14:27.466 --> 01:14:32.708
There was some reason we had

01:14:32.748 --> 01:14:35.550
an issue with the box sinking.

01:14:37.667 --> 01:14:38.747
so long ago I can't remember

01:14:38.787 --> 01:14:39.809
but we were essentially out

01:14:39.849 --> 01:14:40.908
we were down in production

01:14:40.948 --> 01:14:42.050
we'd already done the

01:14:42.090 --> 01:14:43.470
switcheroos and everything

01:14:43.489 --> 01:14:45.411
and I think like looking

01:14:45.451 --> 01:14:46.771
back actually what probably

01:14:46.811 --> 01:14:47.731
had happened was we got

01:14:47.792 --> 01:14:50.613
kicked um for the initial

01:14:50.653 --> 01:14:52.033
downtime you know you just

01:14:52.054 --> 01:14:53.554
get like backed off in

01:14:53.594 --> 01:14:54.654
terms of like that's how

01:14:54.715 --> 01:14:56.274
the the peering works it

01:14:56.335 --> 01:14:58.376
kicks you out um yeah and

01:14:58.395 --> 01:14:59.796
switching over the the the

01:15:01.326 --> 01:15:04.909
the DNS obviously helps you

01:15:04.930 --> 01:15:06.110
for discoverability, right?

01:15:06.150 --> 01:15:07.591
But your node is always

01:15:07.612 --> 01:15:08.872
going to be dialing out.

01:15:09.434 --> 01:15:11.095
The problem is actually that

01:15:11.195 --> 01:15:15.159
the... I remembered what the problem was.

01:15:15.520 --> 01:15:17.081
Okay, I'll get to that in a second.

01:15:17.121 --> 01:15:19.222
But your node should always be dialing out,

01:15:19.283 --> 01:15:19.503
right?

01:15:22.099 --> 01:15:23.920
And it was one of these

01:15:23.939 --> 01:15:25.460
things where just the

01:15:25.539 --> 01:15:27.600
length of time it took to sync the node,

01:15:27.740 --> 01:15:30.162
even on good hardware, was substantial,

01:15:30.221 --> 01:15:30.481
right?

01:15:31.282 --> 01:15:34.582
And so by the time it had

01:15:34.662 --> 01:15:37.444
synced and then we did the switcheroo,

01:15:37.844 --> 01:15:40.064
it was getting late in our time zone.

01:15:42.725 --> 01:15:43.805
This is a classic example of

01:15:43.905 --> 01:15:45.127
you should always wait till the morning,

01:15:45.146 --> 01:15:45.627
of course.

01:15:47.878 --> 01:15:49.439
And, but it was just like, well, look,

01:15:49.460 --> 01:15:50.400
we need to get this done.

01:15:50.640 --> 01:15:52.643
We had a nominal deadline in

01:15:52.682 --> 01:15:54.623
terms of like when we were

01:15:54.684 --> 01:15:55.585
asked to get it done by.

01:15:56.365 --> 01:15:56.945
And then, you know,

01:15:56.966 --> 01:15:58.266
of course the end result is

01:15:58.306 --> 01:15:59.228
that you're down in

01:15:59.247 --> 01:16:00.788
production at two o'clock in the morning.

01:16:02.550 --> 01:16:04.692
And I really, really, really,

01:16:04.731 --> 01:16:06.012
really distinctly remember

01:16:06.653 --> 01:16:08.113
setting an alarm for six

01:16:08.154 --> 01:16:10.015
hours after that and just going to bed.

01:16:12.090 --> 01:16:14.091
Well, some of those, like in that one,

01:16:14.891 --> 01:16:16.733
there's no slashing structure there,

01:16:16.752 --> 01:16:16.912
right?

01:16:18.113 --> 01:16:18.953
I don't think there is.

01:16:19.833 --> 01:16:21.475
No, but you're losing money, obviously.

01:16:21.515 --> 01:16:22.456
You're losing money.

01:16:22.576 --> 01:16:24.277
And you're losing goodwill.

01:16:25.158 --> 01:16:25.457
For sure.

01:16:25.658 --> 01:16:27.578
So, you know,

01:16:27.599 --> 01:16:29.119
with the SLA you've got in place,

01:16:29.140 --> 01:16:32.282
you don't really have that wiggle room.

01:16:32.521 --> 01:16:35.524
But it was one of those where, like,

01:16:38.185 --> 01:16:39.246
obviously the problem here

01:16:39.265 --> 01:16:40.766
is something obvious, right?

01:16:42.881 --> 01:16:43.743
flailing on the end of the

01:16:43.783 --> 01:16:45.684
hook is not going to help

01:16:46.123 --> 01:16:50.686
getting off the hook uh so

01:16:51.067 --> 01:16:52.768
yeah like got six hours

01:16:52.807 --> 01:16:54.788
sleep woke up immediately

01:16:54.828 --> 01:16:56.890
got a cup of coffee and

01:16:57.029 --> 01:16:58.270
solved it within five

01:16:58.310 --> 01:17:00.391
minutes like all of the all

01:17:00.412 --> 01:17:01.233
of the stuff like the sink

01:17:01.292 --> 01:17:02.453
and whatever had solved had

01:17:02.493 --> 01:17:04.354
sorted itself out the the

01:17:04.373 --> 01:17:05.534
node was by that you know

01:17:05.614 --> 01:17:06.555
it was like it was very

01:17:06.595 --> 01:17:07.416
straightforward to confirm

01:17:07.435 --> 01:17:08.556
the node was dialing out

01:17:09.197 --> 01:17:11.717
and the problem was that the

01:17:12.719 --> 01:17:14.841
at least in the Aptos data structure,

01:17:14.961 --> 01:17:17.362
a VFN and a validator node are very,

01:17:17.381 --> 01:17:17.983
very similar.

01:17:18.002 --> 01:17:19.463
They have one or two

01:17:19.503 --> 01:17:20.244
additional pieces of

01:17:20.305 --> 01:17:21.525
metadata and a little bit

01:17:21.626 --> 01:17:24.648
of... There's an additional bit of state,

01:17:24.688 --> 01:17:26.009
isn't there, in the state store,

01:17:26.809 --> 01:17:27.649
but it's very small

01:17:27.710 --> 01:17:30.372
compared to the main database, like,

01:17:35.086 --> 01:17:35.886
Small.

01:17:35.966 --> 01:17:37.068
A gigabyte or two or something?

01:17:37.087 --> 01:17:38.189
It's pretty small, isn't it?

01:17:40.951 --> 01:17:41.530
And there was basically a

01:17:41.551 --> 01:17:42.591
misconfiguration that meant

01:17:42.612 --> 01:17:43.552
that that state couldn't be

01:17:43.573 --> 01:17:44.694
picked up by the runtime.

01:17:45.213 --> 01:17:45.654
That was it.

01:17:46.114 --> 01:17:46.595
It was there.

01:17:47.315 --> 01:17:48.016
And that was the thing you

01:17:48.037 --> 01:17:48.657
tear your hair out.

01:17:48.777 --> 01:17:49.917
You look on the thing and you're like,

01:17:49.957 --> 01:17:51.059
it's mounted, it's there.

01:17:51.599 --> 01:17:52.119
What the fuck?

01:17:52.220 --> 01:17:53.720
And you're like, oh, right, yeah,

01:17:53.780 --> 01:17:54.742
but for this reason,

01:17:54.822 --> 01:17:55.802
it's not being picked up.

01:17:56.243 --> 01:17:56.724
And it was like a,

01:17:56.804 --> 01:17:58.324
it was like a fifteen minute fix.

01:17:58.444 --> 01:17:59.966
And you're like, it wasn't even like,

01:18:00.646 --> 01:18:01.707
and most of that fifteen

01:18:01.747 --> 01:18:03.269
minutes was thinking, oh, right,

01:18:03.309 --> 01:18:03.988
this is why.

01:18:05.189 --> 01:18:06.911
how do you resolve it yeah

01:18:07.051 --> 01:18:08.451
like rather than like if

01:18:08.470 --> 01:18:09.612
you if you're kind of the

01:18:09.671 --> 01:18:11.112
actual and you know this is

01:18:11.131 --> 01:18:12.512
the thing that jack talked

01:18:12.533 --> 01:18:14.873
about before um a very long

01:18:14.913 --> 01:18:15.974
time ago on the podcast

01:18:16.073 --> 01:18:18.095
about um a design decision

01:18:18.135 --> 01:18:21.235
within slashing in um in

01:18:21.256 --> 01:18:22.456
cosmos and tendermint which

01:18:22.496 --> 01:18:24.716
is like the the default time

01:18:26.140 --> 01:18:27.902
which is suggested as whatever it is, like,

01:18:27.942 --> 01:18:28.722
thirteen hours,

01:18:28.802 --> 01:18:30.524
because the idea is if

01:18:30.545 --> 01:18:31.524
something's bad enough,

01:18:31.585 --> 01:18:33.226
a team should be able to just go to bed,

01:18:33.287 --> 01:18:34.167
get some sleep,

01:18:34.207 --> 01:18:35.188
and then attack it with

01:18:35.228 --> 01:18:37.350
a... I don't know.

01:18:38.051 --> 01:18:39.532
I always took that at face

01:18:39.572 --> 01:18:41.092
value as something that

01:18:41.153 --> 01:18:42.514
Jack said that was actually

01:18:42.975 --> 01:18:43.734
done by design.

01:18:43.755 --> 01:18:44.735
I don't know if that's true,

01:18:44.756 --> 01:18:47.278
but I kind of like that idea.

01:18:47.337 --> 01:18:51.541
I think for node operations

01:18:51.601 --> 01:18:53.783
where people have been up,

01:18:53.882 --> 01:18:54.484
if this were...

01:18:55.850 --> 01:18:56.430
if this were like a

01:18:56.490 --> 01:18:58.032
consortium blockchain or it

01:18:58.072 --> 01:18:59.313
were like infrastructure

01:18:59.353 --> 01:19:01.293
paid for by a client, the rules are just,

01:19:02.114 --> 01:19:02.914
you know, completely different.

01:19:02.935 --> 01:19:04.456
If somebody pays you for two

01:19:04.496 --> 01:19:06.556
years with a fixed, um,

01:19:06.856 --> 01:19:07.898
with a fixed fee and says

01:19:07.938 --> 01:19:08.618
at the end of two years,

01:19:08.658 --> 01:19:10.099
we're going to discontinue this contract.

01:19:10.139 --> 01:19:12.520
If the SLA is on that, you know, you just,

01:19:12.761 --> 01:19:13.780
you just go out and you pay

01:19:13.801 --> 01:19:14.702
for people to cover all the

01:19:14.721 --> 01:19:15.903
shifts and you do a

01:19:15.962 --> 01:19:17.023
particular service model.

01:19:17.063 --> 01:19:17.604
It's just a very,

01:19:17.644 --> 01:19:18.463
very different environment.

01:19:18.503 --> 01:19:19.685
But, you know,

01:19:19.704 --> 01:19:20.625
for something permissionless

01:19:20.645 --> 01:19:22.145
that you can't really plan

01:19:22.185 --> 01:19:23.447
for or whatever, it's like,

01:19:25.324 --> 01:19:26.710
Yeah, like what are you going to do?

01:19:26.770 --> 01:19:27.752
Yeah, exactly.

01:19:27.773 --> 01:19:28.737
It's super, super different.

01:19:28.777 --> 01:19:30.582
I think you have to plan around...

01:19:31.680 --> 01:19:32.740
quite different operating

01:19:32.780 --> 01:19:34.181
model for most of your providers.

01:19:35.601 --> 01:19:36.042
Well, not only that,

01:19:36.063 --> 01:19:37.804
I think the whole point of

01:19:37.823 --> 01:19:39.824
this is to build resilient networks.

01:19:40.145 --> 01:19:42.345
So I think part of that

01:19:42.386 --> 01:19:44.907
resiliency is... You're

01:19:44.926 --> 01:19:45.927
only as strong as the

01:19:45.988 --> 01:19:47.889
thirty-three percent of

01:19:47.929 --> 01:19:49.170
your weakest validators.

01:19:49.189 --> 01:19:49.310
Right.

01:19:49.329 --> 01:19:50.609
That's one of the reasons

01:19:50.630 --> 01:19:52.490
for appropriate VP percentages,

01:19:54.051 --> 01:19:55.132
but it's meant to be resilient.

01:19:55.193 --> 01:19:55.853
And same thing,

01:19:55.912 --> 01:19:57.514
some of the stuff we talk

01:19:57.533 --> 01:19:58.534
about on the Chainlink side, too,

01:19:58.555 --> 01:19:58.954
is because

01:19:59.475 --> 01:19:59.654
you know,

01:19:59.675 --> 01:20:01.516
we're providing value and data in there,

01:20:01.556 --> 01:20:02.497
but we're one of many.

01:20:03.018 --> 01:20:04.578
And so the idea is that,

01:20:04.738 --> 01:20:05.659
that the whole purpose of

01:20:05.720 --> 01:20:08.101
this is to build networks that, that,

01:20:08.863 --> 01:20:09.703
that are resilient to this

01:20:09.724 --> 01:20:10.984
type of shit and shit happens.

01:20:11.305 --> 01:20:14.466
Like my example in years and, you know,

01:20:14.487 --> 01:20:15.748
there's, there's a ton of reasons why.

01:20:15.769 --> 01:20:18.230
I know, I know.

01:20:18.251 --> 01:20:19.171
Null's buggered off from the

01:20:19.211 --> 01:20:20.792
podcast for a moment there.

01:20:20.832 --> 01:20:21.432
So he doesn't have to

01:20:21.453 --> 01:20:23.014
volunteer a fuck up that

01:20:23.074 --> 01:20:24.215
big wolves blockchain have made.

01:20:25.076 --> 01:20:25.457
Exactly.

01:20:25.476 --> 01:20:26.497
It's like, it's like,

01:20:26.516 --> 01:20:27.018
I'm getting out of here.

01:20:27.511 --> 01:20:28.672
It's like we're sharing

01:20:28.712 --> 01:20:30.414
stories that make us look

01:20:30.454 --> 01:20:32.695
like battle-hardened organizations,

01:20:32.777 --> 01:20:34.177
but maybe ones that make mistakes.

01:20:34.257 --> 01:20:34.457
Well,

01:20:34.778 --> 01:20:36.078
big balls watching don't make fucking

01:20:36.099 --> 01:20:36.579
mistakes.

01:20:37.501 --> 01:20:38.921
Do we have a Kujira archive note yet?

01:20:39.082 --> 01:20:42.024
No, I don't think we do.

01:20:42.043 --> 01:20:42.145
Yeah.

01:20:42.164 --> 01:20:43.405
No, it's meant to be resilient.

01:20:43.445 --> 01:20:45.287
So I think it's fine.

01:20:45.606 --> 01:20:48.029
Like the whole point of that is, I think,

01:20:48.069 --> 01:20:48.989
and I think Thirteen Hours

01:20:49.010 --> 01:20:50.030
is a pretty good, I mean, we were at,

01:20:50.110 --> 01:20:52.793
I think overall we were at nine or ten.

01:20:55.179 --> 01:20:56.199
that was just because if it

01:20:56.220 --> 01:20:57.039
was during the day it would

01:20:57.060 --> 01:20:57.539
have been it would have

01:20:57.560 --> 01:20:58.621
been totally different time

01:20:58.801 --> 01:20:59.661
but it happened like at the

01:20:59.680 --> 01:21:00.462
worst possible time was

01:21:00.481 --> 01:21:01.462
like seven p.m which was

01:21:01.603 --> 01:21:02.643
which was actually seven or

01:21:02.662 --> 01:21:03.384
eight p.m which was really

01:21:03.404 --> 01:21:04.684
so similar similar to what

01:21:04.703 --> 01:21:06.265
happened to us like yeah

01:21:06.284 --> 01:21:07.225
you go out at seven or

01:21:07.286 --> 01:21:09.546
eight p.m and you're in

01:21:09.587 --> 01:21:12.307
that kind of depending on

01:21:12.328 --> 01:21:13.309
where your head's at and

01:21:13.349 --> 01:21:14.590
how busy your day has been

01:21:14.630 --> 01:21:15.770
like you could be already

01:21:15.829 --> 01:21:17.810
burned from the day I was

01:21:17.952 --> 01:21:18.752
for sure and you'll keep

01:21:18.771 --> 01:21:19.693
bashing your head against

01:21:19.752 --> 01:21:20.872
it till two in the morning

01:21:20.912 --> 01:21:22.634
and just we we were so sure

01:21:22.654 --> 01:21:23.534
that it was an spf

01:21:24.405 --> 01:21:25.506
failure meaning like that

01:21:25.567 --> 01:21:27.627
copper fiber transition and

01:21:27.648 --> 01:21:29.248
like everything made sense

01:21:29.488 --> 01:21:30.909
around that like it totally

01:21:30.930 --> 01:21:31.890
made sense in terms of was

01:21:31.949 --> 01:21:32.390
and then we had a

01:21:32.430 --> 01:21:34.271
realization that we had

01:21:34.311 --> 01:21:35.393
something tagged on a

01:21:35.493 --> 01:21:36.653
switch um in terms of a

01:21:36.712 --> 01:21:38.173
switch port uh that was

01:21:38.234 --> 01:21:39.114
outdated and then we had

01:21:39.154 --> 01:21:40.034
another switch that had

01:21:40.055 --> 01:21:40.775
that something else that

01:21:40.796 --> 01:21:41.536
was tagged and blah blah

01:21:41.555 --> 01:21:42.777
blah and we started like

01:21:42.817 --> 01:21:43.716
let's double check this and

01:21:43.737 --> 01:21:44.658
then once we started to

01:21:44.677 --> 01:21:45.297
realize what the hell was

01:21:45.337 --> 01:21:47.500
going on uh we recognized

01:21:47.520 --> 01:21:48.220
that what we thought was

01:21:48.239 --> 01:21:49.100
the issue was actually not

01:21:49.121 --> 01:21:50.942
the issue and we probably

01:21:50.981 --> 01:21:52.443
spent two or three hours

01:21:53.496 --> 01:21:54.416
talking about how to solve

01:21:54.456 --> 01:21:55.377
that problem and what we

01:21:55.417 --> 01:21:58.217
could do with that and how to verify it.

01:21:58.858 --> 01:21:59.698
And then once we got to that

01:21:59.759 --> 01:22:00.578
point of realizing that

01:22:00.599 --> 01:22:01.338
what the problem we were

01:22:01.378 --> 01:22:02.760
solving was actually changed,

01:22:02.920 --> 01:22:04.300
then it's like we started all over,

01:22:04.921 --> 01:22:05.081
right?

01:22:05.121 --> 01:22:05.501
Now we're at,

01:22:05.961 --> 01:22:07.061
now we're at this problem and

01:22:07.101 --> 01:22:08.261
now we're three hours later and it's,

01:22:08.822 --> 01:22:10.443
it's one right.

01:22:10.502 --> 01:22:10.863
Eastern.

01:22:11.182 --> 01:22:12.764
And now I'm like, fuck it.

01:22:12.804 --> 01:22:13.064
Now I get,

01:22:13.184 --> 01:22:14.765
I get trying to bring up the

01:22:14.805 --> 01:22:16.265
brain power to like, you know, make that,

01:22:16.484 --> 01:22:16.805
you know,

01:22:16.826 --> 01:22:18.046
go through that whole process again.

01:22:18.086 --> 01:22:19.247
Like, okay, how do we start with this?

01:22:19.287 --> 01:22:20.667
And like, okay, what can we do about this?

01:22:20.707 --> 01:22:20.966
And what,

01:22:20.987 --> 01:22:22.387
where's the umbilical structure working?

01:22:23.173 --> 01:22:23.394
you know,

01:22:23.413 --> 01:22:24.894
like you start all over and that's,

01:22:24.994 --> 01:22:25.835
that's really when, like,

01:22:25.935 --> 01:22:27.015
that's when the brain's like, all right,

01:22:27.034 --> 01:22:27.295
fuck this.

01:22:27.314 --> 01:22:30.655
Like, you're just like, all right,

01:22:30.695 --> 01:22:31.496
it's too much.

01:22:32.436 --> 01:22:34.256
Um, and then the point,

01:22:34.277 --> 01:22:35.136
and then like the point was like,

01:22:35.716 --> 01:22:36.738
we thought the stuff that we had,

01:22:36.757 --> 01:22:37.438
I knew we had probably, probably,

01:22:37.457 --> 01:22:38.318
probably, probably, probably, probably,

01:22:38.337 --> 01:22:39.137
probably, probably, probably, probably,

01:22:39.158 --> 01:22:39.497
probably, probably,

01:22:39.518 --> 01:22:40.019
hours or twelve hours of,

01:22:40.038 --> 01:22:40.679
of actual downtime.

01:22:40.698 --> 01:22:41.658
And some of the ones I, again,

01:22:41.679 --> 01:22:42.458
I wasn't worried about.

01:22:42.479 --> 01:22:44.359
And it, it would have, our actions,

01:22:44.439 --> 01:22:45.100
my actions would have been

01:22:45.119 --> 01:22:46.220
different if the,

01:22:46.279 --> 01:22:47.541
if the consequences were different.

01:22:48.001 --> 01:22:49.081
Like, we, I would have done some,

01:22:49.100 --> 01:22:49.900
we would have done a lot of,

01:22:49.921 --> 01:22:50.560
I would have spent time

01:22:50.621 --> 01:22:52.601
elsewhere versus where we spent, um,

01:22:54.167 --> 01:22:55.247
if we had like real flashing

01:22:55.287 --> 01:22:56.088
issues or things like that,

01:22:56.109 --> 01:22:56.828
where we felt like we were

01:22:56.849 --> 01:22:58.310
going to have a problem with it.

01:22:58.329 --> 01:22:59.692
But, but yeah, that's, you know,

01:23:00.091 --> 01:23:00.551
it is what it is.

01:23:00.572 --> 01:23:03.213
The hell are you going to do?

01:23:03.314 --> 01:23:03.935
It's a,

01:23:03.975 --> 01:23:05.636
as your business partner once said to me,

01:23:05.737 --> 01:23:06.957
when I have my phone stolen,

01:23:07.278 --> 01:23:08.658
it's a cheap lesson in the

01:23:08.679 --> 01:23:09.699
grand scheme of things.

01:23:09.920 --> 01:23:11.261
This was probably a cheap lesson.

01:23:11.400 --> 01:23:11.600
Yeah,

01:23:11.621 --> 01:23:12.921
I guess this was probably a cheap

01:23:12.962 --> 01:23:14.444
lesson in the grand scheme of things.

01:23:15.404 --> 01:23:16.164
I don't even know, dude.

01:23:16.926 --> 01:23:19.408
Well, when you see him tell him,

01:23:19.427 --> 01:23:20.228
that's what I said.

01:23:23.929 --> 01:23:24.729
I'm sorry you didn't get any

01:23:24.789 --> 01:23:25.869
sleep and you had a

01:23:26.250 --> 01:23:28.490
monumentally stressful thirteen hours,

01:23:28.551 --> 01:23:30.390
but it's a cheap lesson.

01:23:30.990 --> 01:23:33.011
Do you know who told me that?

01:23:33.072 --> 01:23:34.292
Do you know who told me that?

01:23:34.653 --> 01:23:35.752
Do you know who told me that?

01:23:37.172 --> 01:23:37.793
Actually, you're right.

01:23:37.894 --> 01:23:38.993
It was an incredibly cheap

01:23:39.014 --> 01:23:40.793
lesson in the grand scheme of things.

01:23:41.274 --> 01:23:42.875
Everything was back up.

01:23:42.914 --> 01:23:43.555
The problem with that,

01:23:44.475 --> 01:23:45.615
it's a good one-liner, right?

01:23:45.655 --> 01:23:46.996
But the problem is there's

01:23:47.036 --> 01:23:48.617
very few things in life

01:23:48.657 --> 01:23:49.896
where when you take a step

01:23:49.917 --> 01:23:51.257
back and you have a little

01:23:51.278 --> 01:23:51.877
think about it,

01:23:52.730 --> 01:23:54.011
you know,

01:23:54.051 --> 01:23:55.771
pretty much any situation where

01:23:55.811 --> 01:23:56.733
somebody doesn't come to

01:23:56.872 --> 01:23:59.132
physical harm in the

01:23:59.172 --> 01:24:00.694
grander scheme of things, you're like,

01:24:00.793 --> 01:24:00.974
well,

01:24:01.314 --> 01:24:04.354
I guess it could be seen as a cheap

01:24:04.454 --> 01:24:06.676
lesson given the potential

01:24:06.716 --> 01:24:08.976
worst case scenario or

01:24:09.016 --> 01:24:10.078
possibly I think too much

01:24:10.097 --> 01:24:11.757
about worst case scenarios, but yeah.

01:24:11.778 --> 01:24:15.899
Yeah.

01:24:16.380 --> 01:24:17.100
Yeah.

01:24:17.600 --> 01:24:19.121
It still sucks though, doesn't it?

01:24:21.208 --> 01:24:21.649
That's it, man.

01:24:21.668 --> 01:24:22.489
That's all.

01:24:22.850 --> 01:24:24.332
What else do we have to report this week?

01:24:25.252 --> 01:24:25.493
Well,

01:24:25.672 --> 01:24:29.136
Noel's presumably dealing with baby

01:24:29.296 --> 01:24:30.497
in some way, shape, or form.

01:24:30.617 --> 01:24:31.037
I assume so.

01:24:31.056 --> 01:24:32.158
Yeah, that's probably pretty fair.

01:24:32.518 --> 01:24:33.719
Or maybe the pool's

01:24:33.779 --> 01:24:35.261
overflowing and flooded his house.

01:24:36.041 --> 01:24:38.462
I did watch Samurai Cop,

01:24:38.523 --> 01:24:40.845
which was a total fucking mistake.

01:24:40.864 --> 01:24:41.886
That's actually like a,

01:24:41.966 --> 01:24:42.726
it reminded me of like

01:24:44.326 --> 01:24:45.185
be too young for this but

01:24:45.367 --> 01:24:46.386
all right well this is a

01:24:46.466 --> 01:24:47.688
u.s thing but it reminded

01:24:47.728 --> 01:24:50.269
like a like a nineteen late

01:24:50.389 --> 01:24:51.689
nineties early two thousand

01:24:51.789 --> 01:24:52.729
cinemax film you know what

01:24:52.750 --> 01:24:54.551
that means it's like it's

01:24:54.591 --> 01:24:56.171
like that the really shitty

01:24:56.192 --> 01:24:57.731
movie channel right yeah

01:24:57.771 --> 01:24:58.472
well he's like a secondary

01:24:58.493 --> 01:24:59.472
channel to hbo but it would

01:24:59.493 --> 01:25:00.873
be like it was kind of a

01:25:00.894 --> 01:25:02.234
skin flick number one

01:25:03.015 --> 01:25:03.654
there's a lot of nudity in

01:25:03.675 --> 01:25:08.997
that flick um bad fighting awful lines

01:25:10.336 --> 01:25:12.395
right I mean it was it was a

01:25:12.555 --> 01:25:13.636
terrible movie that I'll

01:25:13.716 --> 01:25:16.216
end up watching again

01:25:16.296 --> 01:25:17.637
exactly as promised that's

01:25:17.957 --> 01:25:20.698
it um I there was um we

01:25:20.717 --> 01:25:21.318
didn't have obviously

01:25:21.358 --> 01:25:22.378
cinemax in this country but

01:25:22.399 --> 01:25:22.998
there are a couple of

01:25:23.019 --> 01:25:26.059
different ones that um it

01:25:26.078 --> 01:25:27.720
carried those sorts of

01:25:27.800 --> 01:25:29.520
movies and and also you

01:25:29.539 --> 01:25:32.581
know obviously at weird times um

01:25:33.293 --> 01:25:35.795
and I always remember that there were,

01:25:35.895 --> 01:25:38.135
there was a very healthy trade in,

01:25:38.475 --> 01:25:40.175
I assume, um,

01:25:40.194 --> 01:25:41.695
these would be the similar sort of thing.

01:25:42.095 --> 01:25:44.355
Um, you know, when there's like, uh,

01:25:45.356 --> 01:25:46.077
and this is like pre

01:25:46.157 --> 01:25:47.856
YouTube and pre people be

01:25:47.877 --> 01:25:48.877
able to kind of meme and

01:25:48.896 --> 01:25:50.438
share these things online

01:25:50.478 --> 01:25:52.037
so that the audience was

01:25:52.057 --> 01:25:56.118
like necessarily very limited, like those,

01:25:56.498 --> 01:25:56.559
um,

01:25:58.685 --> 01:25:59.645
there's those ones that

01:25:59.706 --> 01:26:00.605
where it's like like

01:26:00.645 --> 01:26:02.145
predator has come out right

01:26:02.265 --> 01:26:03.867
and they're like right make

01:26:03.926 --> 01:26:06.028
predator but like your

01:26:06.087 --> 01:26:07.847
budget is like ten thousand

01:26:07.868 --> 01:26:09.908
dollars or whatever and

01:26:09.929 --> 01:26:10.909
they'll be called shit like

01:26:11.010 --> 01:26:13.029
code red or like command

01:26:13.090 --> 01:26:14.430
decision or something like

01:26:14.470 --> 01:26:16.652
that and um although I

01:26:16.692 --> 01:26:17.591
realize actually now I've

01:26:17.612 --> 01:26:19.393
said command decision I

01:26:19.432 --> 01:26:20.493
think that was in my head

01:26:20.533 --> 01:26:22.953
because about eighty hundred

01:26:24.274 --> 01:26:25.895
episodes of this podcast ago

01:26:25.996 --> 01:26:27.478
I think I used the phrase

01:26:27.537 --> 01:26:29.259
command decision as a joke

01:26:29.298 --> 01:26:31.020
to open the podcast anyway

01:26:31.881 --> 01:26:33.282
um they always had names

01:26:33.322 --> 01:26:34.203
like that maybe that was

01:26:34.222 --> 01:26:35.203
maybe it was actually a

01:26:35.243 --> 01:26:36.204
film and that was why it

01:26:36.225 --> 01:26:38.667
was in my head could be I

01:26:38.686 --> 01:26:40.488
think I think actually code

01:26:40.547 --> 01:26:42.109
red was literally the

01:26:42.149 --> 01:26:43.189
predator knockoff

01:26:44.030 --> 01:26:46.172
potentially that kind of brings that shit

01:26:46.899 --> 01:26:49.961
kind of rings a bell um but

01:26:50.640 --> 01:26:51.541
oh man I watched some

01:26:51.740 --> 01:26:53.301
absolutely dreadful stuff

01:26:53.501 --> 01:26:54.341
and I was definitely

01:26:54.822 --> 01:26:56.141
definitely gave myself some

01:26:56.181 --> 01:26:57.282
nightmares from it as well

01:26:58.042 --> 01:27:02.243
at that age oh yeah like I

01:27:02.262 --> 01:27:04.384
couldn't even couldn't even

01:27:04.423 --> 01:27:07.664
number them I was thousands

01:27:07.704 --> 01:27:09.225
of bad knockoff movies

01:27:09.761 --> 01:27:11.521
Well, there was a period of time where the,

01:27:11.582 --> 01:27:13.442
because it was the good sci-fi,

01:27:14.042 --> 01:27:15.542
there's the good sci-fi channel,

01:27:16.122 --> 01:27:17.184
which was the, I guess,

01:27:17.203 --> 01:27:18.644
like the American Canadian one.

01:27:19.364 --> 01:27:20.664
And they didn't syndicate

01:27:20.744 --> 01:27:23.905
everything to the UK sci-fi channel,

01:27:23.985 --> 01:27:25.327
which was maddening.

01:27:26.367 --> 01:27:27.847
um because I remember like

01:27:27.868 --> 01:27:28.728
do you remember when and

01:27:28.768 --> 01:27:29.408
there's like the battle

01:27:29.427 --> 01:27:30.788
star galactica reboot that

01:27:30.828 --> 01:27:31.689
was actually very good yeah

01:27:31.909 --> 01:27:33.729
that was a good show um and

01:27:33.789 --> 01:27:34.671
it just wasn't fucking

01:27:34.690 --> 01:27:37.731
showing over it was shown

01:27:37.752 --> 01:27:38.613
in bits and pieces but it

01:27:38.632 --> 01:27:39.533
wasn't shown in a place I

01:27:39.552 --> 01:27:41.134
could get to it so do you

01:27:41.173 --> 01:27:43.335
remember um I remember

01:27:43.454 --> 01:27:44.375
mainly using this apart

01:27:44.395 --> 01:27:45.855
from this for for for

01:27:45.895 --> 01:27:46.956
watching the simpsons but

01:27:47.055 --> 01:27:48.476
do you remember surf the channel

01:27:49.965 --> 01:27:52.985
no it was like uh it was

01:27:53.006 --> 01:27:56.907
like a non-downloading just

01:27:56.966 --> 01:28:00.868
like super lo-fi streaming

01:28:01.887 --> 01:28:02.807
thing where people just

01:28:02.828 --> 01:28:03.847
like smashed they just

01:28:03.887 --> 01:28:05.349
smashed up a bunch of like

01:28:06.088 --> 01:28:08.088
mirror links and stuff to

01:28:08.109 --> 01:28:10.250
just like stream yeah

01:28:10.270 --> 01:28:10.970
straight up stream to your

01:28:10.989 --> 01:28:12.689
browser like random like

01:28:12.710 --> 01:28:15.430
random stuff usually tv shows and um

01:28:17.365 --> 01:28:18.947
I think I obviously watched

01:28:18.966 --> 01:28:21.368
a bunch of absolutely terrible films,

01:28:21.429 --> 01:28:22.588
potato quality on there,

01:28:23.050 --> 01:28:24.411
but also pretty much the

01:28:24.430 --> 01:28:25.811
whole about Star Galactica,

01:28:26.832 --> 01:28:27.893
the new series that like

01:28:29.033 --> 01:28:32.055
three hundred pixels wide or whatever.

01:28:32.076 --> 01:28:32.876
So there's loads of stuff

01:28:32.895 --> 01:28:33.877
that happened in that TV

01:28:33.917 --> 01:28:36.198
series where I got the audio,

01:28:36.257 --> 01:28:36.838
but I was like,

01:28:38.091 --> 01:28:39.153
why it's squinting at a

01:28:39.172 --> 01:28:40.515
laptop screen like it's

01:28:40.534 --> 01:28:41.695
like this wow look at this

01:28:41.735 --> 01:28:42.877
amazing space battle I'm

01:28:42.896 --> 01:28:45.099
like huh what what's going

01:28:45.118 --> 01:28:46.740
on it's just flat it's just

01:28:46.761 --> 01:28:47.622
a little flashing is all

01:28:47.641 --> 01:28:49.423
that's happening yeah I

01:28:49.444 --> 01:28:50.364
gotta go back and re-watch

01:28:50.385 --> 01:28:51.305
that shit at some point

01:28:51.386 --> 01:28:52.186
like I should go back and

01:28:52.207 --> 01:28:53.108
watch that that was a good

01:28:53.127 --> 01:28:54.309
show actually you must be

01:28:54.328 --> 01:28:55.470
able to just like buy the

01:28:55.489 --> 01:28:57.452
whole lot of like five

01:28:57.471 --> 01:28:58.372
dollars or something now

01:28:58.393 --> 01:28:59.234
you'd have thought right

01:29:00.435 --> 01:29:02.317
It was a few years ago, right?

01:29:02.556 --> 01:29:04.418
I remember that ending sucking, though.

01:29:05.439 --> 01:29:06.301
It was getting cancelled and

01:29:06.320 --> 01:29:08.042
they just fucking chopped

01:29:08.082 --> 01:29:08.962
it off or something.

01:29:09.002 --> 01:29:10.604
I remember something like that.

01:29:11.185 --> 01:29:12.126
They ran out of steam.

01:29:12.565 --> 01:29:13.506
I think the problem was that

01:29:13.527 --> 01:29:16.189
after season three, whenever it was,

01:29:16.208 --> 01:29:17.029
they won the Emmy.

01:29:20.033 --> 01:29:21.094
I think they just didn't

01:29:21.373 --> 01:29:22.335
know where to go from that.

01:29:22.354 --> 01:29:23.074
They were like, oh,

01:29:23.114 --> 01:29:24.317
we've broken outside of

01:29:24.396 --> 01:29:26.078
sci-fi and into this new area.

01:29:26.358 --> 01:29:26.639
And then...

01:29:28.091 --> 01:29:28.891
Yeah, to be fair,

01:29:28.912 --> 01:29:32.253
was it season... Whichever

01:29:32.314 --> 01:29:34.094
was the series after the

01:29:34.113 --> 01:29:34.875
bit where they're stuck on

01:29:34.914 --> 01:29:36.454
the planet and they get occupied?

01:29:36.475 --> 01:29:38.515
Oh, right.

01:29:38.555 --> 01:29:40.077
The one after that where

01:29:40.096 --> 01:29:45.219
they had the reprisals, that was brutal.

01:29:45.658 --> 01:29:47.079
The beginning of that series

01:29:47.439 --> 01:29:49.020
was absolutely insane.

01:29:49.079 --> 01:29:52.902
That was some of the best sci-fi, I think.

01:29:52.962 --> 01:29:53.822
Were they still on the planet?

01:29:55.122 --> 01:29:56.904
No, when they got off the planet.

01:29:56.944 --> 01:29:57.604
Do you remember that bit?

01:29:58.268 --> 01:29:59.530
I don't know.

01:29:59.551 --> 01:29:59.752
Yeah,

01:29:59.771 --> 01:30:00.935
there's like a whole bunch of stuff

01:30:00.954 --> 01:30:01.716
where they're just like...