WEBVTT 00:00:00.109 --> 00:00:01.189 No. 00:00:01.209 --> 00:00:02.690 Do we even have anything figured out? 00:00:04.650 --> 00:00:05.192 What do you mean? 00:00:05.272 --> 00:00:05.692 For what? 00:00:06.152 --> 00:00:06.572 This show? 00:00:08.653 --> 00:00:09.493 Absolutely not. 00:00:25.893 --> 00:00:27.894 so I guess in the absence of 00:00:28.074 --> 00:00:29.635 uh the fray it's down to me 00:00:29.675 --> 00:00:31.577 to welcome everybody to the 00:00:31.597 --> 00:00:35.039 game of nodes so welcome uh 00:00:35.058 --> 00:00:37.680 coming to you today live uh 00:00:37.801 --> 00:00:39.902 usurper and null we are 00:00:40.753 --> 00:00:41.575 I'm early morning. 00:00:42.595 --> 00:00:44.036 Serp's late in the afternoon 00:00:44.218 --> 00:00:45.259 or mid-afternoon. 00:00:46.820 --> 00:00:47.780 He's having a nice wine. 00:00:47.881 --> 00:00:49.902 I'm having a fucking coffee. 00:00:50.143 --> 00:00:52.646 Trying to kickstart the day. 00:00:53.366 --> 00:00:54.747 I'm done with my baby duties 00:00:55.628 --> 00:00:56.409 for the early part. 00:00:56.939 --> 00:00:57.880 We both have. 00:00:57.939 --> 00:00:59.220 We got shots yesterday. 00:01:02.262 --> 00:01:03.061 Actually, it was interesting. 00:01:03.822 --> 00:01:04.623 We went to the doctor. 00:01:05.123 --> 00:01:06.483 The doctor says, all right, 00:01:06.543 --> 00:01:07.462 you get your vaccines. 00:01:07.623 --> 00:01:09.343 Yeah, we give vaccines over here. 00:01:09.364 --> 00:01:10.864 We're not like flat earthers or anything. 00:01:10.984 --> 00:01:13.706 So we're getting damn shots. 00:01:15.725 --> 00:01:19.087 Yeah, so the nurse is like, you know, 00:01:19.186 --> 00:01:21.028 all babies take this stuff differently. 00:01:22.890 --> 00:01:23.890 uh good chance he's going to 00:01:23.909 --> 00:01:25.031 be like irritable for the 00:01:25.070 --> 00:01:26.792 rest of the day you know 00:01:26.912 --> 00:01:28.453 right et cetera et cetera 00:01:28.533 --> 00:01:30.073 small fever maybe some 00:01:30.114 --> 00:01:31.314 reaction yeah exactly like 00:01:31.334 --> 00:01:32.555 keep an eye on his heat so 00:01:32.615 --> 00:01:33.637 like every thirty seconds 00:01:33.656 --> 00:01:34.397 we've got a little oh like 00:01:35.218 --> 00:01:37.179 that you mean the little 00:01:37.218 --> 00:01:38.299 laser thing or like it rubs 00:01:38.319 --> 00:01:39.240 this yeah yeah like the 00:01:39.281 --> 00:01:41.441 little infrared thermometer 00:01:41.481 --> 00:01:42.302 thing that's like click in 00:01:42.322 --> 00:01:44.164 the face yeah like scanning 00:01:44.203 --> 00:01:47.266 his barcode every thirty seconds um 00:01:48.271 --> 00:01:51.334 And, uh, anyways, so we got the shots, 00:01:51.793 --> 00:01:53.375 which were pretty hilarious. 00:01:53.635 --> 00:01:55.856 Um, my partner, she's like, 00:01:55.917 --> 00:01:57.698 I got to go outside. 00:01:57.739 --> 00:02:00.120 So I couldn't watch it. 00:02:00.340 --> 00:02:01.421 And I mean, you don't want to see your, 00:02:01.742 --> 00:02:02.942 your little bundle of joy, 00:02:03.003 --> 00:02:04.224 like losing shit. 00:02:04.263 --> 00:02:04.524 Right. 00:02:04.944 --> 00:02:05.905 Yeah. 00:02:06.064 --> 00:02:06.865 And, um, I mean, 00:02:06.906 --> 00:02:08.026 they do go pretty hard too. 00:02:08.546 --> 00:02:10.788 And so, you know, there's one in each, 00:02:10.848 --> 00:02:11.830 there's an oral one and one 00:02:11.870 --> 00:02:13.091 in each leg for this round. 00:02:13.110 --> 00:02:13.311 Right. 00:02:13.570 --> 00:02:15.211 We didn't get the meningococcal one yet. 00:02:15.252 --> 00:02:15.772 Cause, uh, 00:02:16.367 --> 00:02:17.108 It's a little bit early. 00:02:17.628 --> 00:02:20.090 Yeah. 00:02:20.110 --> 00:02:20.651 So anyway, 00:02:22.771 --> 00:02:24.393 you stab them in the leg with this thing, 00:02:24.413 --> 00:02:24.612 man. 00:02:24.633 --> 00:02:26.114 They go fucking bright red. 00:02:26.454 --> 00:02:27.835 They stop breathing for a second, 00:02:28.074 --> 00:02:33.157 and then they just go ape shit. 00:02:33.217 --> 00:02:34.438 So she was standing outside the door. 00:02:34.598 --> 00:02:35.519 Did she come running in? 00:02:36.060 --> 00:02:37.060 Yeah. 00:02:37.080 --> 00:02:38.300 Did she come running in? 00:02:38.360 --> 00:02:38.501 Yeah. 00:02:39.628 --> 00:02:42.472 Um, and then, uh, and then, 00:02:42.532 --> 00:02:43.532 so you give him a feed, right? 00:02:43.573 --> 00:02:44.715 Shots right up immediately. 00:02:44.895 --> 00:02:45.795 That's it. 00:02:45.915 --> 00:02:46.235 He's done. 00:02:46.556 --> 00:02:46.796 I'm good. 00:02:46.817 --> 00:02:47.418 Yeah. 00:02:47.478 --> 00:02:48.138 Back in the saddle. 00:02:48.618 --> 00:02:49.740 Completely forgets what, 00:02:50.060 --> 00:02:50.661 what was going on. 00:02:50.681 --> 00:02:53.002 He's like, it's all good. 00:02:53.043 --> 00:02:54.145 Some sort of ache in my leg. 00:02:54.164 --> 00:02:54.604 It's all good. 00:02:55.265 --> 00:02:58.549 Um, anyway, yeah. 00:02:58.609 --> 00:03:00.070 Went for a, went for a walk in the park. 00:03:00.591 --> 00:03:00.852 Um, 00:03:01.841 --> 00:03:03.582 And then came home and he 00:03:03.643 --> 00:03:05.283 slept the entire rest of the day. 00:03:05.304 --> 00:03:07.705 He didn't hear a beat from me. 00:03:07.746 --> 00:03:08.806 He woke up once for a feed. 00:03:09.026 --> 00:03:09.788 That was it. 00:03:09.807 --> 00:03:10.949 And then again at night for a feed. 00:03:10.968 --> 00:03:13.550 Do you wake up normal today? 00:03:13.670 --> 00:03:14.412 Like back, 00:03:14.491 --> 00:03:15.612 like activity levels and all 00:03:15.633 --> 00:03:16.394 that kind of stuff or still 00:03:16.413 --> 00:03:17.033 dragging a little bit? 00:03:17.655 --> 00:03:20.937 Well, he was like a little bit irritable. 00:03:21.397 --> 00:03:21.778 Yeah. 00:03:21.799 --> 00:03:23.560 Probably earlier than usual today, 00:03:23.599 --> 00:03:24.341 but it's probably because 00:03:24.381 --> 00:03:25.942 he fucking slept so much yesterday. 00:03:25.961 --> 00:03:26.643 Sure, sure, sure, sure. 00:03:27.048 --> 00:03:27.770 But I don't know. 00:03:27.790 --> 00:03:29.478 He's probably like farting from two a.m. 00:03:29.598 --> 00:03:29.818 Just. 00:03:34.118 --> 00:03:36.040 like just just squirming 00:03:36.061 --> 00:03:37.181 around and farting for like 00:03:37.722 --> 00:03:39.024 a good three hours is he in 00:03:39.063 --> 00:03:40.425 a is he in a bassinet like 00:03:40.724 --> 00:03:41.686 in where you guys are 00:03:41.705 --> 00:03:42.407 sleeping or do you guys 00:03:42.426 --> 00:03:43.247 have something else you 00:03:43.268 --> 00:03:44.669 guys sleep there yeah 00:03:44.709 --> 00:03:46.010 bassinet just yeah just 00:03:46.050 --> 00:03:47.893 next door like in the same 00:03:47.913 --> 00:03:50.675 room just like oh yeah yeah 00:03:50.816 --> 00:03:51.515 well it's too much because 00:03:51.536 --> 00:03:52.697 otherwise it's too much to 00:03:52.717 --> 00:03:53.899 get out of bed to 00:03:53.938 --> 00:03:54.359 especially when he's 00:03:54.378 --> 00:03:55.080 feeding the middle of night 00:03:55.120 --> 00:03:55.721 right you want to have him 00:03:55.741 --> 00:03:56.921 you know I have him close so 00:03:57.745 --> 00:03:57.985 Yeah, 00:03:58.044 --> 00:03:59.866 my partner still like usually gets up 00:03:59.926 --> 00:04:00.127 though. 00:04:00.366 --> 00:04:00.586 Yeah. 00:04:01.207 --> 00:04:01.867 Because I think she tries 00:04:01.907 --> 00:04:02.949 not to wake me up too much 00:04:03.250 --> 00:04:04.950 so I can do the morning shift and stuff. 00:04:05.010 --> 00:04:07.413 And I don't really hear her actually. 00:04:07.432 --> 00:04:08.854 She's pretty good at it now. 00:04:08.873 --> 00:04:10.455 She's like when he starts 00:04:10.735 --> 00:04:11.616 making the noises, 00:04:11.776 --> 00:04:12.897 she just scoops him up and 00:04:13.538 --> 00:04:14.818 takes him next door, gives him a feed, 00:04:15.199 --> 00:04:15.860 puts him back down. 00:04:16.439 --> 00:04:17.120 Yeah. 00:04:17.341 --> 00:04:20.124 And then I get like the four a.m. 00:04:21.103 --> 00:04:22.365 He starts going at four a.m. 00:04:22.464 --> 00:04:22.766 usually. 00:04:22.846 --> 00:04:25.548 So just get up and give him his feed. 00:04:25.567 --> 00:04:26.608 We watch a bit of YouTube. 00:04:27.358 --> 00:04:28.379 hang out. 00:04:28.399 --> 00:04:31.120 Does he fall asleep again on you? 00:04:31.220 --> 00:04:32.560 Usually, yeah. 00:04:32.600 --> 00:04:34.081 He's active for like a 00:04:34.122 --> 00:04:34.942 couple hours and then he'll 00:04:34.982 --> 00:04:35.903 sort of doze off again. 00:04:36.723 --> 00:04:38.624 We go and read the book and 00:04:39.725 --> 00:04:41.026 get in the little play gym 00:04:41.086 --> 00:04:41.805 thing in the morning. 00:04:42.887 --> 00:04:44.648 Do some tummy time and look in the mirror. 00:04:45.067 --> 00:04:47.410 Yeah, yeah, yeah. 00:04:47.490 --> 00:04:48.850 All that shit. 00:04:49.190 --> 00:04:52.391 Rattle the shit. 00:04:52.411 --> 00:04:53.353 Those are his first shots, right? 00:04:55.077 --> 00:04:55.499 Uh, no. 00:04:55.579 --> 00:04:58.040 So what are we now? 00:04:58.180 --> 00:04:58.982 How, what's the age? 00:04:59.081 --> 00:05:00.543 Six weeks, six, six weeks. 00:05:00.663 --> 00:05:00.882 All right. 00:05:01.343 --> 00:05:03.084 So immediately after they get born here, 00:05:03.105 --> 00:05:04.466 they just fill them full of shit as well. 00:05:04.747 --> 00:05:06.468 So I don't think we have 00:05:06.487 --> 00:05:07.988 like a vitamin K shot and 00:05:08.009 --> 00:05:09.591 as well as like some other vaccines. 00:05:10.190 --> 00:05:10.732 I don't think we can tell 00:05:10.812 --> 00:05:11.733 you what's out here. 00:05:11.932 --> 00:05:13.053 They do like, they do like the, uh, 00:05:13.653 --> 00:05:14.454 there's a vitamin K. I 00:05:14.475 --> 00:05:15.536 think it's a vitamin K drop, 00:05:15.555 --> 00:05:16.576 like an eye drop that goes in. 00:05:18.377 --> 00:05:19.538 like instantly get soaked in 00:05:19.639 --> 00:05:22.019 vitamin K. And I don't 00:05:22.040 --> 00:05:23.040 think it's anything at birth. 00:05:23.100 --> 00:05:23.939 I think there's something at one. 00:05:24.079 --> 00:05:24.899 I think it's usually in the 00:05:24.959 --> 00:05:26.860 months at four weeks. 00:05:26.920 --> 00:05:27.740 I just can't remember what's what, 00:05:27.781 --> 00:05:28.942 but yeah, 00:05:28.982 --> 00:05:30.081 there's a lot of shots in the first year. 00:05:30.242 --> 00:05:30.601 I remember that. 00:05:30.622 --> 00:05:31.141 It just takes them. 00:05:31.281 --> 00:05:32.523 Just take some time to get 00:05:32.543 --> 00:05:33.142 through different things. 00:05:33.302 --> 00:05:35.144 Plus you guys are well here. 00:05:35.463 --> 00:05:35.644 Well, 00:05:36.004 --> 00:05:37.124 I guess I don't know if we did like 00:05:37.343 --> 00:05:38.485 flu and that kind of stuff 00:05:38.504 --> 00:05:39.785 for at that age. 00:05:39.805 --> 00:05:41.185 I don't remember how that worked for those, 00:05:41.225 --> 00:05:42.105 but at least all the other, 00:05:42.125 --> 00:05:42.745 all the other backs. 00:05:44.726 --> 00:05:44.887 Well, 00:05:46.012 --> 00:05:46.153 Yeah, 00:05:46.213 --> 00:05:49.074 I think we've got a pretty regimented 00:05:49.115 --> 00:05:49.834 program here. 00:05:51.456 --> 00:05:52.656 The Queensland government 00:05:52.697 --> 00:05:55.379 loves to pay for it and 00:05:55.418 --> 00:05:56.678 stuff your baby full of shit. 00:05:56.778 --> 00:05:58.519 I'm done. 00:05:59.120 --> 00:06:00.482 I'm done. 00:06:00.502 --> 00:06:02.583 I've got a lot of shit and 00:06:02.783 --> 00:06:04.564 I'm still not receiving five G yet. 00:06:04.704 --> 00:06:06.064 I've got to have a 00:06:06.732 --> 00:06:07.713 I'm there. 00:06:07.733 --> 00:06:08.973 Separate device still. 00:06:09.293 --> 00:06:10.053 Fuck. 00:06:10.074 --> 00:06:10.615 Come on, guys. 00:06:11.235 --> 00:06:11.675 Figure it out. 00:06:12.074 --> 00:06:13.817 Do we have any blocks in the 00:06:13.836 --> 00:06:14.536 lorry yet or no? 00:06:14.896 --> 00:06:16.617 Oh, it didn't look. 00:06:16.757 --> 00:06:17.619 I looked to make sure that 00:06:17.639 --> 00:06:18.639 we're doing the page file 00:06:18.660 --> 00:06:19.540 thing and then I was like, 00:06:20.920 --> 00:06:22.461 I shut all the alerts off of that chain, 00:06:22.482 --> 00:06:22.961 I think. 00:06:24.264 --> 00:06:25.285 So that's actually, 00:06:26.185 --> 00:06:26.946 that's the thing I was 00:06:26.966 --> 00:06:27.726 going to say to you. 00:06:29.028 --> 00:06:30.009 Oh, this is the wrong server. 00:06:31.430 --> 00:06:36.757 So we were talking Zabbix and, you know, 00:06:38.499 --> 00:06:39.660 a lot of development, 00:06:39.680 --> 00:06:40.901 a lot of things happening 00:06:41.661 --> 00:06:43.283 on the monitoring scene these days, 00:06:43.904 --> 00:06:45.786 new ideas coming through anyway. 00:06:46.750 --> 00:06:47.750 One of the most annoying 00:06:47.771 --> 00:06:50.093 things is when there's like 00:06:50.112 --> 00:06:51.194 a slow upgrade, right? 00:06:51.213 --> 00:06:53.076 Because it's not just nodes back up. 00:06:53.697 --> 00:06:54.958 It misses out on that three 00:06:54.978 --> 00:06:57.300 minute window before you get like the, 00:06:57.339 --> 00:06:57.439 well, 00:06:57.459 --> 00:06:58.521 I think it's like a minute and a half, 00:06:58.560 --> 00:06:58.781 right? 00:06:58.802 --> 00:06:59.661 So doing thirty second 00:06:59.682 --> 00:07:00.642 polling and then there's 00:07:01.603 --> 00:07:04.526 misses three number before 00:07:04.567 --> 00:07:05.528 it's like a no data alert. 00:07:06.588 --> 00:07:06.968 Yeah. 00:07:07.009 --> 00:07:08.911 So, yeah. 00:07:10.052 --> 00:07:12.894 My idea is that now that 00:07:12.915 --> 00:07:15.879 we've started the governance monitoring, 00:07:16.158 --> 00:07:16.800 right? 00:07:16.839 --> 00:07:19.002 So I've been keeping an eye 00:07:19.041 --> 00:07:19.603 on that as well. 00:07:19.702 --> 00:07:26.529 It is picking up the governance proposals. 00:07:26.750 --> 00:07:28.391 So it is picking those up. 00:07:29.192 --> 00:07:30.754 so that's step one step one 00:07:30.994 --> 00:07:32.596 um need to filter out all 00:07:32.615 --> 00:07:33.336 the other crap though I 00:07:33.355 --> 00:07:35.117 think the well actually no 00:07:35.338 --> 00:07:36.218 we were going to keep this 00:07:36.377 --> 00:07:38.300 when we so we can monitor 00:07:38.319 --> 00:07:39.100 whether the vote's being 00:07:39.120 --> 00:07:40.081 done or not or at least 00:07:40.442 --> 00:07:42.382 yeah because we it also it 00:07:42.403 --> 00:07:43.144 all depends on whether I 00:07:43.163 --> 00:07:43.884 can monitor if the vote 00:07:43.903 --> 00:07:44.964 happened right that makes 00:07:45.004 --> 00:07:46.146 sense so if if I haven't 00:07:46.165 --> 00:07:46.786 voted on something I like 00:07:46.805 --> 00:07:48.387 to be known but if if I 00:07:48.406 --> 00:07:49.908 have to do two things right 00:07:49.928 --> 00:07:50.788 I have to go vote there go 00:07:50.829 --> 00:07:51.709 turn this fucking thing off 00:07:52.591 --> 00:07:53.031 that's no good 00:07:53.779 --> 00:07:57.023 So for those listening and 00:07:57.103 --> 00:07:59.004 while watching as well that don't know, 00:07:59.725 --> 00:08:02.427 Zabbix is just a monitoring platform. 00:08:02.447 --> 00:08:03.447 This is what me and Serp 00:08:03.728 --> 00:08:04.949 both use for our main 00:08:05.389 --> 00:08:07.791 monitoring platform for all 00:08:07.831 --> 00:08:08.872 of our node operations. 00:08:08.971 --> 00:08:11.935 It's not like prolific, 00:08:13.096 --> 00:08:14.216 particularly like in the... 00:08:15.685 --> 00:08:16.846 It's extremely prolific, 00:08:16.886 --> 00:08:18.026 just not in this industry. 00:08:18.206 --> 00:08:18.947 Yeah. 00:08:19.346 --> 00:08:20.286 For what we use it for. 00:08:20.307 --> 00:08:21.146 Yeah, yeah, yeah. 00:08:21.226 --> 00:08:22.427 In the NodeOps world, 00:08:22.487 --> 00:08:23.788 it is not very prolific. 00:08:25.048 --> 00:08:26.288 I have noticed that some of 00:08:26.348 --> 00:08:27.629 the validators that I 00:08:28.329 --> 00:08:30.088 respect a lot use it, 00:08:30.769 --> 00:08:32.570 and the ones that I don't, don't use it. 00:08:32.590 --> 00:08:34.409 It's just, I guess, the way it is. 00:08:34.570 --> 00:08:35.530 Yeah. 00:08:38.831 --> 00:08:41.113 But anyway, at every opportunity, 00:08:41.173 --> 00:08:43.793 we shill Zabbix and try to get people. 00:08:43.913 --> 00:08:44.374 Well, I mean, 00:08:44.413 --> 00:08:45.554 not really try to get people. 00:08:45.595 --> 00:08:46.674 We'll just flex on them that 00:08:46.735 --> 00:08:50.017 our shit's better than theirs. 00:08:50.397 --> 00:08:52.597 And what we're talking about now is just, 00:08:52.898 --> 00:08:53.198 you know, 00:08:54.178 --> 00:08:55.359 it's a constant development 00:08:55.600 --> 00:08:57.639 we uh come up with ideas of 00:08:57.679 --> 00:08:59.020 how we can improve things 00:08:59.081 --> 00:09:00.140 and make sure that we're in 00:09:00.181 --> 00:09:01.121 the know with everything 00:09:01.162 --> 00:09:02.182 because you know you get 00:09:02.422 --> 00:09:04.023 loads of nodes and lots of 00:09:04.082 --> 00:09:05.503 networks it's very hard to 00:09:05.523 --> 00:09:06.583 stay on top of everything 00:09:06.903 --> 00:09:08.344 and um to have this do it 00:09:08.384 --> 00:09:09.605 for you for the most part 00:09:09.725 --> 00:09:12.625 is the aim of the game yeah 00:09:12.706 --> 00:09:15.267 so we have an issue well 00:09:15.307 --> 00:09:16.086 it's not an issue at the 00:09:16.106 --> 00:09:17.027 moment but a life 00:09:17.067 --> 00:09:17.807 improvement that we're 00:09:17.827 --> 00:09:19.327 trying to make is now 00:09:19.427 --> 00:09:21.249 monitoring on-chain governance 00:09:22.009 --> 00:09:23.551 And then through that monitoring, 00:09:23.770 --> 00:09:25.793 give certain alerts under 00:09:25.852 --> 00:09:28.274 certain conditions about, you know, 00:09:28.294 --> 00:09:29.456 the happenings in governance. 00:09:29.855 --> 00:09:31.378 So one thing that we want 00:09:31.837 --> 00:09:33.240 initially was that we're 00:09:33.259 --> 00:09:34.581 going to monitor governance 00:09:34.780 --> 00:09:36.423 and then we are going to 00:09:36.883 --> 00:09:38.244 monitor if we have voted 00:09:39.105 --> 00:09:39.745 for the governance. 00:09:40.488 --> 00:09:43.890 So, you know, the idea is, I suppose, 00:09:43.931 --> 00:09:46.452 that once a governance proposal pops up, 00:09:46.472 --> 00:09:47.833 which we use discovery for, 00:09:48.374 --> 00:09:50.775 which is a mechanism that 00:09:50.796 --> 00:09:51.755 just will automatically 00:09:51.775 --> 00:09:54.437 find these things once it's told to do so, 00:09:54.477 --> 00:09:55.538 like programmed to do so. 00:09:56.159 --> 00:09:59.142 So it'll find the latest proposal. 00:09:59.948 --> 00:10:02.049 bring it up and then check the wallet, 00:10:02.269 --> 00:10:03.110 see if it's voted. 00:10:03.190 --> 00:10:05.171 If it hasn't give a warning. 00:10:05.630 --> 00:10:06.772 And then once it's voted, 00:10:07.072 --> 00:10:08.032 the warning go away. 00:10:08.251 --> 00:10:09.173 And then once the governance 00:10:09.192 --> 00:10:10.013 proposal is done, 00:10:10.293 --> 00:10:11.693 the governance proposal disappear. 00:10:12.153 --> 00:10:13.073 I guess that's the flow that 00:10:13.094 --> 00:10:14.294 we're going for, for that, right? 00:10:14.355 --> 00:10:14.934 Is that, 00:10:15.054 --> 00:10:16.796 is that your understanding of that? 00:10:16.895 --> 00:10:18.336 Or eventually that it just 00:10:18.355 --> 00:10:19.557 votes yes on every single 00:10:19.576 --> 00:10:22.118 fucking thing and then tweets about it. 00:10:22.298 --> 00:10:24.118 And then then calls out 00:10:24.197 --> 00:10:25.519 other validators for not voting. 00:10:26.659 --> 00:10:27.460 you know that's entirely 00:10:27.559 --> 00:10:28.760 possible right and that is 00:10:28.841 --> 00:10:29.961 entirely possible why the 00:10:30.001 --> 00:10:31.261 fuck hasn't blah blah blah 00:10:32.143 --> 00:10:35.184 who has more vp so we can 00:10:35.384 --> 00:10:37.846 use we can use zabix to 00:10:37.905 --> 00:10:42.089 send a um an alert to 00:10:42.489 --> 00:10:45.650 ansible yeah uh what's it 00:10:45.671 --> 00:10:46.951 called ansible rule books 00:10:47.851 --> 00:10:48.673 which can then run a 00:10:48.712 --> 00:10:50.533 playbook to do exactly that 00:10:50.994 --> 00:10:52.835 yeah I think and that's where we're we're 00:10:53.975 --> 00:10:55.537 since a lot of us build 00:10:55.956 --> 00:10:57.577 binaries and there's no 00:10:57.658 --> 00:10:59.057 direct tie between the 00:10:59.118 --> 00:11:02.019 upgrade and the binary version, 00:11:02.240 --> 00:11:04.640 it's not a field that's in there. 00:11:06.101 --> 00:11:06.741 That is, yeah, 00:11:06.761 --> 00:11:07.482 that's the problem that 00:11:07.523 --> 00:11:08.363 we're bumping up against. 00:11:08.383 --> 00:11:09.562 There's no actual, like, 00:11:09.822 --> 00:11:11.224 it'd be great to have a 00:11:11.303 --> 00:11:13.004 GitHub commit in each of those. 00:11:13.024 --> 00:11:13.624 That would be amazing. 00:11:13.644 --> 00:11:14.705 Because this is the commit 00:11:14.725 --> 00:11:15.445 that we're looking for. 00:11:15.945 --> 00:11:17.144 And even if that commit changed, 00:11:17.524 --> 00:11:19.046 or didn't change, but if that commit, 00:11:19.365 --> 00:11:21.605 then we could at least take 00:11:21.625 --> 00:11:22.246 some action off that. 00:11:22.287 --> 00:11:25.527 Now, there are, in those upgrades, 00:11:25.667 --> 00:11:27.067 I'm not sure if every chain does this. 00:11:28.442 --> 00:11:30.225 but you can link to 00:11:30.365 --> 00:11:32.649 pre-built binaries that the 00:11:32.668 --> 00:11:33.530 team is providing. 00:11:33.551 --> 00:11:35.254 Yeah, it's in the metadata. 00:11:35.955 --> 00:11:36.755 A lot of the time it's in the metadata. 00:11:36.775 --> 00:11:38.379 I think it's a field actually, isn't it? 00:11:38.399 --> 00:11:39.000 I thought it was something 00:11:39.019 --> 00:11:39.441 that was actually 00:11:39.461 --> 00:11:40.423 structured in that prop. 00:11:41.397 --> 00:11:41.798 Used to be. 00:11:41.878 --> 00:11:43.779 I think they changed it at some point. 00:11:43.840 --> 00:11:44.639 I'm not a hundred percent sure. 00:11:44.941 --> 00:11:45.642 But you can, you can like, 00:11:45.682 --> 00:11:46.702 you can basically parse that out. 00:11:46.722 --> 00:11:47.823 So I can look for, you know, 00:11:48.865 --> 00:11:49.826 arm sixty four version of 00:11:49.926 --> 00:11:51.246 yada yada and go grab it 00:11:51.267 --> 00:11:53.269 and go use it and put it in, you know, 00:11:53.610 --> 00:11:54.049 bring that in. 00:11:54.090 --> 00:11:54.910 And then I could probably 00:11:54.971 --> 00:11:55.491 automate that whole 00:11:55.511 --> 00:11:56.493 structure from start to pack. 00:11:56.533 --> 00:11:57.813 I would know what the binary is. 00:11:57.854 --> 00:11:58.975 I would know all that kind of stuff. 00:11:59.355 --> 00:11:59.995 And I thought wasn't. 00:12:00.096 --> 00:12:00.897 And I think that isn't a 00:12:00.937 --> 00:12:01.918 version of Cosmovisor. 00:12:01.937 --> 00:12:03.078 Does that exact same fucking thing? 00:12:04.558 --> 00:12:05.759 The latest version of Cosmovisor, 00:12:05.840 --> 00:12:06.899 if that binary is in there, 00:12:06.940 --> 00:12:08.200 you can have it go actually 00:12:08.240 --> 00:12:10.221 go grab it out of there and something. 00:12:10.360 --> 00:12:12.942 I think Cosmovisor does that, too. 00:12:12.981 --> 00:12:13.922 I've never used it. 00:12:13.961 --> 00:12:15.562 Cosmovisor has always done that, 00:12:16.322 --> 00:12:17.263 except everyone turns it 00:12:17.322 --> 00:12:20.303 off because it's like I've never used it. 00:12:20.323 --> 00:12:24.725 There's like a bit of a stigma, I guess. 00:12:25.225 --> 00:12:26.066 Yeah, too easy to fuck it up. 00:12:26.758 --> 00:12:26.998 Well, 00:12:27.038 --> 00:12:31.423 it's too easy for someone to insert 00:12:31.443 --> 00:12:33.086 something in there which is malicious. 00:12:33.426 --> 00:12:34.067 Yeah, for sure. 00:12:34.327 --> 00:12:34.927 Yeah. 00:12:34.988 --> 00:12:35.269 Anyway, 00:12:35.288 --> 00:12:36.009 I don't know if we would take 00:12:36.110 --> 00:12:36.750 automatic action. 00:12:36.770 --> 00:12:37.532 It would be nice to... 00:12:39.690 --> 00:12:40.950 it would be nice to find a 00:12:41.009 --> 00:12:43.431 way to automate that 00:12:43.471 --> 00:12:45.231 process where I'm not 00:12:45.432 --> 00:12:47.231 updating an Ansible file 00:12:47.251 --> 00:12:48.332 with the right version or 00:12:48.373 --> 00:12:49.773 an upgrade name and all 00:12:49.793 --> 00:12:50.312 that kind of stuff, 00:12:50.352 --> 00:12:51.354 which I think I could do, actually. 00:12:51.374 --> 00:12:52.114 I could probably find a way 00:12:53.234 --> 00:12:54.995 to at least do some part of 00:12:55.014 --> 00:12:55.715 that automated. 00:12:56.414 --> 00:12:58.235 But yeah, that is a process. 00:12:58.275 --> 00:13:01.476 Process would be vote exists, vote gets, 00:13:01.917 --> 00:13:03.057 turns into a small alert, 00:13:03.297 --> 00:13:05.998 I go take action on that, alert goes away, 00:13:06.357 --> 00:13:06.518 right? 00:13:06.538 --> 00:13:07.097 That's kind of the idea. 00:13:07.118 --> 00:13:09.379 And the other thing you could do is, 00:13:10.332 --> 00:13:10.953 The other thing you could do 00:13:10.974 --> 00:13:12.254 is you could even say if 00:13:12.274 --> 00:13:13.876 that upgrade alert exists 00:13:14.018 --> 00:13:14.958 that came out of governance, 00:13:15.099 --> 00:13:15.679 I could do two things. 00:13:15.779 --> 00:13:15.840 One, 00:13:15.860 --> 00:13:17.542 I could go check and see if that vote 00:13:17.581 --> 00:13:17.841 exists. 00:13:17.861 --> 00:13:18.182 Number two, 00:13:18.222 --> 00:13:19.183 I could have it go check the 00:13:19.244 --> 00:13:21.145 node and all the nodes that 00:13:21.265 --> 00:13:24.750 match that to say, hey, for this node, 00:13:24.931 --> 00:13:26.572 does that folder exist? 00:13:26.712 --> 00:13:28.034 And is there a binary in that folder? 00:13:28.054 --> 00:13:28.735 That's the other thing you 00:13:28.755 --> 00:13:29.336 can do too is to say, 00:13:29.956 --> 00:13:31.438 is that the alert won't go 00:13:31.499 --> 00:13:32.679 away until a you voted and 00:13:32.720 --> 00:13:33.801 b you've staged it which 00:13:33.941 --> 00:13:35.302 which would be actually a 00:13:35.363 --> 00:13:37.125 nice little addition to 00:13:37.144 --> 00:13:38.206 that too for it to double 00:13:38.245 --> 00:13:38.866 check and make sure it's 00:13:38.886 --> 00:13:40.389 staged which is not really 00:13:40.849 --> 00:13:41.730 that's really easy thing to 00:13:41.769 --> 00:13:42.610 do I think because we would 00:13:42.990 --> 00:13:44.352 checking the staging is 00:13:44.894 --> 00:13:46.075 yeah that that was 00:13:46.095 --> 00:13:47.496 something we talked about I 00:13:47.557 --> 00:13:48.376 think that's entirely 00:13:48.418 --> 00:13:50.038 possible yeah I think um 00:13:51.115 --> 00:13:52.796 yeah you know the biggest 00:13:54.035 --> 00:13:55.096 yeah the funny thing is 00:13:55.157 --> 00:13:57.018 like so you know we can't 00:13:57.057 --> 00:13:58.479 you got to keep like node 00:13:59.078 --> 00:13:59.698 so we've talked about this 00:13:59.739 --> 00:14:01.779 as well you got to keep um 00:14:01.799 --> 00:14:02.500 you know the node 00:14:02.520 --> 00:14:03.681 monitoring separate to the 00:14:03.721 --> 00:14:04.642 validator monitoring 00:14:04.942 --> 00:14:06.001 because you've got bunch 00:14:06.042 --> 00:14:06.783 like you know you might run 00:14:06.822 --> 00:14:07.682 twenty nodes or something 00:14:07.702 --> 00:14:09.844 for the same network um and 00:14:09.884 --> 00:14:11.024 then so you know some 00:14:11.044 --> 00:14:12.004 things you want to know on 00:14:12.024 --> 00:14:13.145 the validator level like 00:14:13.166 --> 00:14:14.086 whether or not you voted or 00:14:14.125 --> 00:14:14.927 not and that type of 00:14:15.547 --> 00:14:16.727 And then some things you 00:14:16.748 --> 00:14:17.828 want to know on the node level, 00:14:17.849 --> 00:14:18.730 like whether or not you've 00:14:19.149 --> 00:14:20.211 staged the upgrade and that 00:14:20.250 --> 00:14:21.491 type of stuff with Cosmovisor, 00:14:21.511 --> 00:14:22.451 depending on how you're doing it. 00:14:22.972 --> 00:14:25.254 So, you know, 00:14:25.455 --> 00:14:27.596 the complexity there becomes like, 00:14:27.995 --> 00:14:29.738 do you then start polling 00:14:29.778 --> 00:14:30.918 governance with both nodes? 00:14:31.818 --> 00:14:33.299 um or do you have an add-on 00:14:33.900 --> 00:14:35.261 somewhere that you can then 00:14:35.522 --> 00:14:37.583 like see that through the 00:14:37.624 --> 00:14:39.966 other monitors yeah so yeah 00:14:39.985 --> 00:14:40.706 I mean because everything's 00:14:40.745 --> 00:14:41.826 sort of broken up into host 00:14:41.866 --> 00:14:43.488 and stuff right so you know 00:14:43.508 --> 00:14:45.068 there's that's the the 00:14:45.089 --> 00:14:46.250 thinking is going on there 00:14:46.610 --> 00:14:48.072 at the moment to try and 00:14:48.131 --> 00:14:49.232 think about how that can be 00:14:49.273 --> 00:14:51.293 done yeah probably it's 00:14:51.313 --> 00:14:52.254 almost like you need a 00:14:52.315 --> 00:14:53.875 third-party source that has 00:14:54.015 --> 00:14:55.778 a if that governance exists 00:14:55.837 --> 00:14:56.878 then that third-party 00:14:57.778 --> 00:14:58.918 data source knows that the 00:14:58.960 --> 00:14:59.700 governance is out there. 00:14:59.720 --> 00:15:01.240 And then once all the nodes are updated, 00:15:01.259 --> 00:15:01.681 I don't know. 00:15:01.701 --> 00:15:02.961 I don't want to get too, 00:15:02.980 --> 00:15:03.941 which wouldn't be hard to do. 00:15:04.000 --> 00:15:06.201 Like we could, we could make that API, 00:15:06.241 --> 00:15:06.902 but I mean, you don't, 00:15:07.003 --> 00:15:07.503 you don't want to be 00:15:07.562 --> 00:15:09.724 relying on another piece of software. 00:15:10.264 --> 00:15:11.683 So, yeah. 00:15:11.703 --> 00:15:11.864 I mean, 00:15:11.884 --> 00:15:13.404 this isn't really that much of a 00:15:13.445 --> 00:15:14.384 critical thing. 00:15:14.404 --> 00:15:15.926 It's not like, 00:15:15.946 --> 00:15:17.166 I don't want to rely on other shit. 00:15:17.485 --> 00:15:18.466 I don't even care about governance. 00:15:18.527 --> 00:15:20.326 Number one, quote me on that. 00:15:20.488 --> 00:15:22.508 And number two, I haven't really missed. 00:15:22.528 --> 00:15:24.308 I mean, this is not really that, 00:15:24.509 --> 00:15:25.830 that valuable for main net, main net, 00:15:26.049 --> 00:15:26.389 main net, 00:15:27.190 --> 00:15:29.071 gov props I you know I we 00:15:29.091 --> 00:15:29.770 haven't missed a main net 00:15:29.811 --> 00:15:31.350 upgrade I think ever uh 00:15:31.431 --> 00:15:33.231 testnet yeah like testnet 00:15:33.532 --> 00:15:34.873 those things are not like I 00:15:34.893 --> 00:15:35.952 don't really pull those at 00:15:35.972 --> 00:15:36.812 all I don't have like I 00:15:36.832 --> 00:15:37.793 have I have made net 00:15:37.852 --> 00:15:38.774 upgrades up on my board 00:15:38.813 --> 00:15:39.614 above me so I know what's 00:15:39.634 --> 00:15:40.553 going on there but that's 00:15:40.594 --> 00:15:41.235 not I don't really watch 00:15:41.254 --> 00:15:42.235 there's not it's not really 00:15:42.254 --> 00:15:43.294 a good source of that data 00:15:43.495 --> 00:15:45.395 to be honest um unless it's 00:15:45.456 --> 00:15:46.855 on chain and even though 00:15:46.895 --> 00:15:48.136 sometimes like we get caught on 00:15:48.817 --> 00:15:49.457 and upgrades that don't 00:15:49.498 --> 00:15:50.578 happen via gov props like 00:15:50.599 --> 00:15:52.259 that's that's that happens 00:15:52.419 --> 00:15:53.660 like because we run rpc for 00:15:53.681 --> 00:15:54.261 a bunch of chains that we 00:15:54.282 --> 00:15:55.423 don't validate on and 00:15:55.442 --> 00:15:56.783 there'll be like a hot fix 00:15:57.104 --> 00:15:58.466 something or other hey this 00:15:58.505 --> 00:15:59.506 isn't state breaking but it 00:15:59.567 --> 00:16:00.567 might break a state at some 00:16:00.607 --> 00:16:01.548 point down the line and we 00:16:01.567 --> 00:16:02.328 miss it and the next thing 00:16:02.349 --> 00:16:03.610 you know got to rebuild 00:16:03.649 --> 00:16:05.292 nodes because some fucking thing 00:16:06.347 --> 00:16:07.229 I think the best thing we've 00:16:07.249 --> 00:16:09.030 added to the Solana host 00:16:09.091 --> 00:16:12.173 monitor is that fucking the, 00:16:12.214 --> 00:16:13.115 the version upgrades 00:16:13.134 --> 00:16:14.035 because they may be 00:16:14.076 --> 00:16:15.216 upgrading that shit all the time. 00:16:15.236 --> 00:16:15.356 Yeah. 00:16:16.418 --> 00:16:16.558 Yeah. 00:16:16.578 --> 00:16:18.520 So that, that actually, so that, 00:16:18.541 --> 00:16:20.722 that's another interesting thing is, is, 00:16:21.023 --> 00:16:21.163 um, 00:16:22.244 --> 00:16:23.303 That's not really, I think, 00:16:23.344 --> 00:16:24.365 valid for... Since 00:16:24.384 --> 00:16:25.524 everything is state-breaking, 00:16:26.025 --> 00:16:27.285 it's not that valid, I think, 00:16:27.326 --> 00:16:28.385 for the Cosmos ecosystem. 00:16:28.446 --> 00:16:29.326 Although we could do 00:16:29.366 --> 00:16:30.746 something that looks at the 00:16:30.826 --> 00:16:32.687 latest release out of GitHub. 00:16:32.788 --> 00:16:34.849 I did add that on all my EVM chains. 00:16:35.649 --> 00:16:37.889 But EVM is tough because you 00:16:37.909 --> 00:16:39.791 have both a consensus 00:16:39.850 --> 00:16:41.392 engine as well as an execution. 00:16:41.912 --> 00:16:42.912 And a lot of these teams 00:16:43.111 --> 00:16:44.253 don't necessarily... 00:16:45.421 --> 00:16:46.701 They might package those 00:16:47.241 --> 00:16:48.783 into a bundle to say like, 00:16:49.144 --> 00:16:49.984 I think a good example is I 00:16:50.004 --> 00:16:50.764 could look up right here. 00:16:51.926 --> 00:16:52.485 I might've actually, 00:16:52.505 --> 00:16:53.187 I might've deleted them. 00:16:54.303 --> 00:16:57.384 um maybe linea or scroll or 00:16:57.404 --> 00:16:58.265 something else scroll might 00:16:58.306 --> 00:16:59.886 say hey we're on version 00:16:59.947 --> 00:17:01.647 point zero four three well 00:17:01.726 --> 00:17:02.548 point zero four three 00:17:02.587 --> 00:17:04.729 scroll really means node 00:17:05.288 --> 00:17:07.309 one point oh and geth one 00:17:07.329 --> 00:17:08.829 oh eight you know what I 00:17:08.849 --> 00:17:10.471 mean and so yeah so 00:17:10.550 --> 00:17:11.692 whenever I if I'm querying 00:17:11.731 --> 00:17:12.592 github I'm getting a 00:17:12.912 --> 00:17:14.353 package number but that 00:17:14.373 --> 00:17:14.972 package number doesn't 00:17:14.992 --> 00:17:15.913 exist anywhere on chain 00:17:15.933 --> 00:17:16.953 it's just a made-up type of 00:17:16.973 --> 00:17:17.773 thing so there's a lot of 00:17:17.794 --> 00:17:18.694 that kind of going on 00:17:19.355 --> 00:17:20.296 But there was just a new 00:17:20.336 --> 00:17:21.576 version of Geth that just 00:17:21.777 --> 00:17:22.718 popped yesterday, 00:17:23.077 --> 00:17:24.579 and I got alerted on twenty 00:17:24.640 --> 00:17:26.540 fucking things because now 00:17:26.602 --> 00:17:27.662 there's a Geth update, right? 00:17:27.701 --> 00:17:28.623 Because the execution engine 00:17:28.643 --> 00:17:29.824 knows that it's been upgraded, 00:17:29.844 --> 00:17:30.605 which is actually great. 00:17:30.964 --> 00:17:32.066 I like to be able to know that. 00:17:32.386 --> 00:17:34.428 But every chain might not 00:17:34.509 --> 00:17:36.309 say that they're ready for that. 00:17:36.369 --> 00:17:37.892 So that might be okay in the L-I's, 00:17:37.912 --> 00:17:39.913 but the L-II's, they might need to do... 00:17:41.222 --> 00:17:41.383 You know, 00:17:41.403 --> 00:17:43.304 just because Geth updated doesn't 00:17:43.364 --> 00:17:45.925 mean that their own version 00:17:45.965 --> 00:17:47.126 of the consensus engine is 00:17:47.166 --> 00:17:47.928 going to work with that. 00:17:48.107 --> 00:17:50.628 So that happened actually on... Actually, 00:17:50.669 --> 00:17:51.130 it wasn't Geth. 00:17:51.170 --> 00:17:52.830 It was OpGeth on the Optimism side. 00:17:52.851 --> 00:17:53.451 So there's a bunch of chains 00:17:53.471 --> 00:17:54.112 that use OpGeth. 00:17:54.632 --> 00:17:55.472 So that changed there, 00:17:55.512 --> 00:17:55.992 but that doesn't 00:17:56.012 --> 00:17:57.394 necessarily mean that all 00:17:57.413 --> 00:17:59.174 the subchains that use 00:17:59.596 --> 00:18:01.257 OpGeth and all the Optimism 00:18:01.436 --> 00:18:03.238 stack are ready for that upgrade. 00:18:03.258 --> 00:18:05.400 So it doesn't... It's all 00:18:05.420 --> 00:18:06.881 kind of a fucking mess, to be honest. 00:18:06.941 --> 00:18:08.281 So Solana works perfect because... 00:18:08.902 --> 00:18:11.222 There's one fucking binary and you know, 00:18:11.262 --> 00:18:12.182 if you're on it or not, 00:18:12.323 --> 00:18:13.844 like we just got alerted two days ago, 00:18:13.864 --> 00:18:13.983 right. 00:18:14.003 --> 00:18:14.805 Or whatever it was like, 00:18:15.244 --> 00:18:16.184 it's just a new version. 00:18:16.464 --> 00:18:17.125 And then you can go see if 00:18:17.185 --> 00:18:18.105 it's worthwhile and what 00:18:18.125 --> 00:18:18.625 you want to do there, 00:18:18.645 --> 00:18:19.226 which is really great. 00:18:19.246 --> 00:18:19.326 Yeah. 00:18:19.586 --> 00:18:20.527 And then you can delete the 00:18:20.567 --> 00:18:21.346 fucking discord. 00:18:22.607 --> 00:18:23.387 Yeah, exactly. 00:18:24.087 --> 00:18:24.269 Yeah. 00:18:24.348 --> 00:18:25.148 And, and, and well, 00:18:25.169 --> 00:18:25.788 that team also does a 00:18:25.808 --> 00:18:26.348 really good job with 00:18:26.409 --> 00:18:27.430 tagging and they do a good 00:18:27.450 --> 00:18:28.710 job at latest releases and 00:18:28.730 --> 00:18:29.351 they do good job. 00:18:29.391 --> 00:18:30.490 Like that has been, 00:18:30.711 --> 00:18:31.912 that's been quite good. 00:18:33.771 --> 00:18:34.332 And other ones are just, 00:18:34.352 --> 00:18:34.751 they're just more 00:18:34.791 --> 00:18:35.352 complicated when you have 00:18:35.412 --> 00:18:36.232 multiple engines and all 00:18:36.252 --> 00:18:37.393 that kind of shit just turns into a, 00:18:38.053 --> 00:18:39.093 just trying to track it down. 00:18:39.173 --> 00:18:40.554 But as long as, I mean, I'm not, 00:18:40.574 --> 00:18:41.034 for the most part, 00:18:41.074 --> 00:18:42.054 those things I'm not too worried about. 00:18:42.114 --> 00:18:42.314 I just, 00:18:42.433 --> 00:18:44.355 I just get worried about like state 00:18:44.375 --> 00:18:44.934 breaking changes, 00:18:44.954 --> 00:18:45.815 state breaking changes on 00:18:45.855 --> 00:18:47.875 the EVM ecosystem are 00:18:47.914 --> 00:18:49.076 usually pretty far between. 00:18:49.455 --> 00:18:50.796 And you have a, 00:18:51.256 --> 00:18:52.997 usually a pretty decent 00:18:53.017 --> 00:18:54.576 knowledge of when that's occurring, 00:18:54.636 --> 00:18:55.777 but maybe you don't know 00:18:55.836 --> 00:18:56.798 what version is actually 00:18:56.817 --> 00:18:57.557 going to be the one that's 00:18:57.577 --> 00:18:59.357 going to get you past the fork. 00:19:00.317 --> 00:19:01.078 Yeah. 00:19:01.098 --> 00:19:02.578 So usually I know when you're good shits. 00:19:03.894 --> 00:19:04.115 Yeah. 00:19:04.134 --> 00:19:05.796 Which over there, well, right there, 00:19:05.816 --> 00:19:07.615 you're not like we tried to, we try to, 00:19:08.096 --> 00:19:08.737 there's a couple of flags 00:19:08.757 --> 00:19:09.576 you can do where if it, 00:19:09.696 --> 00:19:10.758 if it knows that there's a break, 00:19:10.837 --> 00:19:11.778 it'll actually shut down. 00:19:12.577 --> 00:19:13.199 Cause otherwise it would, 00:19:13.298 --> 00:19:14.138 it would corrupt the database, 00:19:14.159 --> 00:19:15.419 which was a fucking disaster. 00:19:16.259 --> 00:19:16.460 Yeah. 00:19:16.519 --> 00:19:18.780 So I think with Solana, they, 00:19:18.921 --> 00:19:22.021 they usually actually in the discord, 00:19:22.061 --> 00:19:23.142 they say you got to upgrade 00:19:23.182 --> 00:19:25.103 by this epoch or you're 00:19:25.123 --> 00:19:26.183 fucked or you're fucked. 00:19:26.243 --> 00:19:26.384 Right. 00:19:26.403 --> 00:19:26.824 You get dropped. 00:19:27.223 --> 00:19:27.423 Yeah. 00:19:27.564 --> 00:19:28.625 Yeah. 00:19:28.845 --> 00:19:29.964 Which might be before that 00:19:29.984 --> 00:19:30.704 sound like a page of duty 00:19:30.724 --> 00:19:31.326 notification there. 00:19:31.346 --> 00:19:31.826 Yeah. 00:19:32.705 --> 00:19:33.527 I know that I know that 00:19:33.547 --> 00:19:35.469 number um hopefully that's 00:19:35.568 --> 00:19:37.410 not a laura it's a laura is 00:19:37.450 --> 00:19:40.393 it yeah so I it says 00:19:40.452 --> 00:19:41.473 stalled so I'm guessing I 00:19:41.493 --> 00:19:43.214 passed the uh oh yeah you 00:19:43.234 --> 00:19:44.477 guys are now just waiting 00:19:44.497 --> 00:19:45.376 for all the nodes that's 00:19:45.416 --> 00:19:47.239 right yeah I haven't 00:19:47.259 --> 00:19:47.980 brought up tm top or 00:19:48.000 --> 00:19:49.361 anything like that I that 00:19:50.122 --> 00:19:52.083 that chain man there is it 00:19:52.143 --> 00:19:56.326 is it is I put the I put 00:19:56.346 --> 00:19:57.387 the time out on a nose 00:19:57.407 --> 00:19:58.348 stalled like ten minutes 00:20:00.050 --> 00:20:01.972 Well, I turned it off. 00:20:03.374 --> 00:20:04.315 I just disabled it. 00:20:05.615 --> 00:20:05.996 Completely. 00:20:06.135 --> 00:20:07.656 I just disabled that alert on Allura. 00:20:07.676 --> 00:20:08.557 Just because, dude, 00:20:08.577 --> 00:20:09.999 we had like five-minute blocks, 00:20:10.220 --> 00:20:11.220 like seven-minute blocks in 00:20:11.240 --> 00:20:11.921 there and shit. 00:20:11.941 --> 00:20:12.582 There would be one that was 00:20:12.642 --> 00:20:13.923 fifteen minutes every once in a while. 00:20:14.002 --> 00:20:16.545 I don't want to know about that. 00:20:16.825 --> 00:20:18.185 And it all happens when I'm sleeping, too. 00:20:18.246 --> 00:20:19.227 They get alerts at like two a.m. 00:20:19.326 --> 00:20:21.348 Allura's taking a shit with 00:20:21.368 --> 00:20:24.311 a twelve-minute block or something. 00:20:25.132 --> 00:20:28.432 yeah anyway um yeah it's 00:20:28.472 --> 00:20:29.493 kind of fucking annoying 00:20:29.594 --> 00:20:30.934 but um I mean they're going 00:20:30.954 --> 00:20:32.115 to fix that up now they're 00:20:32.134 --> 00:20:33.154 looking at doing some um 00:20:33.194 --> 00:20:34.756 they're going to add like a 00:20:34.796 --> 00:20:36.896 gas limit and stuff but so 00:20:36.957 --> 00:20:38.298 anyway back back to the 00:20:38.458 --> 00:20:40.157 governance the idea I had I 00:20:40.178 --> 00:20:40.878 never got to the fucking 00:20:40.939 --> 00:20:41.598 idea that I was going to 00:20:41.638 --> 00:20:42.919 tell you about um because 00:20:42.939 --> 00:20:43.700 we talked about all that 00:20:43.720 --> 00:20:46.340 other shit before but my my grand idea 00:20:46.721 --> 00:20:47.501 So, you know, 00:20:47.521 --> 00:20:48.303 it occurred to me this 00:20:48.363 --> 00:20:50.625 morning when I was getting, you know, 00:20:50.925 --> 00:20:52.928 an alert when Alora started 00:20:53.229 --> 00:20:54.390 upgrading and obviously 00:20:54.770 --> 00:20:56.893 there's like a no data alert, right? 00:20:56.952 --> 00:20:59.736 So I propose that we, 00:21:00.237 --> 00:21:01.397 I'm not sure if this is possible, 00:21:01.519 --> 00:21:03.641 but potentially you can then, 00:21:04.442 --> 00:21:05.603 once you've collected those 00:21:05.702 --> 00:21:06.884 upgrade alerts, 00:21:09.118 --> 00:21:13.361 uh, plans from the, uh, chain on chain. 00:21:13.421 --> 00:21:15.500 So once it is a plan, uh, 00:21:16.061 --> 00:21:17.501 pick it up with low level discovery, 00:21:18.461 --> 00:21:21.643 note the block, keep monitoring the, um, 00:21:22.663 --> 00:21:25.005 the, the chain blocks. 00:21:25.545 --> 00:21:33.607 And then once it gets to the, uh, the, um, 00:21:33.708 --> 00:21:37.288 upgrade height, have a, uh, 00:21:39.190 --> 00:21:41.132 an alert or just a 00:21:41.731 --> 00:21:43.273 information that comes up 00:21:44.434 --> 00:21:45.476 like an information alert. 00:21:45.496 --> 00:21:48.218 So you can make that out of the, you know, 00:21:48.258 --> 00:21:50.640 pager duty alerts and then 00:21:50.720 --> 00:21:52.340 put the node into maintenance mode, 00:21:52.821 --> 00:21:53.962 the host in the maintenance mode. 00:21:55.364 --> 00:21:57.726 Yeah. 00:21:57.746 --> 00:21:58.926 So that you're not getting alerts. 00:21:59.948 --> 00:22:01.148 How the fuck do you put a 00:22:01.169 --> 00:22:02.130 node into a maintenance mode? 00:22:02.836 --> 00:22:03.695 Yeah, like I've got, 00:22:04.196 --> 00:22:06.057 so I got hosts that go into 00:22:06.096 --> 00:22:07.298 maintenance mode every day 00:22:07.637 --> 00:22:09.558 because of the backup scripts. 00:22:09.598 --> 00:22:10.578 Like sometimes it takes an 00:22:10.638 --> 00:22:13.619 hour to fucking restart an archive. 00:22:13.640 --> 00:22:15.361 Is that in Zabbix? 00:22:15.441 --> 00:22:16.681 Yeah, man, Zabbix. 00:22:16.701 --> 00:22:17.301 Holy shit. 00:22:17.862 --> 00:22:18.061 Yeah, 00:22:18.102 --> 00:22:19.142 so actually I think you can do it 00:22:19.162 --> 00:22:20.403 with the API as well. 00:22:20.482 --> 00:22:22.284 So I've been thinking about 00:22:22.344 --> 00:22:23.384 various things with that. 00:22:24.644 --> 00:22:25.365 Maintenance, baby. 00:22:26.000 --> 00:22:26.682 That's a good idea. 00:22:26.942 --> 00:22:28.306 I like that. 00:22:28.326 --> 00:22:30.289 I thought about this too, actually. 00:22:30.509 --> 00:22:31.471 You can go maintenance mode. 00:22:31.491 --> 00:22:32.654 It will keep collecting data, 00:22:32.674 --> 00:22:34.077 but it won't make alerts. 00:22:34.769 --> 00:22:35.411 right just to shut the 00:22:35.431 --> 00:22:37.412 alerts off yeah yeah the 00:22:37.471 --> 00:22:38.732 idea I had with that was I 00:22:38.752 --> 00:22:39.794 mean one thing that we do 00:22:39.933 --> 00:22:41.115 on all other chains except 00:22:41.134 --> 00:22:43.375 for cosmos space chains is 00:22:43.856 --> 00:22:45.758 I always ping a public rpc 00:22:46.218 --> 00:22:47.378 right so I always look at 00:22:47.398 --> 00:22:49.280 the gap between me and 00:22:49.540 --> 00:22:51.342 quote-unquote public so I 00:22:51.382 --> 00:22:52.502 did add that to the cosmos 00:22:52.843 --> 00:22:54.163 base thing and I just 00:22:54.203 --> 00:22:55.785 basically it's easy on 00:22:55.805 --> 00:22:56.625 mainnet because you can 00:22:56.645 --> 00:22:57.807 just kind of go you can kind of 00:22:58.207 --> 00:22:59.127 Thanks to Cosmo's directory 00:22:59.147 --> 00:22:59.867 and the great work there. 00:23:00.188 --> 00:23:02.209 You can go grab a base thing 00:23:02.229 --> 00:23:03.449 and just pull the network thing and say, 00:23:03.469 --> 00:23:04.869 Hey, I'm going to hit this every minute, 00:23:05.470 --> 00:23:05.630 right? 00:23:05.650 --> 00:23:06.951 I'm going to hit this chain every minute. 00:23:07.851 --> 00:23:09.392 It doesn't have to hit the shit out of it. 00:23:09.412 --> 00:23:09.852 I just want to get the 00:23:09.872 --> 00:23:10.892 latest height every minute 00:23:10.971 --> 00:23:13.232 from this chain and then do 00:23:13.272 --> 00:23:14.874 a gap between where my, 00:23:14.913 --> 00:23:16.674 this node is the specific node, right? 00:23:16.835 --> 00:23:17.414 And that thing. 00:23:18.215 --> 00:23:20.236 And if that, 00:23:20.856 --> 00:23:22.497 so what you could do is do 00:23:22.517 --> 00:23:23.458 that same stall and, 00:23:24.320 --> 00:23:26.484 But if the public is also stalled, 00:23:26.925 --> 00:23:29.089 then make that a dependency to say, hey, 00:23:29.170 --> 00:23:31.674 I'm not going to alert on 00:23:31.714 --> 00:23:34.380 that unless the public is also stalled. 00:23:36.126 --> 00:23:36.787 So you can do something 00:23:36.807 --> 00:23:37.748 along that lines to say, 00:23:37.907 --> 00:23:39.528 if the public is stalled and I'm stalled, 00:23:39.548 --> 00:23:40.189 that's okay. 00:23:41.388 --> 00:23:42.430 Or it might be okay, 00:23:42.730 --> 00:23:44.691 but if it happens for fifteen minutes, 00:23:44.851 --> 00:23:45.431 then I want to know. 00:23:45.451 --> 00:23:46.491 Or if it happens for thirty minutes, 00:23:46.571 --> 00:23:47.251 then I want to know. 00:23:47.672 --> 00:23:48.512 Or I might not be worried 00:23:48.532 --> 00:23:49.752 because the public is 00:23:49.772 --> 00:23:50.294 stalled and I'm stalled. 00:23:50.334 --> 00:23:51.693 Now that's not a no data situation, 00:23:51.713 --> 00:23:52.275 because you still have to 00:23:52.315 --> 00:23:53.035 check if there's no data, 00:23:53.055 --> 00:23:53.894 if the public is down, 00:23:53.914 --> 00:23:56.195 because publics are really, I think, 00:23:56.596 --> 00:23:57.136 like public garbage. 00:23:57.196 --> 00:23:58.637 Yeah, they're like all over the place. 00:23:58.657 --> 00:23:59.438 They're all over the place, 00:23:59.458 --> 00:24:00.278 like they're fucking miserable. 00:24:01.818 --> 00:24:02.638 Which is why we don't run them. 00:24:04.479 --> 00:24:06.220 But it wouldn't be too hard. 00:24:06.240 --> 00:24:08.701 We could just make two items 00:24:08.961 --> 00:24:10.761 basically running or stalled. 00:24:11.261 --> 00:24:13.063 Yeah, it's a real simple start of that. 00:24:13.143 --> 00:24:14.323 And then say if it's stalled 00:24:14.403 --> 00:24:15.324 and also the public stalled, 00:24:15.344 --> 00:24:16.284 it's a pretty good guess 00:24:16.304 --> 00:24:17.345 that that's not the case. 00:24:17.404 --> 00:24:18.164 And maybe I'm going to wait 00:24:18.265 --> 00:24:19.306 then for fifteen minutes 00:24:19.326 --> 00:24:20.306 before I do an alert where 00:24:20.766 --> 00:24:21.686 where otherwise if the 00:24:21.727 --> 00:24:23.467 public is moving and that gap grows, 00:24:23.787 --> 00:24:25.008 if that gap grows after 00:24:25.028 --> 00:24:26.588 three after three polls, 00:24:26.669 --> 00:24:27.909 I want to know because then 00:24:27.969 --> 00:24:29.250 something is definitely wrong with that. 00:24:30.109 --> 00:24:31.451 Um, it's, it's relying on the public, 00:24:31.490 --> 00:24:32.771 but honestly, for the most part, 00:24:32.872 --> 00:24:33.712 if you're hitting a public 00:24:33.813 --> 00:24:34.993 once a minute or every thirty seconds, 00:24:35.334 --> 00:24:37.035 just for a height, it's no big deal. 00:24:37.174 --> 00:24:38.036 Like, it's very, 00:24:38.536 --> 00:24:39.836 you're literally talking what, like, 00:24:40.617 --> 00:24:41.557 fucking hundred twenty hits 00:24:41.578 --> 00:24:42.499 a minute or something like that. 00:24:42.538 --> 00:24:43.380 It's, it's no big deal. 00:24:44.045 --> 00:24:45.565 Um, so that's another option to do too. 00:24:45.684 --> 00:24:47.665 I do that on the EVM side. 00:24:48.526 --> 00:24:50.165 Um, and then there's a, 00:24:50.405 --> 00:24:51.767 there's a order within there. 00:24:51.807 --> 00:24:52.807 I think we did in Solana too, don't we? 00:24:53.606 --> 00:24:53.787 Um, 00:24:54.346 --> 00:24:55.667 where we kind of look and see what the 00:24:55.748 --> 00:24:56.567 public height is and then 00:24:56.587 --> 00:24:57.768 also do a gap analysis and 00:24:57.788 --> 00:24:58.847 if that gap grows and all 00:24:58.867 --> 00:24:59.448 that kind of shit. 00:25:00.008 --> 00:25:01.409 Um, but if you want to know, 00:25:01.489 --> 00:25:02.128 you're doing that, 00:25:02.229 --> 00:25:03.909 I assume we're doing that. 00:25:04.409 --> 00:25:05.130 I can't remember if I'm 00:25:05.150 --> 00:25:05.930 doing that in Solana or not. 00:25:05.950 --> 00:25:06.670 No, I think I do. 00:25:06.710 --> 00:25:07.509 I do a gap in Solana. 00:25:07.549 --> 00:25:09.871 I can go look, but, um, but that's a good, 00:25:09.911 --> 00:25:10.770 but that's a good idea. 00:25:11.191 --> 00:25:11.290 Um, 00:25:12.344 --> 00:25:13.586 So that's, that's, 00:25:14.307 --> 00:25:15.368 that's something we can do. 00:25:16.069 --> 00:25:16.871 I I've just never, 00:25:17.692 --> 00:25:19.294 I've never actually worked to, yeah, 00:25:19.334 --> 00:25:19.694 I have a, 00:25:19.755 --> 00:25:21.257 I have a gap height difference 00:25:21.277 --> 00:25:22.538 to public within Solana. 00:25:23.631 --> 00:25:24.332 which is pretty good since 00:25:24.352 --> 00:25:25.532 those blocks are fucking faster. 00:25:25.553 --> 00:25:26.713 Well, they have three hundred milliseconds, 00:25:27.054 --> 00:25:28.015 four hundred eighty milliseconds. 00:25:29.496 --> 00:25:31.417 I know that Solana is a real 00:25:31.438 --> 00:25:32.317 fucking pain in the ass 00:25:32.337 --> 00:25:34.400 because you can like fall 00:25:34.440 --> 00:25:35.961 behind real quick when you 00:25:36.101 --> 00:25:38.762 like when I know just restarting a node, 00:25:38.803 --> 00:25:39.963 you can get like a thousand 00:25:39.983 --> 00:25:41.425 blocks behind and it 00:25:41.465 --> 00:25:42.645 doesn't catch up very fast. 00:25:42.727 --> 00:25:43.606 You can throw all the 00:25:43.626 --> 00:25:44.567 hardware in the world that 00:25:44.667 --> 00:25:45.588 it still just does not 00:25:45.628 --> 00:25:46.569 catch up very fast. 00:25:46.670 --> 00:25:46.809 So. 00:25:47.269 --> 00:25:48.790 If you restart, if you restart that binary, 00:25:48.810 --> 00:25:52.872 it's, it's, it's, it's, it's, it's, it's, 00:25:52.971 --> 00:25:56.834 it's, it's, it's, it's, it's, it's, it's, 00:25:56.874 --> 00:26:00.535 it's, it's, it's, it's, it's, it's, it's, 00:26:08.503 --> 00:26:10.365 um yeah yeah so annoying it 00:26:10.425 --> 00:26:12.008 is it's it takes a long 00:26:12.048 --> 00:26:13.950 time for it to it does that 00:26:14.049 --> 00:26:14.911 network check and it does 00:26:14.931 --> 00:26:15.691 all that shit and it 00:26:15.731 --> 00:26:16.432 connects out to all these 00:26:16.471 --> 00:26:17.814 fucking peers and I don't 00:26:17.854 --> 00:26:18.994 know it's kind of like a 00:26:19.915 --> 00:26:20.636 like starting like a 00:26:20.836 --> 00:26:23.038 fucking sixty chevy like 00:26:23.058 --> 00:26:24.460 you just gotta pump the gas 00:26:24.480 --> 00:26:25.701 a couple times you gotta 00:26:25.721 --> 00:26:26.782 like like starting my 00:26:26.803 --> 00:26:27.884 seventies fucking lawnmower 00:26:28.183 --> 00:26:30.425 you gotta lean on it yeah so 00:26:33.858 --> 00:26:35.182 I think the governance idea is a good idea, 00:26:35.201 --> 00:26:35.362 though. 00:26:35.481 --> 00:26:36.924 That maintenance idea is 00:26:36.964 --> 00:26:37.546 pretty interesting. 00:26:40.412 --> 00:26:41.594 I don't know if you can 00:26:41.733 --> 00:26:43.096 enter maintenance through... 00:26:45.810 --> 00:26:47.152 Do the API? 00:26:47.392 --> 00:26:48.613 Yeah, you can with the API. 00:26:48.813 --> 00:26:51.375 Well, you have to put in a window. 00:26:51.555 --> 00:26:53.257 I don't think you can just 00:26:53.957 --> 00:26:55.238 start maintenance, stop maintenance. 00:26:55.357 --> 00:26:57.220 I think you have to create a 00:26:57.259 --> 00:26:57.900 maintenance window, 00:26:57.940 --> 00:26:59.181 which is how I use it at the moment. 00:26:59.741 --> 00:27:00.522 I don't know if there's any 00:27:00.583 --> 00:27:01.363 other mechanism with 00:27:01.403 --> 00:27:02.163 maintenance to be able to 00:27:02.183 --> 00:27:03.064 put it into maintenance. 00:27:04.005 --> 00:27:07.028 So, yeah, I don't know. 00:27:07.067 --> 00:27:11.131 Maybe you can apply a tag and then... 00:27:12.596 --> 00:27:13.538 Remove the tag. 00:27:14.077 --> 00:27:14.719 I don't know if you can do 00:27:14.739 --> 00:27:15.278 that with alerts. 00:27:15.298 --> 00:27:16.759 Filter out the tag because it's, you know, 00:27:16.779 --> 00:27:17.621 it's a known upgrade height 00:27:17.641 --> 00:27:18.280 or something like that. 00:27:18.641 --> 00:27:18.781 Yeah. 00:27:18.922 --> 00:27:19.201 Yeah. 00:27:19.281 --> 00:27:19.481 Yeah. 00:27:19.541 --> 00:27:20.262 Like, you know, 00:27:20.843 --> 00:27:22.743 status upgrade or something like that. 00:27:22.864 --> 00:27:24.605 And then filter that tag 00:27:25.445 --> 00:27:26.527 because we filter certain, 00:27:26.686 --> 00:27:27.827 like I just have tags that 00:27:27.867 --> 00:27:28.969 I have for filtering, 00:27:29.148 --> 00:27:30.430 filtering out alerts for 00:27:30.470 --> 00:27:31.150 things that I don't want to 00:27:31.170 --> 00:27:31.730 be annoyed by. 00:27:31.990 --> 00:27:33.192 Just add the tag and then it 00:27:33.251 --> 00:27:34.712 no longer does the alerting. 00:27:35.513 --> 00:27:36.634 This is the quality of node 00:27:36.673 --> 00:27:37.515 operations that you get 00:27:37.555 --> 00:27:38.695 from professional validators. 00:27:38.715 --> 00:27:38.816 Yeah. 00:27:40.167 --> 00:27:41.048 We spend a lot of time 00:27:41.088 --> 00:27:42.148 scratching our head about 00:27:42.250 --> 00:27:44.431 how to not be alerted. 00:27:45.471 --> 00:27:46.492 If you're bored right now, 00:27:46.553 --> 00:27:48.554 you should delegate to the 00:27:48.574 --> 00:27:50.035 two of us because there's 00:27:50.055 --> 00:27:50.976 not a lot of... Honestly, 00:27:51.016 --> 00:27:53.336 there's a lot of good work 00:27:53.477 --> 00:27:54.577 that I do for monitoring 00:27:54.597 --> 00:27:55.118 and stuff like this. 00:27:55.199 --> 00:27:56.278 I think we sometimes take it 00:27:56.298 --> 00:27:58.540 a little bit to the extreme, I think, 00:27:58.601 --> 00:28:00.182 versus others, but it's all good. 00:28:00.301 --> 00:28:00.481 Well, 00:28:01.063 --> 00:28:03.944 I think most just rely on other 00:28:03.984 --> 00:28:07.768 people's tools and creations, which is... 00:28:08.976 --> 00:28:11.017 A little bit disappointing, but also like, 00:28:11.057 --> 00:28:11.396 you know, 00:28:11.696 --> 00:28:14.198 how much value is there in 00:28:14.258 --> 00:28:15.478 recreating the wheel? 00:28:15.817 --> 00:28:17.219 Thirty five thousand times. 00:28:17.419 --> 00:28:17.818 So. 00:28:17.858 --> 00:28:18.118 Right. 00:28:18.219 --> 00:28:18.398 Right. 00:28:18.778 --> 00:28:20.440 I mean, if it ain't broke, don't fix it. 00:28:20.480 --> 00:28:21.619 But also you should make an 00:28:21.661 --> 00:28:22.540 effort to know what's under 00:28:22.580 --> 00:28:23.641 the hood and how it works. 00:28:24.181 --> 00:28:25.221 I would like to. 00:28:25.261 --> 00:28:26.162 There's some tender to the 00:28:26.182 --> 00:28:28.163 items I would like to bring into Zabbix. 00:28:28.182 --> 00:28:31.183 So like consecutive miss blocks. 00:28:32.981 --> 00:28:33.942 half that functionality I 00:28:33.962 --> 00:28:35.983 think half consecutive miss 00:28:36.084 --> 00:28:37.525 blocks is very difficult 00:28:37.565 --> 00:28:39.865 yeah I know I don't think 00:28:39.885 --> 00:28:41.067 it's possible to be honest 00:28:41.106 --> 00:28:42.106 half that functionality we 00:28:42.146 --> 00:28:45.368 have yeah consecutive miss 00:28:45.409 --> 00:28:46.829 blocks would be would be a 00:28:46.890 --> 00:28:48.490 challenge I mean we're only 00:28:48.529 --> 00:28:49.530 pulling every whatever 00:28:49.570 --> 00:28:51.172 fifteen thirty seconds so I 00:28:51.192 --> 00:28:51.912 don't know how you would do 00:28:51.932 --> 00:28:52.712 that unless you're 00:28:53.606 --> 00:28:56.909 It would be so nice if there 00:28:56.929 --> 00:28:58.109 was like some additional 00:28:58.309 --> 00:29:01.291 functionality in Zabbix to 00:29:01.311 --> 00:29:02.031 be able to listen to a 00:29:02.051 --> 00:29:04.713 WebSocket and do some stuff off it. 00:29:05.034 --> 00:29:06.915 But could you imagine how 00:29:06.935 --> 00:29:07.935 many fucking WebSockets 00:29:07.955 --> 00:29:08.616 you'd be listening to? 00:29:08.997 --> 00:29:11.999 That's what we do now. 00:29:13.516 --> 00:29:15.597 Yeah, but we do, like, a HTTP poll. 00:29:15.678 --> 00:29:16.637 No, I meant, like, 00:29:16.999 --> 00:29:18.420 in tender duty and little safety things, 00:29:18.440 --> 00:29:18.559 right? 00:29:18.660 --> 00:29:19.359 Oh, yeah, yeah. 00:29:19.721 --> 00:29:21.582 But, I mean, as, you know, 00:29:21.602 --> 00:29:22.682 this type of system, 00:29:23.303 --> 00:29:25.525 I don't think would be 00:29:25.565 --> 00:29:28.567 great for listening to... Yeah. 00:29:28.606 --> 00:29:29.428 I mean, again, 00:29:29.448 --> 00:29:30.067 you could create a 00:29:30.147 --> 00:29:31.288 third-party API that 00:29:31.328 --> 00:29:32.809 listens and then reports back. 00:29:33.151 --> 00:29:33.750 Sure, sure. 00:29:33.931 --> 00:29:35.673 I mean, they work well together, 00:29:35.712 --> 00:29:36.952 so I don't really have much 00:29:36.993 --> 00:29:37.493 of a problem with it. 00:29:37.673 --> 00:29:38.815 Like, I don't mind... I mean, 00:29:38.894 --> 00:29:39.715 tender duty's been great. 00:29:39.796 --> 00:29:40.115 Like, even... 00:29:41.395 --> 00:29:42.675 just, you know, just still, 00:29:42.996 --> 00:29:43.896 still use it and still get 00:29:43.916 --> 00:29:44.797 alerts based off of it. 00:29:45.116 --> 00:29:46.057 I stopped with the RPC 00:29:46.076 --> 00:29:47.896 alerts and all that kind of shit. 00:29:47.916 --> 00:29:49.538 Cause I don't really need 00:29:49.917 --> 00:29:51.478 tender duty warning me about that stuff. 00:29:51.498 --> 00:29:53.198 Cause I have better, not better. 00:29:53.377 --> 00:29:54.459 I have different solutions for that, 00:29:54.999 --> 00:29:56.219 but for consecutive miss 00:29:56.259 --> 00:29:57.378 blocks or things like that, 00:29:57.419 --> 00:29:58.818 like we were missing on, 00:29:58.939 --> 00:29:59.578 I forget what the hell 00:29:59.818 --> 00:30:00.839 fucking test net it was. 00:30:01.359 --> 00:30:02.279 It just needed to restart. 00:30:02.420 --> 00:30:03.640 Like it was just, it was a shit in the bed, 00:30:03.819 --> 00:30:04.819 restarted and everything was fine. 00:30:05.579 --> 00:30:06.721 But that would, that would have taken, 00:30:06.760 --> 00:30:07.681 I would have had to get to, 00:30:08.839 --> 00:30:10.121 If that was Zabbix only, 00:30:10.621 --> 00:30:11.601 I wouldn't have known about 00:30:11.661 --> 00:30:13.422 that until it got to like five percent. 00:30:14.021 --> 00:30:15.762 You know what I mean? 00:30:15.803 --> 00:30:16.823 Where I knew about it within 00:30:17.182 --> 00:30:18.384 like I missed twenty five 00:30:18.403 --> 00:30:19.284 blocks in a row. 00:30:19.723 --> 00:30:20.384 I knew about it instantly. 00:30:20.424 --> 00:30:22.085 I restarted within four 00:30:22.144 --> 00:30:23.125 seconds and it was back. 00:30:23.425 --> 00:30:23.945 You know what I mean? 00:30:23.986 --> 00:30:24.826 Where at five percent, 00:30:24.865 --> 00:30:25.386 that thing would have been 00:30:25.406 --> 00:30:26.967 shit in the bed for, I don't know, 00:30:27.047 --> 00:30:27.548 a while. 00:30:28.367 --> 00:30:30.189 And maybe I could always we 00:30:30.209 --> 00:30:31.808 could always lower that, too. 00:30:32.549 --> 00:30:33.289 But the percentages are 00:30:33.329 --> 00:30:34.710 tough because it's it's 00:30:34.750 --> 00:30:37.692 always like I'm even at one percent. 00:30:37.791 --> 00:30:38.112 I don't know. 00:30:39.449 --> 00:30:40.109 I don't know. 00:30:40.390 --> 00:30:41.630 It's difficult to... It's 00:30:41.650 --> 00:30:42.431 the difference between time 00:30:42.451 --> 00:30:44.112 series data and just taking 00:30:44.132 --> 00:30:44.991 a point-of-time slice and 00:30:45.011 --> 00:30:46.512 trying to make decisions based off of it. 00:30:46.532 --> 00:30:52.376 Ah, yes. 00:30:52.537 --> 00:30:53.376 Anyway. 00:30:53.396 --> 00:30:54.198 I'm a little bit distracted. 00:30:54.577 --> 00:30:55.459 We could talk Zabbix all day. 00:30:56.459 --> 00:30:56.679 I know. 00:30:56.739 --> 00:30:58.941 It's probably something that 00:30:59.060 --> 00:31:00.260 absolutely no one listening 00:31:00.280 --> 00:31:00.882 to this would find 00:31:01.961 --> 00:31:03.762 interesting or valuable whatsoever. 00:31:03.782 --> 00:31:04.023 I know. 00:31:04.044 --> 00:31:05.584 I know. 00:31:05.604 --> 00:31:06.444 It's fucking good, though. 00:31:07.265 --> 00:31:07.986 It's such good stuff. 00:31:08.839 --> 00:31:11.000 yeah it makes it's it's a 00:31:11.040 --> 00:31:12.102 hard sell though like when 00:31:12.682 --> 00:31:14.002 uh so I remember shilling 00:31:14.682 --> 00:31:16.384 zabix to you and others and 00:31:16.924 --> 00:31:18.546 I was I was like oh man 00:31:18.566 --> 00:31:20.007 fucking grafana works great 00:31:20.626 --> 00:31:23.409 it grafana grafana works 00:31:23.469 --> 00:31:25.549 great grafana works great 00:31:25.789 --> 00:31:29.211 it does um but each of 00:31:29.231 --> 00:31:31.854 those alerts is handcrafted 00:31:32.867 --> 00:31:35.170 and when you when you and 00:31:35.190 --> 00:31:35.990 there's temple ties don't 00:31:36.010 --> 00:31:37.372 get me wrong this was my 00:31:37.451 --> 00:31:39.032 selling point this was 00:31:39.113 --> 00:31:40.354 exactly my selling point 00:31:40.733 --> 00:31:42.215 that was the thing that I 00:31:42.296 --> 00:31:44.317 hated about grafana yeah 00:31:44.356 --> 00:31:45.878 was that the alerting is 00:31:46.019 --> 00:31:48.721 bespoke every time every 00:31:48.740 --> 00:31:51.123 time so so like I have in 00:31:51.143 --> 00:31:52.364 my grafana I have this like 00:31:53.023 --> 00:31:56.106 node dump which basically 00:31:56.146 --> 00:31:57.948 takes every single node of 00:31:58.008 --> 00:31:59.630 a type like every cosmos node 00:32:00.494 --> 00:32:01.836 and it dumps it all into one 00:32:01.875 --> 00:32:02.916 fucking miserable graph 00:32:02.936 --> 00:32:04.397 with a thousand fucking lines on it. 00:32:04.718 --> 00:32:06.199 And if something goes above X, 00:32:06.239 --> 00:32:06.940 then it alerts. 00:32:07.500 --> 00:32:08.441 That's really what you have to do. 00:32:08.520 --> 00:32:08.760 So you have, 00:32:09.141 --> 00:32:09.761 so you can have all your 00:32:09.781 --> 00:32:11.762 beautiful dashboards and then separately, 00:32:12.203 --> 00:32:13.724 you kind of have to have like a, 00:32:15.006 --> 00:32:16.646 oh shit dashboard that has 00:32:16.707 --> 00:32:18.208 all these fucking ugly ass 00:32:18.228 --> 00:32:19.528 graphs on it that, you know, 00:32:19.588 --> 00:32:23.071 in case something explodes, do this. 00:32:23.112 --> 00:32:24.692 So, but I hate it so much. 00:32:25.133 --> 00:32:26.834 I hate it. 00:32:26.854 --> 00:32:27.075 But I mean, 00:32:27.115 --> 00:32:28.375 I don't have any Grafana anyway, 00:32:28.435 --> 00:32:28.796 by the way. 00:32:29.560 --> 00:32:30.721 No, I use it a lot. 00:32:30.861 --> 00:32:31.020 Well, 00:32:31.101 --> 00:32:33.642 one thing I do like about that is for 00:32:33.701 --> 00:32:35.883 visualizations of chain 00:32:35.923 --> 00:32:37.383 performance and node performance, 00:32:37.823 --> 00:32:39.083 when it comes to some 00:32:39.123 --> 00:32:40.104 specific blockchain data, 00:32:40.624 --> 00:32:42.683 the graphing in there is so 00:32:42.723 --> 00:32:43.463 much better than Xamarin. 00:32:43.624 --> 00:32:45.204 Yeah, of course. 00:32:45.244 --> 00:32:45.525 Actually, 00:32:45.605 --> 00:32:46.825 vServant has some different 00:32:46.865 --> 00:32:47.464 graphing options. 00:32:47.664 --> 00:32:47.825 I know. 00:32:48.465 --> 00:32:49.286 That's one reason I want to... 00:32:49.905 --> 00:32:50.586 kind of move forward. 00:32:51.366 --> 00:32:52.666 Because the stuff that I've 00:32:52.707 --> 00:32:53.807 built over time in Grafana 00:32:53.867 --> 00:32:55.567 and the visualizations of 00:32:55.627 --> 00:32:57.169 understanding when something's wrong, 00:32:57.269 --> 00:32:58.048 like how do I figure out 00:32:58.088 --> 00:32:59.369 exactly what the fuck is wrong? 00:32:59.890 --> 00:33:00.750 That's the tool that you 00:33:00.789 --> 00:33:01.631 want to figure out what the 00:33:01.651 --> 00:33:02.171 fuck is wrong. 00:33:02.851 --> 00:33:03.851 It's really difficult to do 00:33:03.871 --> 00:33:04.271 that in Zabbix. 00:33:04.291 --> 00:33:05.071 There's just not enough. 00:33:05.092 --> 00:33:06.271 You're creating an item for 00:33:06.291 --> 00:33:07.553 every item and I can't just 00:33:07.613 --> 00:33:09.953 query off a... There's five new... 00:33:10.773 --> 00:33:11.634 five new metrics in 00:33:11.673 --> 00:33:12.734 Prometheus I want to bring in. 00:33:12.795 --> 00:33:14.355 I want to just drag those in and like, 00:33:14.414 --> 00:33:15.215 you know, 00:33:15.236 --> 00:33:16.256 graph those against each other 00:33:16.316 --> 00:33:17.957 and like find blah, blah, blah, 00:33:17.977 --> 00:33:18.517 blah like that. 00:33:18.696 --> 00:33:19.797 It takes like no time. 00:33:19.836 --> 00:33:21.018 And that's really valuable. 00:33:22.518 --> 00:33:23.739 And I know you can also bring, 00:33:24.479 --> 00:33:25.440 you can also bring Zabbix 00:33:25.519 --> 00:33:27.059 data into Grafana, right? 00:33:27.220 --> 00:33:28.661 Like you can use Zabbix as a data source. 00:33:29.381 --> 00:33:30.481 I think you can. 00:33:30.801 --> 00:33:31.002 Yeah. 00:33:31.403 --> 00:33:32.663 That might be interesting too, because, 00:33:33.305 --> 00:33:34.506 but it's not, is it? 00:33:34.625 --> 00:33:36.688 It's, it's not, I don't know. 00:33:36.708 --> 00:33:37.729 It's not time series though, right? 00:33:37.989 --> 00:33:38.148 Yeah. 00:33:38.169 --> 00:33:39.190 It's cause it's not time series. 00:33:39.230 --> 00:33:40.411 It wouldn't, it wouldn't work. 00:33:40.631 --> 00:33:41.592 You'd be taking like, 00:33:42.673 --> 00:33:44.213 you're far better off just 00:33:44.233 --> 00:33:46.435 to pull the fucking, the, um, 00:33:47.116 --> 00:33:49.239 Prometheus time series data 00:33:49.318 --> 00:33:49.858 from the node. 00:33:50.359 --> 00:33:52.362 But I was looking at also, I don't know. 00:33:52.442 --> 00:33:53.201 I don't want to turn into a 00:33:53.221 --> 00:33:54.163 fucking Zabbix episode. 00:33:54.202 --> 00:33:55.663 Yeah. 00:33:56.464 --> 00:33:57.326 you're in my relationship is 00:33:57.346 --> 00:33:58.766 like I mean given given any 00:33:58.806 --> 00:34:00.807 opportunity we will find 00:34:00.866 --> 00:34:01.827 our relationship together 00:34:01.948 --> 00:34:02.647 is at least twenty five 00:34:02.667 --> 00:34:05.750 percent um but but the uh 00:34:05.769 --> 00:34:06.650 there is that that time 00:34:06.690 --> 00:34:07.810 series database that runs 00:34:08.530 --> 00:34:11.693 next to postgres right um 00:34:11.733 --> 00:34:13.012 that you can you can use 00:34:13.032 --> 00:34:13.514 and they'll actually 00:34:13.534 --> 00:34:14.574 compress time series data 00:34:14.594 --> 00:34:15.815 down and like it doesn't it 00:34:15.835 --> 00:34:16.594 doesn't work exactly like 00:34:16.635 --> 00:34:17.596 prometheus but there's like 00:34:17.655 --> 00:34:18.536 a add-on that you can use 00:34:18.556 --> 00:34:19.215 for time series because I'm 00:34:19.235 --> 00:34:20.036 looking at that with going 00:34:20.056 --> 00:34:21.657 to seven it's called like 00:34:21.697 --> 00:34:23.117 time I forget what the fuck 00:34:23.137 --> 00:34:24.059 it's called I should know what it is 00:34:25.418 --> 00:34:27.360 Somebody might yell at me 00:34:27.400 --> 00:34:27.880 here in a second. 00:34:28.481 --> 00:34:30.202 But there's a pretty cool add-on to that. 00:34:30.603 --> 00:34:31.523 And then so you can actually 00:34:31.702 --> 00:34:33.184 use that to push some, 00:34:33.844 --> 00:34:34.985 you can use that to push 00:34:35.284 --> 00:34:40.329 all Zabbix timescale data into that. 00:34:40.909 --> 00:34:42.510 And then it has some compression to it. 00:34:42.650 --> 00:34:44.331 It has much faster querying, like, 00:34:44.670 --> 00:34:45.492 eighty percent faster 00:34:45.512 --> 00:34:46.932 querying and a whole bunch 00:34:46.952 --> 00:34:48.012 of other types of benefits to that. 00:34:48.032 --> 00:34:49.074 But it's outside of Postgres. 00:34:50.278 --> 00:34:52.360 So it, you can look at it. 00:34:52.380 --> 00:34:53.119 So it basically, 00:34:53.360 --> 00:34:54.101 it's not necessarily a 00:34:54.161 --> 00:34:55.400 hundred percent the same as 00:34:55.882 --> 00:34:59.023 how Prometheus work that works though. 00:34:59.063 --> 00:35:00.664 Like it's a, it is time series, 00:35:00.704 --> 00:35:02.304 but I think it's also still taking, well, 00:35:02.324 --> 00:35:03.224 I guess Prometheus taking 00:35:03.244 --> 00:35:04.166 slices of time as well. 00:35:04.945 --> 00:35:05.766 But I'm looking at that for 00:35:05.826 --> 00:35:06.967 this big upgrade is I'm 00:35:07.166 --> 00:35:08.126 kind of building a new 00:35:08.166 --> 00:35:09.708 architecture for how we use 00:35:09.728 --> 00:35:11.809 that Excel and that kind of stuff. 00:35:11.829 --> 00:35:12.469 Cause that looks actually, 00:35:12.570 --> 00:35:14.911 it looks pretty cool to be able to use. 00:35:15.811 --> 00:35:17.552 The fucking upgrade. 00:35:17.632 --> 00:35:17.811 I know. 00:35:19.929 --> 00:35:20.150 Yeah. 00:35:20.170 --> 00:35:21.391 I was so close the other day. 00:35:21.951 --> 00:35:22.751 And then I was like, 00:35:23.853 --> 00:35:24.532 do I really want to be 00:35:24.552 --> 00:35:25.494 doing this at ten o'clock 00:35:25.534 --> 00:35:27.755 at night with a baby? 00:35:27.795 --> 00:35:28.617 Timescale DB. 00:35:28.657 --> 00:35:29.237 That's what it's called. 00:35:30.905 --> 00:35:33.487 Well, I, and really the, I mean, I mean, 00:35:33.547 --> 00:35:35.128 we, I was just looking at the, 00:35:35.347 --> 00:35:36.929 our dependency on this fucking tool set. 00:35:37.048 --> 00:35:39.731 This tool set, we, we pull right now, 00:35:39.751 --> 00:35:41.231 we pull around seven 00:35:41.251 --> 00:35:42.233 hundred values a second. 00:35:43.052 --> 00:35:43.293 Okay. 00:35:43.333 --> 00:35:45.434 So seven hundred, what is that? 00:35:45.994 --> 00:35:48.456 Seven hundred times sixty 00:35:48.577 --> 00:35:51.438 times sixty times twenty four. 00:35:51.699 --> 00:35:53.179 So we're pulling sixty and a 00:35:53.239 --> 00:35:55.222 half million values a day. 00:35:56.954 --> 00:35:59.016 of quite some value of all 00:35:59.036 --> 00:36:01.038 that shit which is which is 00:36:01.137 --> 00:36:02.338 pretty like when you think 00:36:02.378 --> 00:36:03.360 about how much data is 00:36:03.380 --> 00:36:04.800 getting pulled in and how 00:36:04.820 --> 00:36:05.501 much monitoring and 00:36:05.541 --> 00:36:06.302 triggers are going with 00:36:06.342 --> 00:36:07.284 that that's really the 00:36:07.304 --> 00:36:09.144 value of of this of this 00:36:09.184 --> 00:36:10.646 tool set and like how I think 00:36:11.567 --> 00:36:12.748 it's a good compliment to anything, 00:36:12.788 --> 00:36:14.289 whoever's running any sort of monitoring. 00:36:14.789 --> 00:36:16.530 Like there is a load of 00:36:16.570 --> 00:36:17.650 things that are being watched. 00:36:17.951 --> 00:36:18.150 Right. 00:36:18.231 --> 00:36:20.052 And sometimes that can be alert overload. 00:36:20.072 --> 00:36:20.833 Cause I've found that with 00:36:20.853 --> 00:36:22.653 like upgrades and, you know, 00:36:22.875 --> 00:36:23.775 other types of things or 00:36:23.835 --> 00:36:25.115 smart data or this or that. 00:36:25.135 --> 00:36:26.856 And like, you can get, holy shit, man. 00:36:26.896 --> 00:36:29.059 When, when I dumped the database, 00:36:29.898 --> 00:36:32.121 like yesterday, the day before yesterday, 00:36:33.021 --> 00:36:34.181 it locked to the database 00:36:34.322 --> 00:36:37.384 and fucking the hundreds of 00:36:37.423 --> 00:36:38.945 alerts came through on, 00:36:39.204 --> 00:36:41.047 on page of Judy hundreds. 00:36:43.108 --> 00:36:43.748 I don't know. 00:36:43.788 --> 00:36:44.889 The server must like, 00:36:45.710 --> 00:36:47.851 it must not like it when 00:36:47.871 --> 00:36:49.351 you disconnect it from the backend. 00:36:49.891 --> 00:36:51.333 So yeah, man. 00:36:51.552 --> 00:36:51.632 Yeah. 00:36:52.833 --> 00:36:54.534 Well that lock is just like fucking, 00:36:54.614 --> 00:36:55.896 it just made everything shit. 00:36:56.115 --> 00:36:57.356 All the no data is just went 00:36:57.496 --> 00:36:58.536 fucking ballistic. 00:36:58.577 --> 00:36:58.876 Yeah. 00:36:58.958 --> 00:36:59.757 I guess if it doesn't, 00:37:00.038 --> 00:37:00.938 if it can't make a connection, 00:37:00.958 --> 00:37:01.478 it's a no data. 00:37:01.559 --> 00:37:01.778 Right. 00:37:01.818 --> 00:37:03.940 So it doesn't, it doesn't see a record. 00:37:03.960 --> 00:37:05.581 So therefore. 00:37:05.722 --> 00:37:05.802 Yeah. 00:37:05.822 --> 00:37:06.422 I'm letting you know. 00:37:06.442 --> 00:37:06.842 Lost it. 00:37:07.161 --> 00:37:08.362 Totally fucking lost it. 00:37:08.802 --> 00:37:09.023 Yeah. 00:37:09.043 --> 00:37:09.664 It was hilarious. 00:37:10.605 --> 00:37:10.864 Yeah. 00:37:11.943 --> 00:37:14.824 I know yeah so uh our 00:37:14.905 --> 00:37:17.427 newsman is incredibly busy 00:37:17.487 --> 00:37:18.447 at work at the moment he's 00:37:18.507 --> 00:37:20.708 taking jobs yeah well I 00:37:20.748 --> 00:37:21.909 mean two jobs I don't know 00:37:21.929 --> 00:37:22.971 if it's two jobs but he's 00:37:23.070 --> 00:37:24.731 taken on a new role in his 00:37:25.072 --> 00:37:26.152 in his other job and then 00:37:26.172 --> 00:37:27.753 good for him it's quite 00:37:27.793 --> 00:37:28.695 intense at the moment I 00:37:28.715 --> 00:37:30.255 believe so hopefully it's a 00:37:30.275 --> 00:37:32.817 good uh move maybe the 00:37:32.856 --> 00:37:34.217 other job has been a dad I don't know 00:37:36.351 --> 00:37:38.012 I do know that since we've 00:37:38.032 --> 00:37:39.373 been talking about our monitoring, 00:37:39.594 --> 00:37:40.755 absolutely no one has 00:37:40.775 --> 00:37:41.835 fucking made a comment. 00:37:41.894 --> 00:37:43.376 So they're either fully 00:37:43.416 --> 00:37:45.336 engaged and absolutely 00:37:45.376 --> 00:37:46.697 loving this content or do 00:37:46.737 --> 00:37:47.717 not give a fuck. 00:37:47.958 --> 00:37:48.759 And they're also watching 00:37:48.818 --> 00:37:50.860 SpongeBob on the side. 00:37:50.900 --> 00:37:52.081 We do have some other things, right? 00:37:54.572 --> 00:37:58.396 Well, yeah, I don't know. 00:37:58.416 --> 00:37:59.175 Do we? 00:37:59.195 --> 00:38:01.858 It has been pretty slow at the moment. 00:38:01.898 --> 00:38:03.579 There's new networks and stuff, 00:38:03.619 --> 00:38:05.280 but everything... I've 00:38:05.340 --> 00:38:07.081 noticed there is a bit of a... 00:38:08.364 --> 00:38:11.266 hold on uh anything 00:38:11.367 --> 00:38:12.748 launching there's a lot of 00:38:12.849 --> 00:38:14.250 like faff in the background 00:38:14.269 --> 00:38:15.510 and people like well we're 00:38:15.530 --> 00:38:17.112 ready to go except we're 00:38:17.132 --> 00:38:18.773 just sort of not going to 00:38:19.353 --> 00:38:21.434 right now right right now 00:38:21.474 --> 00:38:22.195 everyone's waiting for a 00:38:22.275 --> 00:38:23.097 stronger market there are 00:38:23.117 --> 00:38:25.398 going to be a lot of 00:38:25.478 --> 00:38:26.559 launches when the market 00:38:26.579 --> 00:38:28.420 improves like you know when 00:38:28.661 --> 00:38:29.581 when number go up more 00:38:29.621 --> 00:38:31.802 consistently well I think 00:38:31.822 --> 00:38:33.023 the consistency that's a 00:38:33.063 --> 00:38:34.204 good point because like 00:38:34.224 --> 00:38:35.405 we've had some nice little 00:38:35.465 --> 00:38:36.186 pumps here in the last 00:38:36.206 --> 00:38:37.088 couple weeks of like 00:38:37.884 --> 00:38:38.103 I mean, 00:38:38.264 --> 00:38:39.945 they've been pretty large swings to 00:38:39.965 --> 00:38:41.606 be honest, um, 00:38:41.706 --> 00:38:42.987 on a bunch of tokens that we look at, but, 00:38:43.327 --> 00:38:45.228 but, um, they haven't lasted very long. 00:38:45.409 --> 00:38:47.849 Like they've been under two 00:38:47.889 --> 00:38:49.690 or three hours types of pumps, right? 00:38:49.811 --> 00:38:52.413 Like it's been very, very intimate. 00:38:53.052 --> 00:38:54.094 You need some limit orders 00:38:54.313 --> 00:38:56.355 set the day before type of shit. 00:38:57.576 --> 00:38:58.617 Um, yeah. 00:38:58.637 --> 00:38:59.197 So I don't know. 00:38:59.336 --> 00:39:00.137 I don't know exactly. 00:39:01.378 --> 00:39:02.639 And like Ethereum we're back on. 00:39:02.818 --> 00:39:03.579 Well, actually it's, 00:39:04.001 --> 00:39:04.963 It says twenty five hundred now. 00:39:05.025 --> 00:39:05.706 I thought it was under the 00:39:05.786 --> 00:39:06.829 twenty twos or twenty 00:39:06.869 --> 00:39:08.293 threes earlier today. 00:39:08.353 --> 00:39:09.715 So that's been great. 00:39:13.108 --> 00:39:15.989 You know, Ethereum is, 00:39:16.809 --> 00:39:17.568 I really thought it would 00:39:17.588 --> 00:39:18.768 have had like a bit more of 00:39:18.789 --> 00:39:20.289 a bounce back from, you know, 00:39:20.329 --> 00:39:21.969 the three thousands when, 00:39:21.989 --> 00:39:22.929 because it was a pretty 00:39:22.969 --> 00:39:24.710 savage drop from like, what, thirty, 00:39:25.351 --> 00:39:27.831 thirty three, thirty four down to like, 00:39:28.170 --> 00:39:29.590 what, twenty two, twenty three. 00:39:29.610 --> 00:39:30.231 Yeah. 00:39:30.291 --> 00:39:31.411 And it hasn't, you know, 00:39:31.731 --> 00:39:32.992 it really has not recovered 00:39:33.012 --> 00:39:35.853 from that for months. 00:39:35.873 --> 00:39:35.913 So. 00:39:36.233 --> 00:39:36.552 No. 00:39:36.652 --> 00:39:38.753 Yeah. 00:39:38.813 --> 00:39:39.152 And Bitcoin. 00:39:39.213 --> 00:39:40.173 Interestingly, though, 00:39:40.233 --> 00:39:41.914 like it didn't necessarily drag. 00:39:43.248 --> 00:39:44.088 A lot of, well, 00:39:44.128 --> 00:39:46.009 particularly the tokens that we monitor, 00:39:46.090 --> 00:39:47.949 that we're interested in. 00:39:48.271 --> 00:39:48.911 I don't think it really 00:39:48.971 --> 00:39:50.672 drags them down that much. 00:39:51.672 --> 00:39:52.492 It did initially. 00:39:52.793 --> 00:39:53.713 A lot of those other tokens 00:39:53.733 --> 00:39:55.014 seem to be recovered to 00:39:55.054 --> 00:39:56.355 around where they were when 00:39:57.034 --> 00:39:59.335 Ethereum was tracking like thirty three, 00:39:59.356 --> 00:40:00.237 thirty four hundred. 00:40:01.797 --> 00:40:02.398 But, you know, 00:40:02.677 --> 00:40:04.159 that's not to say that like 00:40:04.739 --> 00:40:06.280 if there is some decent 00:40:06.320 --> 00:40:07.219 price action in the 00:40:07.260 --> 00:40:09.942 Ethereum realm is likely drag up. 00:40:10.601 --> 00:40:11.822 the other shit coins that we 00:40:12.121 --> 00:40:15.463 watch so right um which is 00:40:15.503 --> 00:40:17.483 what you want the other 00:40:17.643 --> 00:40:19.983 shit coins yeah well you 00:40:20.023 --> 00:40:21.403 know there's bitcoin and 00:40:21.463 --> 00:40:22.445 ethereum and then there's 00:40:22.465 --> 00:40:23.405 shit coins right there's 00:40:23.445 --> 00:40:24.885 everything else yeah no I 00:40:24.925 --> 00:40:28.166 think that it I mean for 00:40:28.186 --> 00:40:30.846 the most well and then 00:40:30.965 --> 00:40:31.885 everything that rama 00:40:31.925 --> 00:40:33.507 watches is ultrasound shit 00:40:33.806 --> 00:40:36.527 coin things are moving like 00:40:36.547 --> 00:40:37.487 a hundred thousand percent 00:40:37.527 --> 00:40:38.987 a day and then plus or minus 00:40:40.253 --> 00:40:41.255 I mean, for the most part, 00:40:42.456 --> 00:40:44.016 there hasn't been a ton of 00:40:45.657 --> 00:40:46.958 correlation in the stuff 00:40:46.978 --> 00:40:47.619 that I'm watching. 00:40:47.980 --> 00:40:49.081 Like, I have everything from, like, 00:40:49.340 --> 00:40:50.581 Ethereum this week down 00:40:50.742 --> 00:40:51.583 four and a half percent. 00:40:52.463 --> 00:40:53.943 Helium up sixteen percent. 00:40:54.925 --> 00:40:56.266 Are you still watching Helium? 00:40:56.425 --> 00:40:57.086 I still watch Helium. 00:40:57.206 --> 00:40:57.847 I still have stuff. 00:40:59.688 --> 00:41:00.509 I feel like you've just had 00:41:00.528 --> 00:41:01.889 a massive bag of Helium 00:41:01.909 --> 00:41:03.271 you've been waiting to dump for, like, 00:41:03.391 --> 00:41:03.931 five years. 00:41:04.650 --> 00:41:07.530 uh not massive but but we we 00:41:07.811 --> 00:41:08.771 when we first got into that 00:41:08.911 --> 00:41:10.771 like I we I bought enough 00:41:10.831 --> 00:41:12.012 in to get a validator which 00:41:12.032 --> 00:41:13.313 was ten thousand hnt at the 00:41:13.333 --> 00:41:14.914 time which was like which 00:41:14.934 --> 00:41:17.155 was literally such a stupid 00:41:17.195 --> 00:41:19.655 move like sounds like 00:41:20.175 --> 00:41:21.996 looking back oracle moves 00:41:22.277 --> 00:41:23.456 it was it was such a twenty 00:41:23.496 --> 00:41:24.797 twenty move right like it 00:41:24.818 --> 00:41:27.059 was so fucking twenty twenty um 00:41:27.829 --> 00:41:29.010 Yeah, so fucking twenty twenty. 00:41:29.572 --> 00:41:31.994 And you like a full and a full at fucking. 00:41:32.014 --> 00:41:33.355 It's I mean, honestly, 00:41:33.916 --> 00:41:35.117 even now it's up one hundred percent. 00:41:35.657 --> 00:41:37.699 So I can't you can't complain too much. 00:41:37.778 --> 00:41:37.940 Right. 00:41:37.980 --> 00:41:40.041 Like like even now and it's 00:41:40.081 --> 00:41:40.961 only three years later, 00:41:40.981 --> 00:41:41.682 I guess three and a half 00:41:41.702 --> 00:41:43.965 years later or something like that. 00:41:43.985 --> 00:41:44.806 But I still watch it because 00:41:44.826 --> 00:41:45.226 I just want it. 00:41:45.286 --> 00:41:47.168 But but the the unbonding there, 00:41:47.288 --> 00:41:48.829 my unbond period, dude, is six months. 00:41:50.338 --> 00:41:53.101 So if I unbond now, it's whatever it is, 00:41:53.340 --> 00:41:53.981 fucking April. 00:41:54.641 --> 00:41:56.222 So right now, 00:41:56.302 --> 00:41:58.264 I'm just going to... That was 00:41:58.284 --> 00:41:59.123 the going in when they made 00:41:59.143 --> 00:42:03.007 the change from HNT to a token on Solana. 00:42:04.608 --> 00:42:05.887 It was always six months, actually. 00:42:05.967 --> 00:42:08.050 I think it was six months or a year. 00:42:08.170 --> 00:42:08.849 I forget what the original 00:42:08.869 --> 00:42:09.670 unbonding period was. 00:42:09.811 --> 00:42:10.931 It was six months, I think it was. 00:42:10.951 --> 00:42:12.072 It was automatically six months. 00:42:12.793 --> 00:42:14.893 And then once you transferred, 00:42:14.914 --> 00:42:16.155 you can make the decision to say, 00:42:16.755 --> 00:42:18.456 I can go for six months or 00:42:18.496 --> 00:42:19.757 a year or two years. 00:42:20.748 --> 00:42:23.809 And then at the end of that period, 00:42:24.268 --> 00:42:25.969 then the timer starts and 00:42:25.989 --> 00:42:27.050 then it's another six months. 00:42:28.389 --> 00:42:30.411 So I'm at the lowest rate, 00:42:30.431 --> 00:42:32.132 which means I earned the least, 00:42:32.572 --> 00:42:33.751 but if I unbond today, 00:42:33.831 --> 00:42:34.692 it's ready in April. 00:42:35.112 --> 00:42:35.992 And other people might say, 00:42:37.264 --> 00:42:38.244 I'm locking in for a minimum 00:42:38.264 --> 00:42:39.666 of two or three years or more. 00:42:39.686 --> 00:42:40.385 I think it was even a longer, 00:42:40.405 --> 00:42:41.346 it's like wind, right? 00:42:41.365 --> 00:42:42.166 Wasn't there like a ten-year 00:42:42.206 --> 00:42:42.706 option in wind? 00:42:42.766 --> 00:42:43.565 I think I still have that going. 00:42:44.226 --> 00:42:44.746 Oh, yeah. 00:42:44.887 --> 00:42:46.487 I'm fucking deep into that. 00:42:46.527 --> 00:42:46.927 And then, 00:42:46.947 --> 00:42:48.086 so I think you could do like two 00:42:48.106 --> 00:42:49.387 years and then at the end of two years, 00:42:49.547 --> 00:42:50.307 then you can make a 00:42:50.327 --> 00:42:51.387 decision to either re-up 00:42:51.427 --> 00:42:53.708 that or you can start a six-month unbond. 00:42:54.188 --> 00:42:57.668 So there's, yeah, it's pretty crazy. 00:42:58.030 --> 00:43:00.070 So it takes a while. 00:43:00.110 --> 00:43:00.750 So I do watch it. 00:43:00.949 --> 00:43:01.230 I do watch, 00:43:01.269 --> 00:43:02.010 even though it doesn't matter 00:43:02.090 --> 00:43:03.610 because I couldn't do anything anyway. 00:43:03.630 --> 00:43:04.291 It doesn't make a difference. 00:43:06.275 --> 00:43:07.817 And Aptos just changed their 00:43:07.876 --> 00:43:08.478 unbinding period too. 00:43:08.498 --> 00:43:09.219 They used to be a month. 00:43:09.239 --> 00:43:10.139 I know they just made it two weeks. 00:43:11.481 --> 00:43:12.161 They changed theirs. 00:43:12.242 --> 00:43:12.521 Yeah, 00:43:12.541 --> 00:43:13.744 they changed the unlock period to two 00:43:13.764 --> 00:43:14.503 weeks and they changed the 00:43:14.625 --> 00:43:15.264 unbinding period. 00:43:15.364 --> 00:43:16.106 I guess it's in line, 00:43:16.126 --> 00:43:17.347 but the unbinding period is 00:43:17.527 --> 00:43:17.927 now two weeks. 00:43:18.148 --> 00:43:20.550 Are we trying to fucking encourage? 00:43:20.570 --> 00:43:21.612 I don't know. 00:43:22.092 --> 00:43:23.574 Reduce the risk, I suppose. 00:43:24.434 --> 00:43:26.257 I think it was trying to increase staking. 00:43:26.952 --> 00:43:27.552 Yeah, exactly, 00:43:27.592 --> 00:43:29.253 by reducing the risk of staking. 00:43:29.534 --> 00:43:29.693 Yeah, 00:43:30.014 --> 00:43:32.835 a one-month unbinding was just too long, 00:43:32.936 --> 00:43:34.135 right? 00:43:34.175 --> 00:43:36.478 So I think that really 00:43:36.538 --> 00:43:37.778 pushed them to do more. 00:43:40.300 --> 00:43:42.360 Yeah, it's probably unnecessary as well. 00:43:47.344 --> 00:43:48.625 If there's a mass unbinding, 00:43:48.664 --> 00:43:50.146 if it's two weeks or a month, 00:43:50.166 --> 00:43:50.726 what's the difference? 00:43:52.327 --> 00:43:53.268 I think it probably makes a difference. 00:43:53.427 --> 00:43:54.389 I don't know. 00:43:54.449 --> 00:43:54.989 Actually, I don't know. 00:43:55.248 --> 00:43:56.050 I don't know if it does or not. 00:43:56.876 --> 00:44:00.018 A month is... Well, actually, 00:44:00.179 --> 00:44:01.099 there it's a little bit different. 00:44:02.139 --> 00:44:03.280 There's an unlocked date. 00:44:03.619 --> 00:44:04.920 So there's an unlocked date 00:44:04.940 --> 00:44:06.621 and time that exists. 00:44:06.902 --> 00:44:10.304 And then it would be exactly 00:44:10.324 --> 00:44:11.204 a month from that point. 00:44:11.364 --> 00:44:13.945 So I think you can... 00:44:15.871 --> 00:44:17.210 I think if I, I don't even, 00:44:17.331 --> 00:44:18.152 I'm not a hundred percent sure, 00:44:18.172 --> 00:44:19.032 but I think if I, if it, 00:44:19.371 --> 00:44:20.572 if my date was next week, I could, 00:44:20.913 --> 00:44:21.992 I could choose to unbond now. 00:44:22.092 --> 00:44:23.673 And then the month starts at that date. 00:44:23.833 --> 00:44:24.353 You know what I mean? 00:44:24.373 --> 00:44:25.673 Like it would start then at that point, 00:44:26.255 --> 00:44:26.775 something like that, 00:44:26.815 --> 00:44:27.655 but it doesn't matter. 00:44:29.695 --> 00:44:30.856 So in other news, 00:44:31.175 --> 00:44:32.976 balls deep in Proxmox at the moment. 00:44:34.637 --> 00:44:38.081 Fucking just waving my dick 00:44:38.101 --> 00:44:38.963 around in that pool. 00:44:39.143 --> 00:44:39.784 Fucking hell. 00:44:41.065 --> 00:44:45.753 So the new rack, new servers, 00:44:46.434 --> 00:44:47.074 new problems. 00:44:49.478 --> 00:44:51.320 Validating out of Australia. 00:44:52.735 --> 00:44:54.375 you have to make a decision 00:44:54.394 --> 00:44:55.516 when you do this shit, don't you? 00:44:55.536 --> 00:44:56.956 Like, what are you going to use? 00:44:57.576 --> 00:45:00.436 And then, you know, there's XCPNG, 00:45:00.577 --> 00:45:02.697 there's Proxmox, vSphere, fucking, 00:45:02.958 --> 00:45:04.858 you know, if you're an idiot, 00:45:04.878 --> 00:45:05.557 there's other stuff. 00:45:06.217 --> 00:45:06.858 I was going to say, 00:45:06.878 --> 00:45:10.099 I was going to name shit, but, you know, 00:45:10.139 --> 00:45:11.380 that's just me being a dick. 00:45:12.840 --> 00:45:17.420 So, so much nuance in all of these things. 00:45:18.061 --> 00:45:20.081 There's like your hardware stuff, 00:45:20.887 --> 00:45:22.469 You know, your compute hardware, 00:45:22.949 --> 00:45:25.789 what decisions you make on like RAM, CPU. 00:45:26.230 --> 00:45:28.130 Then you've got like networking. 00:45:28.230 --> 00:45:29.670 Then you've got storage. 00:45:29.931 --> 00:45:31.592 Then you've got like your 00:45:32.592 --> 00:45:33.572 storage software. 00:45:33.632 --> 00:45:34.492 Like which way are you going 00:45:34.512 --> 00:45:35.092 to go with that? 00:45:35.652 --> 00:45:37.072 So many different decisions, man. 00:45:37.833 --> 00:45:39.753 And we've been like working 00:45:39.833 --> 00:45:41.235 our way through all of this shit. 00:45:42.498 --> 00:45:44.338 And I think we're almost there. 00:45:44.557 --> 00:45:46.199 It's starting to look good. 00:45:46.219 --> 00:45:47.599 We're getting really good 00:45:47.639 --> 00:45:49.018 performance out of man. 00:45:49.099 --> 00:45:51.940 Ceph is just some shit. 00:45:52.659 --> 00:45:54.880 It's I don't really, I mean, 00:45:54.940 --> 00:45:55.900 I ran it for a little bit. 00:45:56.581 --> 00:45:57.661 Uh, it was a while ago though. 00:45:58.260 --> 00:45:59.822 Um, I forgot most of it. 00:46:00.262 --> 00:46:06.862 I remember it being a bit, um, not shady. 00:46:07.103 --> 00:46:07.784 I remember being like, 00:46:07.903 --> 00:46:09.143 how the fuck is this actually working? 00:46:09.969 --> 00:46:10.268 It is, 00:46:10.389 --> 00:46:11.510 it's just the black box of 00:46:11.530 --> 00:46:12.351 distributed storage. 00:46:12.371 --> 00:46:13.692 You just add the drives to 00:46:13.711 --> 00:46:14.331 it and you're like, all right, 00:46:14.351 --> 00:46:14.771 do your thing. 00:46:14.871 --> 00:46:16.313 All right. 00:46:16.333 --> 00:46:17.653 I have a YAML file that says 00:46:17.673 --> 00:46:18.215 you're going to use it. 00:46:18.474 --> 00:46:18.795 Um, 00:46:18.875 --> 00:46:19.956 I remember being a little bit like that, 00:46:20.335 --> 00:46:21.496 but I mean, be sure it's the same way. 00:46:21.556 --> 00:46:22.117 Like you don't understand 00:46:22.137 --> 00:46:23.338 how the fucking data stores work. 00:46:24.099 --> 00:46:24.219 Um, 00:46:25.358 --> 00:46:26.840 Yeah, I remember that. 00:46:26.900 --> 00:46:28.740 It's like a Megatron of storage, you know, 00:46:28.760 --> 00:46:29.820 like all that shit is just 00:46:29.860 --> 00:46:30.961 sort of all mashed together 00:46:31.001 --> 00:46:31.862 and then it's like awesome 00:46:31.882 --> 00:46:32.523 fucking Megatron. 00:46:32.563 --> 00:46:33.503 You just throw it into the bin, 00:46:33.543 --> 00:46:34.043 like throw it over the 00:46:34.083 --> 00:46:35.563 fence and hopefully at some 00:46:35.583 --> 00:46:37.264 point it'll be there when you want it. 00:46:37.344 --> 00:46:37.684 Yeah. 00:46:37.704 --> 00:46:39.385 So like people do just add 00:46:39.445 --> 00:46:42.527 fucking like spinners and 00:46:42.807 --> 00:46:44.789 SSD and whatever trash 00:46:44.809 --> 00:46:46.088 they've got lying around in 00:46:46.128 --> 00:46:47.269 the Ceph clusters and it 00:46:47.289 --> 00:46:48.851 just creates this mishmash 00:46:48.911 --> 00:46:50.831 Megatron of bullshit fucking storage, 00:46:50.851 --> 00:46:50.972 right? 00:46:51.072 --> 00:46:51.192 Yeah. 00:46:51.831 --> 00:46:53.333 which is somehow still performant. 00:46:53.393 --> 00:46:54.574 Like it just figures it all out. 00:46:54.594 --> 00:46:56.016 It's like, Oh, this is slow. 00:46:56.697 --> 00:46:57.538 I'll stick some shit. 00:46:57.557 --> 00:46:59.659 That's not being used much over there. 00:46:59.699 --> 00:47:00.681 And then like, you know, 00:47:00.701 --> 00:47:02.884 this is needs to be fast over here. 00:47:03.403 --> 00:47:03.684 Crazy. 00:47:03.784 --> 00:47:06.327 And it's just, just some crazy shit. 00:47:06.688 --> 00:47:07.208 But, um, 00:47:08.188 --> 00:47:09.449 you know we don't we don't 00:47:09.489 --> 00:47:10.550 do that over here at king 00:47:10.750 --> 00:47:13.670 king knows we got we got 00:47:13.690 --> 00:47:15.471 all gen five storage baby 00:47:15.570 --> 00:47:17.371 we're getting like it's 00:47:17.431 --> 00:47:18.552 network bound at the moment 00:47:18.592 --> 00:47:19.751 we're you know the read 00:47:19.791 --> 00:47:21.932 writes are three gigabytes 00:47:21.952 --> 00:47:22.972 per second which is about 00:47:23.092 --> 00:47:24.353 the limit of my twenty five 00:47:24.393 --> 00:47:26.853 gig network so upping that 00:47:26.873 --> 00:47:29.014 to fifty gig today my my 00:47:29.054 --> 00:47:30.614 new cables get here really 00:47:31.353 --> 00:47:35.523 Yeah, so we're ganging the NICs. 00:47:36.164 --> 00:47:38.210 So we have a gig for the 00:47:38.250 --> 00:47:40.856 storage network and then a gig for the 00:47:42.170 --> 00:47:43.170 the management network. 00:47:43.190 --> 00:47:43.789 Cause we're actually, 00:47:43.829 --> 00:47:45.690 when you migrate nodes around, 00:47:45.971 --> 00:47:47.150 they use the management network. 00:47:47.190 --> 00:47:49.052 You can set it to use different networks, 00:47:49.112 --> 00:47:50.711 but I thought, fuck it. 00:47:50.731 --> 00:47:53.012 We'll just, we'll bond, um, two, 00:47:53.052 --> 00:47:54.432 ten gigs for the management 00:47:54.472 --> 00:47:55.934 network so that it's got a 00:47:55.994 --> 00:47:58.414 redundancy and be more, more bandwidth to, 00:47:58.534 --> 00:47:59.994 um, but, but it's actually, 00:48:00.074 --> 00:48:00.934 it's still live like during 00:48:00.954 --> 00:48:01.635 that migration, right? 00:48:03.096 --> 00:48:04.416 So live migration. 00:48:04.856 --> 00:48:10.958 So I, I did, um, so I did a test and, um, 00:48:12.275 --> 00:48:13.277 on the gig network, 00:48:13.757 --> 00:48:15.257 the thing is it has to transfer the Ram, 00:48:15.277 --> 00:48:15.396 right? 00:48:16.498 --> 00:48:17.418 So it has to snapshot the 00:48:17.458 --> 00:48:18.438 Ram and then transfer it. 00:48:20.639 --> 00:48:21.760 So obviously the more Ram you have, 00:48:21.780 --> 00:48:22.760 the longer that takes in on 00:48:22.800 --> 00:48:24.141 a gig network that can take 00:48:24.161 --> 00:48:24.661 a bit of time. 00:48:24.702 --> 00:48:25.842 If you've got like, you know, 00:48:25.922 --> 00:48:26.563 half a gig of Ram, 00:48:26.722 --> 00:48:27.822 a half terabyte of Ram or something, 00:48:27.842 --> 00:48:28.844 you're going to be waiting all day. 00:48:29.724 --> 00:48:30.744 But for a smaller nodes, 00:48:30.824 --> 00:48:32.505 it'll snapshot it and then 00:48:33.726 --> 00:48:35.045 send it across and then 00:48:35.106 --> 00:48:35.806 start it up again. 00:48:35.847 --> 00:48:36.027 Right. 00:48:36.646 --> 00:48:38.248 So I think when I was just, I was, 00:48:38.387 --> 00:48:39.648 I was pushing around a, 00:48:40.768 --> 00:48:42.909 you know, at VM to just test it. 00:48:43.650 --> 00:48:44.951 And it was having fourteen 00:48:45.070 --> 00:48:50.333 nanoseconds of actual downtime when it, 00:48:50.873 --> 00:48:52.853 like, did the switch. 00:48:52.994 --> 00:48:54.333 So, yeah. 00:48:55.355 --> 00:48:56.275 It was taking, like, 00:48:58.396 --> 00:49:00.476 not very long to transfer. 00:49:00.697 --> 00:49:01.456 It wasn't a lot of RAM. 00:49:01.476 --> 00:49:01.817 It was, like, 00:49:01.856 --> 00:49:03.557 eight gig of RAM or something 00:49:03.597 --> 00:49:04.297 just to test it. 00:49:05.398 --> 00:49:06.438 I think it was transferring 00:49:06.458 --> 00:49:06.818 and then just... 00:49:07.733 --> 00:49:08.954 That might be similar to vSphere. 00:49:08.974 --> 00:49:09.695 I've never really noticed 00:49:09.735 --> 00:49:10.876 any sort of... The node's 00:49:10.896 --> 00:49:12.297 always active in those periods. 00:49:12.356 --> 00:49:13.018 I never found anything. 00:49:13.038 --> 00:49:13.117 Yeah, 00:49:13.358 --> 00:49:14.998 like if you have a ping screen up or 00:49:15.018 --> 00:49:16.460 something, you don't see it still. 00:49:16.539 --> 00:49:17.280 Yeah, you wouldn't see it. 00:49:17.840 --> 00:49:19.141 But even I've had running 00:49:19.161 --> 00:49:20.322 blockchain nodes and other 00:49:20.342 --> 00:49:20.762 types of things, 00:49:20.802 --> 00:49:21.684 I never noticed missing a 00:49:21.724 --> 00:49:22.423 block or anything else 00:49:22.463 --> 00:49:22.945 during that period. 00:49:24.197 --> 00:49:27.159 It is pretty impressive, I think. 00:49:28.561 --> 00:49:28.860 That's cool. 00:49:28.900 --> 00:49:29.201 That's good. 00:49:29.461 --> 00:49:30.061 The only thing, I mean, 00:49:30.081 --> 00:49:32.423 you're kind of building a SAN with Ceph. 00:49:32.503 --> 00:49:33.965 And so the largest outages 00:49:34.125 --> 00:49:35.326 in network and service 00:49:35.365 --> 00:49:36.766 history have always been SAN related. 00:49:38.389 --> 00:49:40.170 So not to freak you out, 00:49:40.230 --> 00:49:41.170 but like once that thing 00:49:41.231 --> 00:49:42.311 dies or like something 00:49:42.351 --> 00:49:44.092 happens and everything 00:49:44.112 --> 00:49:45.494 that's associated to that 00:49:45.514 --> 00:49:47.215 cannot be started and 00:49:47.235 --> 00:49:48.016 you're trying to figure out 00:49:48.077 --> 00:49:48.777 how the fuck am I going to 00:49:48.797 --> 00:49:49.597 rebuild this shit? 00:49:50.585 --> 00:49:51.987 well it's it's distributed 00:49:52.047 --> 00:49:54.887 sand so it's like you know 00:49:54.907 --> 00:49:55.867 you've got redundancy 00:49:55.887 --> 00:49:57.349 across hardware nodes as 00:49:57.369 --> 00:49:59.048 well as like all the yeah 00:49:59.289 --> 00:50:00.389 all the yeah of course yeah 00:50:00.409 --> 00:50:03.971 yeah yeah but um I mean you 00:50:03.990 --> 00:50:05.090 can you can lose a fair bit 00:50:05.150 --> 00:50:06.831 of stuff before it'll stop 00:50:06.871 --> 00:50:08.353 working yeah you should 00:50:08.373 --> 00:50:09.413 pull some cables see what happens 00:50:10.717 --> 00:50:11.416 Do what? 00:50:11.478 --> 00:50:12.097 You should pull some cables. 00:50:12.117 --> 00:50:12.757 Oh, pull cables? 00:50:12.978 --> 00:50:13.518 Yeah. 00:50:13.599 --> 00:50:14.278 I've pulled nodes. 00:50:14.800 --> 00:50:15.380 You have? 00:50:15.440 --> 00:50:16.460 I've turned off nodes to see 00:50:16.481 --> 00:50:17.360 if it still keeps working. 00:50:17.521 --> 00:50:18.021 Okay, good. 00:50:18.041 --> 00:50:18.782 It doesn't miss a beat. 00:50:19.161 --> 00:50:21.324 Like, kill some cables, pull a drive out. 00:50:21.503 --> 00:50:22.543 Well, you can't pull a drive out. 00:50:22.644 --> 00:50:23.545 I guess you can't pull a drive. 00:50:23.565 --> 00:50:25.786 Those are all run-of-the-box 00:50:25.827 --> 00:50:28.047 type of things, right? 00:50:28.068 --> 00:50:28.148 Yeah. 00:50:28.168 --> 00:50:28.887 Pulling drives out? 00:50:28.927 --> 00:50:29.548 Yeah, it's fine. 00:50:29.568 --> 00:50:29.648 Yeah. 00:50:29.688 --> 00:50:32.030 Just pop a couple out. 00:50:32.630 --> 00:50:32.731 Yeah, 00:50:32.751 --> 00:50:34.052 you can pull three drives across 00:50:34.132 --> 00:50:35.672 the... So with mine, 00:50:35.693 --> 00:50:36.492 you can pull out three 00:50:36.532 --> 00:50:38.393 drives across the cluster. 00:50:38.855 --> 00:50:39.135 Nice. 00:50:39.869 --> 00:50:40.070 Yeah. 00:50:40.329 --> 00:50:42.791 Or all of one and then some of the others. 00:50:42.831 --> 00:50:45.891 And you can definitely like 00:50:45.911 --> 00:50:47.092 hot swap networks and stuff. 00:50:47.152 --> 00:50:49.532 It doesn't affect it. 00:50:49.572 --> 00:50:51.793 Well, once we get the LACP going anyway. 00:50:53.934 --> 00:50:54.454 That's cool. 00:50:55.034 --> 00:50:56.855 Yeah. 00:50:57.014 --> 00:50:57.934 The more you think about it 00:50:57.954 --> 00:50:58.635 and the more you're like, oh, 00:50:58.655 --> 00:50:59.795 I want some redundancy there. 00:50:59.856 --> 00:51:00.175 Fuck. 00:51:00.275 --> 00:51:01.376 It just keeps adding and 00:51:01.456 --> 00:51:05.376 adding and adding costs and switches. 00:51:05.637 --> 00:51:06.737 I'm like, 00:51:06.797 --> 00:51:07.838 when I first started running 00:51:07.878 --> 00:51:08.458 stuff out of, 00:51:09.059 --> 00:51:10.719 you know, uh, data center, 00:51:11.260 --> 00:51:15.003 I have like this, uh, it's, you know, 00:51:15.264 --> 00:51:17.806 like, uh, it's got two and a half gig, um, 00:51:17.945 --> 00:51:20.188 uplink and then like a gig ports. 00:51:20.228 --> 00:51:22.009 And then that was my, my router, right. 00:51:22.289 --> 00:51:22.849 And micro tick. 00:51:23.927 --> 00:51:25.228 I'll be five Oh nine, 00:51:25.748 --> 00:51:27.190 no hardware upload or anything like that. 00:51:27.510 --> 00:51:30.291 And, um, I mean, it's served fine for, 00:51:30.331 --> 00:51:31.472 for quite a while with what 00:51:31.492 --> 00:51:32.054 we were running on it. 00:51:32.893 --> 00:51:34.896 And then moving into like 00:51:35.556 --> 00:51:38.577 now with our own racks, own internet, 00:51:38.657 --> 00:51:39.398 that type of stuff. 00:51:40.278 --> 00:51:41.780 And then, uh, you know, 00:51:41.940 --> 00:51:43.081 your redundancy all the way 00:51:43.121 --> 00:51:43.822 down the chain. 00:51:44.021 --> 00:51:46.202 And then, so you like two routers, two, 00:51:46.643 --> 00:51:48.103 two high bandwidth switches, 00:51:48.164 --> 00:51:49.304 two low bandwidth switches, 00:51:49.344 --> 00:51:50.746 two fucking this, two fucking that. 00:51:51.387 --> 00:51:52.407 And then like this cable's 00:51:52.447 --> 00:51:53.027 going everywhere. 00:51:53.047 --> 00:51:53.128 Yeah. 00:51:53.777 --> 00:51:54.358 Yeah. 00:51:54.378 --> 00:51:57.000 I mean, even, I mean, we had, yeah, 00:51:57.079 --> 00:51:58.079 even have it like we had 00:51:58.099 --> 00:51:59.081 Cisco switch Brown out 00:51:59.101 --> 00:51:59.822 wherever the hell that was 00:51:59.842 --> 00:52:00.402 a couple weeks ago. 00:52:00.422 --> 00:52:03.764 Like you start to realize, and then that, 00:52:04.184 --> 00:52:04.945 like those types of 00:52:04.965 --> 00:52:06.086 situations and like how 00:52:06.226 --> 00:52:07.307 then I need to imagine how 00:52:07.347 --> 00:52:08.367 am I going to reroute and like, 00:52:08.427 --> 00:52:09.489 what is dependent on that? 00:52:09.528 --> 00:52:12.251 And like there it is when 00:52:12.271 --> 00:52:12.871 you're starting to build 00:52:12.891 --> 00:52:14.012 your own data center and it 00:52:14.032 --> 00:52:14.532 depends on like, 00:52:14.632 --> 00:52:15.934 like some people take that seriously. 00:52:15.954 --> 00:52:16.514 Some people don't. 00:52:17.034 --> 00:52:19.056 Um, and, and. 00:52:20.199 --> 00:52:20.418 There's, 00:52:20.679 --> 00:52:22.300 there's so many different layers of, 00:52:22.820 --> 00:52:25.340 of that type of high availability, right? 00:52:25.360 --> 00:52:28.380 Like you can, like, like example, you can, 00:52:28.820 --> 00:52:29.260 you can do rate. 00:52:29.340 --> 00:52:30.621 If you're going to do HA, 00:52:30.681 --> 00:52:31.922 you have to do HA through 00:52:31.942 --> 00:52:33.262 the entire fucking chain, 00:52:33.541 --> 00:52:35.103 starting with your data center, 00:52:35.143 --> 00:52:36.483 which is really expensive. 00:52:37.302 --> 00:52:39.903 And, and it is so fucking like, 00:52:40.423 --> 00:52:41.304 not only is it expensive, 00:52:41.364 --> 00:52:43.005 but it gets complicated and 00:52:43.025 --> 00:52:43.945 it gets complicated to the 00:52:44.005 --> 00:52:45.724 point where small issues 00:52:45.764 --> 00:52:46.525 that wouldn't affect a 00:52:46.565 --> 00:52:48.126 regular node operator affect you. 00:52:48.826 --> 00:52:50.746 Because it's an HA situation. 00:52:50.766 --> 00:52:51.786 You go, oh, fuck. 00:52:52.168 --> 00:52:52.268 Right. 00:52:52.288 --> 00:52:53.228 Because you're worried about 00:52:53.327 --> 00:52:54.309 you're building around 00:52:54.349 --> 00:52:55.909 these failures and you 00:52:55.929 --> 00:52:56.789 might have situations that 00:52:56.809 --> 00:52:59.251 are totally benign for anyone else. 00:52:59.271 --> 00:53:00.570 But for you are a big issue 00:53:00.590 --> 00:53:02.532 because in an HA situation, 00:53:02.952 --> 00:53:04.432 it turns into a larger 00:53:05.094 --> 00:53:06.594 problem because it's not 00:53:06.614 --> 00:53:07.375 like a simple change 00:53:07.414 --> 00:53:08.414 because it requires changes 00:53:08.494 --> 00:53:09.936 all the way down that stack type of idea. 00:53:10.835 --> 00:53:11.094 Um, 00:53:11.135 --> 00:53:12.235 like we've had the issue with PDU 00:53:12.275 --> 00:53:13.094 monitoring and a bunch of 00:53:13.114 --> 00:53:13.815 other types of things. 00:53:13.856 --> 00:53:16.315 And well, in our, in our DCs, we actually, 00:53:16.416 --> 00:53:17.695 well, one of the new DCs that we run, 00:53:17.715 --> 00:53:18.516 but one of them has to have 00:53:18.815 --> 00:53:19.597 battery backup units. 00:53:20.396 --> 00:53:21.536 Um, and so even with that, 00:53:21.677 --> 00:53:24.317 like even a BBU failing, um, 00:53:24.556 --> 00:53:25.737 can cause like all sorts of 00:53:25.797 --> 00:53:26.597 fucking problems. 00:53:28.137 --> 00:53:31.199 Like, cause not everything has, well, 00:53:31.259 --> 00:53:31.978 I think everything we have 00:53:31.998 --> 00:53:33.478 in that rack has multiple PSS, 00:53:33.539 --> 00:53:35.639 but sometimes it can be, 00:53:35.719 --> 00:53:36.579 it can be a bit of a challenge. 00:53:37.420 --> 00:53:38.420 Thank God the, uh, 00:53:40.306 --> 00:53:42.288 like all of the power backup 00:53:42.347 --> 00:53:44.148 is managed by the DC where we are. 00:53:44.329 --> 00:53:46.730 So we're like, we're two N plus one. 00:53:47.070 --> 00:53:50.072 So N plus backup, I guess. 00:53:50.452 --> 00:53:50.672 Yeah. 00:53:50.751 --> 00:53:51.452 Right. 00:53:51.913 --> 00:53:52.193 Yeah. 00:53:52.253 --> 00:53:53.213 So it's like, it's fucking, 00:53:53.273 --> 00:53:55.373 it's very redundant, but you know, 00:53:55.393 --> 00:53:57.295 there's multiple inlinks of 00:53:57.434 --> 00:53:58.416 dark fiber as well. 00:53:58.436 --> 00:53:59.815 Like all the carriers have 00:53:59.916 --> 00:54:03.338 multiple fiber inlets. 00:54:03.438 --> 00:54:04.677 So if someone on, you know, 00:54:04.697 --> 00:54:05.918 the South side of Brisbane 00:54:05.978 --> 00:54:07.460 digs through a cable, he's still good. 00:54:08.179 --> 00:54:10.402 yeah um yeah multiple inlets 00:54:10.442 --> 00:54:12.704 of power multiple backups 00:54:13.204 --> 00:54:14.005 they got batteries and 00:54:14.065 --> 00:54:15.967 generators they got fucking 00:54:16.007 --> 00:54:18.489 sure even they've even got ab on the um 00:54:19.684 --> 00:54:21.065 So when you do structured 00:54:21.106 --> 00:54:22.365 cabling and cross connects, right, 00:54:23.126 --> 00:54:24.867 that you can, they've got two. 00:54:27.467 --> 00:54:29.509 So one side of the room has 00:54:29.708 --> 00:54:30.789 structured cabling coming 00:54:30.869 --> 00:54:31.730 in and then the other side 00:54:31.750 --> 00:54:32.710 of the room has structured 00:54:32.750 --> 00:54:34.891 cabling coming in. 00:54:35.471 --> 00:54:37.032 And then you can get two 00:54:37.291 --> 00:54:39.552 sets of structured cabling to your rack. 00:54:40.092 --> 00:54:41.695 which go to two separate 00:54:43.556 --> 00:54:44.378 cross-connect rooms. 00:54:45.418 --> 00:54:46.079 So patch rooms? 00:54:46.219 --> 00:54:46.581 Yeah, sure. 00:54:46.681 --> 00:54:47.541 And then, yeah, 00:54:47.621 --> 00:54:48.623 and all the carriers come 00:54:48.682 --> 00:54:50.324 to two separate patch rooms. 00:54:50.505 --> 00:54:50.844 Right. 00:54:50.885 --> 00:54:52.427 So you can have like one 00:54:52.467 --> 00:54:54.489 connection from one carrier 00:54:54.688 --> 00:54:56.791 from one cross-connect and 00:54:56.811 --> 00:54:58.273 then another from another 00:54:58.293 --> 00:54:59.474 cross-connect that come in 00:55:00.278 --> 00:55:00.518 you know, 00:55:00.798 --> 00:55:02.219 into the data halls and do two 00:55:02.239 --> 00:55:02.940 different directions, 00:55:03.001 --> 00:55:04.922 but both have dark fibers 00:55:04.961 --> 00:55:05.862 come in from two different 00:55:05.882 --> 00:55:06.563 directions as well. 00:55:06.603 --> 00:55:07.222 It's fucking insane. 00:55:07.262 --> 00:55:07.382 Yeah. 00:55:07.402 --> 00:55:07.483 Yeah. 00:55:08.423 --> 00:55:08.664 And, 00:55:10.405 --> 00:55:11.844 and this building is a bit messy though. 00:55:11.864 --> 00:55:12.945 It's a tier four data center, 00:55:13.025 --> 00:55:13.726 but the building is a 00:55:13.766 --> 00:55:14.686 little bit messy because it 00:55:14.706 --> 00:55:15.768 was an existing building. 00:55:16.188 --> 00:55:17.409 And it was like a fucking 00:55:17.469 --> 00:55:18.168 labyrinth when you're 00:55:18.188 --> 00:55:21.951 trying to get down to the, the, you know, 00:55:22.010 --> 00:55:23.552 like the dock and that type of shit. 00:55:24.411 --> 00:55:25.713 It's cool. 00:55:25.833 --> 00:55:25.932 Yeah. 00:55:25.992 --> 00:55:27.673 I got so lost in there. 00:55:27.733 --> 00:55:29.074 So can I throw some hardware in there? 00:55:31.244 --> 00:55:31.864 In my rank? 00:55:32.164 --> 00:55:32.384 Yeah. 00:55:33.284 --> 00:55:36.465 I don't even have power for me. 00:55:36.885 --> 00:55:38.206 I busted my power limit so 00:55:38.246 --> 00:55:39.686 hard when I put the 00:55:39.766 --> 00:55:42.407 lowest... Power is the 00:55:42.447 --> 00:55:43.568 most... Nobody gives a shit 00:55:43.588 --> 00:55:45.289 about networks or this or that or space. 00:55:45.329 --> 00:55:48.070 It's all about fucking wattage. 00:55:48.090 --> 00:55:48.371 You know, 00:55:48.471 --> 00:55:49.990 it's really weird at this data 00:55:50.010 --> 00:55:50.771 center too because... 00:55:52.815 --> 00:55:54.617 You know how normally when 00:55:54.637 --> 00:55:56.878 you scale things, prices go down? 00:55:57.559 --> 00:55:59.001 Well, here prices go up. 00:55:59.541 --> 00:56:01.842 So like if you add a kilowatt, 00:56:02.664 --> 00:56:03.844 every time you add a kilowatt, 00:56:04.364 --> 00:56:05.045 it keeps getting more 00:56:05.085 --> 00:56:06.186 expensive per kilowatt. 00:56:06.206 --> 00:56:07.188 You're like, wait a second. 00:56:07.940 --> 00:56:09.282 It's like your first bit of 00:56:09.302 --> 00:56:11.543 crack is two-fifty a kilowatt, 00:56:12.023 --> 00:56:13.164 and then your second bag of 00:56:13.204 --> 00:56:15.264 crack is fucking three-fifty a kilowatt. 00:56:15.746 --> 00:56:16.346 Right up to, 00:56:16.545 --> 00:56:18.226 I think it limits out at like 00:56:18.266 --> 00:56:19.268 five-fifty a kilowatt or 00:56:19.288 --> 00:56:20.387 something like that, and then that's it. 00:56:20.648 --> 00:56:24.731 It's like the inverse volume bargain, 00:56:24.931 --> 00:56:25.170 right? 00:56:25.791 --> 00:56:27.472 I can tell you it's not what I expected. 00:56:27.891 --> 00:56:29.853 Like when I said, oh, you know, what's... 00:56:32.594 --> 00:56:33.693 what does it cost per kilowatt? 00:56:33.713 --> 00:56:34.474 And he's like, Oh, you know, 00:56:34.494 --> 00:56:35.355 it's a sliding scale. 00:56:35.375 --> 00:56:36.375 And I'm like, cool. 00:56:36.635 --> 00:56:37.195 That sounds good. 00:56:37.255 --> 00:56:38.775 Thinking that it goes down and he's like, 00:56:38.835 --> 00:56:39.795 it starts at three fifty 00:56:40.916 --> 00:56:41.777 and it goes up from there. 00:56:41.856 --> 00:56:44.097 I'm like, wait a second. 00:56:44.177 --> 00:56:45.617 It goes up from there? 00:56:45.657 --> 00:56:46.378 I didn't lie. 00:56:46.538 --> 00:56:47.617 I didn't lie. 00:56:47.657 --> 00:56:48.018 No. 00:56:48.858 --> 00:56:49.518 And then like, 00:56:49.639 --> 00:56:50.699 I've got other friends here 00:56:50.719 --> 00:56:52.420 in Brisbane that seen my 00:56:52.440 --> 00:56:53.699 shit and they like want to 00:56:53.739 --> 00:56:55.099 play with some toys now and 00:56:55.119 --> 00:56:56.481 they're like researching 00:56:56.541 --> 00:56:57.481 data centers and stuff. 00:56:57.541 --> 00:56:57.641 Yeah. 00:56:57.681 --> 00:57:00.681 Fuck man, I can get like a whole rack for, 00:57:00.981 --> 00:57:01.242 uh, 00:57:01.851 --> 00:57:02.813 I don't know, like a thousand bucks. 00:57:02.833 --> 00:57:04.454 Like, you know, 00:57:04.494 --> 00:57:05.735 we run out of a half rack 00:57:05.795 --> 00:57:08.016 here in Brisbane and, you know, 00:57:08.077 --> 00:57:08.996 even that's expensive. 00:57:09.818 --> 00:57:10.398 And they're like, 00:57:11.139 --> 00:57:11.878 we can get a whole rack 00:57:11.918 --> 00:57:14.240 down at this other data center for like, 00:57:14.681 --> 00:57:14.942 you know, 00:57:14.981 --> 00:57:16.121 half the price you're paying for 00:57:16.161 --> 00:57:17.302 your full rack. 00:57:17.322 --> 00:57:21.485 And I'm like, okay, where is it? 00:57:21.505 --> 00:57:24.809 Or where is that fucking thing? 00:57:24.909 --> 00:57:26.309 It's like, I'm like, 00:57:26.789 --> 00:57:27.610 I'd rather pay more money. 00:57:27.630 --> 00:57:28.811 Yeah. 00:57:28.851 --> 00:57:29.931 Have it in the city right 00:57:29.972 --> 00:57:30.813 next to the cookie shop. 00:57:31.282 --> 00:57:33.143 I think we have, yeah, we have fifty amps, 00:57:33.264 --> 00:57:34.605 I think, in one, 00:57:34.644 --> 00:57:36.465 and then we have another ten in the other, 00:57:36.766 --> 00:57:37.427 or fifteen amps. 00:57:37.447 --> 00:57:38.547 I think we have like 00:57:38.588 --> 00:57:39.829 sixty-five or seventy amps total. 00:57:39.849 --> 00:57:41.789 That's a lot. 00:57:41.849 --> 00:57:42.429 I've got a couple of 00:57:42.449 --> 00:57:43.291 kilowatts at the moment, 00:57:43.331 --> 00:57:45.211 but we just keep bumping it 00:57:45.251 --> 00:57:45.632 up as we need. 00:57:45.652 --> 00:57:47.253 You keep bumping it up, right. 00:57:47.634 --> 00:57:49.534 Well, I mean, here, for us, it's never, 00:57:49.875 --> 00:57:50.856 but it's like, hey, 00:57:50.896 --> 00:57:51.836 you want a twenty amp line? 00:57:51.876 --> 00:57:52.916 You want another thirty amp line? 00:57:52.936 --> 00:57:53.858 You want another thirty amp line? 00:57:53.898 --> 00:57:55.978 I can't go from like thirty to thirty-one, 00:57:56.539 --> 00:57:57.739 because literally they drop 00:57:57.900 --> 00:57:59.621 a different fucking outlet. 00:58:00.342 --> 00:58:01.282 on a different breaker, 00:58:01.362 --> 00:58:02.625 on a different structure, on a different, 00:58:02.744 --> 00:58:03.525 like, you know what I mean? 00:58:03.846 --> 00:58:07.048 Like there is, it's like, okay, 00:58:07.108 --> 00:58:08.650 so we need a new PDU for that. 00:58:08.670 --> 00:58:09.891 Or I need to do something else with that. 00:58:09.931 --> 00:58:10.592 Or I can, you know, 00:58:10.612 --> 00:58:11.614 I can go from this thirty 00:58:11.653 --> 00:58:13.936 to a fifty and our PDUs are all fifties. 00:58:13.996 --> 00:58:16.719 But, but like, you know, and even PDUs, 00:58:16.739 --> 00:58:18.561 like we're buying those PDUs you got, 00:58:18.601 --> 00:58:18.760 right? 00:58:18.780 --> 00:58:19.501 Those were in the rack. 00:58:20.407 --> 00:58:21.027 In the rack, yeah. 00:58:21.106 --> 00:58:22.188 Oh, yeah, we buy dollars. 00:58:22.847 --> 00:58:24.548 So you pay a setup fee here? 00:58:24.728 --> 00:58:26.028 Like, everything's monitored as well. 00:58:26.128 --> 00:58:28.610 So, like, when you get your rack, like, 00:58:28.670 --> 00:58:29.811 none of the racks have anything in them. 00:58:30.271 --> 00:58:31.612 Like, all the new racks, they're just new. 00:58:31.731 --> 00:58:32.492 There's nothing in them. 00:58:33.032 --> 00:58:34.632 Unless they've been used previously. 00:58:34.873 --> 00:58:36.173 They'll still charge you a setup fee, 00:58:36.193 --> 00:58:36.313 but... 00:58:37.074 --> 00:58:38.175 you know, once people are in racks, 00:58:38.195 --> 00:58:39.255 they don't fucking leave anyway. 00:58:39.356 --> 00:58:39.556 Right. 00:58:39.596 --> 00:58:40.275 They just, 00:58:40.315 --> 00:58:41.317 they get a rack and then that's it. 00:58:41.717 --> 00:58:41.876 Right. 00:58:41.916 --> 00:58:45.018 But, um, because, you know, it's like AWS, 00:58:45.159 --> 00:58:45.699 Google, 00:58:45.780 --> 00:58:47.079 like they just fucking buy like a 00:58:47.099 --> 00:58:48.460 whole data hall and then be like, yeah. 00:58:48.681 --> 00:58:49.342 Oh yeah. 00:58:49.402 --> 00:58:49.882 Yeah. 00:58:50.081 --> 00:58:50.963 Isn't the one that we're in. 00:58:51.242 --> 00:58:51.422 Yeah. 00:58:51.922 --> 00:58:52.123 Yeah. 00:58:52.884 --> 00:58:54.405 Um, so, um, 00:58:55.485 --> 00:58:56.166 What was I going with that? 00:58:56.286 --> 00:58:56.567 Oh, yeah. 00:58:56.606 --> 00:58:57.887 So you pay a setup fee when 00:58:57.907 --> 00:58:58.889 you get the rack set up. 00:58:58.969 --> 00:59:00.311 And they just bring everything. 00:59:00.331 --> 00:59:01.391 They got a fucking massive 00:59:01.411 --> 00:59:03.132 breaker they put up at the 00:59:03.172 --> 00:59:04.715 top of your rack. 00:59:04.775 --> 00:59:07.197 Two PDUs that stretch the entire length. 00:59:07.376 --> 00:59:08.677 The PDUs are all monitored. 00:59:08.858 --> 00:59:09.920 So you can see the kilowatts 00:59:09.960 --> 00:59:11.021 running through each PDU. 00:59:11.041 --> 00:59:11.860 Do you monitor those? 00:59:11.880 --> 00:59:14.483 They monitor them. 00:59:14.503 --> 00:59:15.364 They're in the web portal. 00:59:15.644 --> 00:59:16.264 I can go in there. 00:59:16.365 --> 00:59:17.806 Can you bring them into Zabbix? 00:59:17.827 --> 00:59:18.847 Because I bring ours into Zabbix. 00:59:20.489 --> 00:59:21.291 I do not think so. 00:59:21.791 --> 00:59:21.851 No. 00:59:21.871 --> 00:59:22.032 Okay. 00:59:22.132 --> 00:59:22.893 All right. 00:59:22.952 --> 00:59:24.014 I don't think they have an API. 00:59:26.195 --> 00:59:27.918 But you can see like the 00:59:28.177 --> 00:59:31.521 inlet air temperature, humidity, 00:59:31.702 --> 00:59:32.182 all that type of shit. 00:59:32.202 --> 00:59:35.025 Yeah. 00:59:35.606 --> 00:59:36.987 Usually about twenty five degrees. 00:59:37.068 --> 00:59:37.969 It's pretty nice in there. 00:59:38.710 --> 00:59:40.231 Unless you stand in the hot aisle, 00:59:40.512 --> 00:59:41.172 in which case. 00:59:42.677 --> 00:59:43.876 You're going to fucking die real quick? 00:59:44.177 --> 00:59:44.777 Yeah, that's nice. 00:59:44.797 --> 00:59:45.677 It's fucking hot in there. 00:59:45.757 --> 00:59:47.478 One of our DCs has a hot-cold wall, 00:59:47.498 --> 00:59:48.297 and the other one does not. 00:59:48.378 --> 00:59:49.059 So the other one is just, 00:59:49.239 --> 00:59:50.099 it's just the whole thing. 00:59:50.438 --> 00:59:52.239 Actually, it's mostly empty. 00:59:52.418 --> 00:59:53.400 It's a really empty room 00:59:53.440 --> 00:59:54.360 that's over-cooled. 00:59:54.940 --> 00:59:57.601 So, like, it's cold in there. 00:59:57.860 --> 00:59:59.760 Like, it is, I don't even know. 00:59:59.780 --> 01:00:00.840 I could probably look at some drives, 01:00:00.880 --> 01:00:02.161 but I'm guessing it's probably more like, 01:00:03.561 --> 01:00:05.842 uh, eighteen degrees C? 01:00:07.483 --> 01:00:08.003 Something like that? 01:00:08.597 --> 01:00:09.657 Yeah. 01:00:12.338 --> 01:00:13.478 If they don't have hot, cold, 01:00:13.998 --> 01:00:15.559 they must be like spending 01:00:15.719 --> 01:00:16.998 some money on cooling. 01:00:17.059 --> 01:00:17.599 I don't give a shit. 01:00:18.420 --> 01:00:20.340 Yeah. 01:00:20.360 --> 01:00:21.119 There's not, again, 01:00:21.179 --> 01:00:22.059 it's a pretty large air 01:00:22.099 --> 01:00:22.920 handler and there's not a, 01:00:22.940 --> 01:00:23.880 there's not a ton of room in there. 01:00:23.920 --> 01:00:25.221 Like we're in the same DC as like, 01:00:25.820 --> 01:00:27.121 so in the U S and Amazon, uh, 01:00:27.900 --> 01:00:28.981 I don't know if they do that there, 01:00:29.021 --> 01:00:31.101 but like, um, they show, well, 01:00:31.161 --> 01:00:33.282 obviously we rephrase that 01:00:33.722 --> 01:00:35.483 in the U S they stream 01:00:35.643 --> 01:00:36.762 Thursday night NFL games. 01:00:38.409 --> 01:00:39.231 Like football games? 01:00:39.572 --> 01:00:40.235 So Amazon has... 01:00:41.661 --> 01:00:43.222 So they have a fuckload of 01:00:43.242 --> 01:00:44.583 DCs that do that. 01:00:44.742 --> 01:00:46.284 There's also here where I'm at, 01:00:46.664 --> 01:00:47.623 I know that there's an 01:00:47.764 --> 01:00:49.065 Amazon rack there that does 01:00:49.164 --> 01:00:50.465 that specifically for that streaming. 01:00:50.565 --> 01:00:51.565 And also the rack that we 01:00:51.606 --> 01:00:54.728 happen to be in where our main DC is, 01:00:54.748 --> 01:00:55.548 which is in Michigan, 01:00:55.889 --> 01:00:57.068 they also have a rack there. 01:00:57.349 --> 01:00:58.789 And so, yes, that's exactly what they do. 01:00:58.829 --> 01:01:00.650 They put racks in that just 01:01:00.771 --> 01:01:02.811 rebroadcast those streams 01:01:03.132 --> 01:01:04.994 to get lower latency to a 01:01:05.034 --> 01:01:05.753 shitload of viewers. 01:01:05.793 --> 01:01:06.954 So it's basically a CDN 01:01:07.614 --> 01:01:10.697 specifically for retransmitting 01:01:11.601 --> 01:01:12.217 one game a week. 01:01:13.327 --> 01:01:14.208 Wow. 01:01:14.268 --> 01:01:15.108 But it might be more than that. 01:01:15.148 --> 01:01:15.929 They might do other stuff 01:01:15.949 --> 01:01:16.951 with Amazon prime and like 01:01:16.990 --> 01:01:18.751 the video thing, but for the most part, 01:01:18.911 --> 01:01:19.893 it is a shitload of 01:01:19.913 --> 01:01:20.833 Thursday night football stuff. 01:01:21.273 --> 01:01:22.934 Just a, just a video relay. 01:01:22.956 --> 01:01:23.036 Yeah. 01:01:23.076 --> 01:01:25.838 Like, like, like, like we did a, 01:01:26.057 --> 01:01:27.018 we did a DC review. 01:01:27.039 --> 01:01:28.780 We here where I'm at. 01:01:28.820 --> 01:01:29.380 And I know the, 01:01:29.621 --> 01:01:31.001 the guy who runs that DC 01:01:31.041 --> 01:01:31.762 and we walked through and 01:01:31.782 --> 01:01:32.422 we were walking through 01:01:32.483 --> 01:01:33.884 like just the power and everything else. 01:01:33.903 --> 01:01:35.045 And like where we have stuff 01:01:35.065 --> 01:01:35.865 in there and everything else, 01:01:35.905 --> 01:01:36.405 which is cool. 01:01:37.126 --> 01:01:39.228 And, and we walked the AWS rack. 01:01:39.248 --> 01:01:40.108 I'm like, is that, is like, 01:01:40.148 --> 01:01:41.130 is it my exact S three? 01:01:41.190 --> 01:01:41.349 Yeah. 01:01:41.909 --> 01:01:43.351 Is that, you know, is that just a, 01:01:43.532 --> 01:01:44.632 you know, like people who like, 01:01:44.693 --> 01:01:45.393 cause a lot of people can, 01:01:45.833 --> 01:01:47.474 you can buy an S three rack 01:01:47.856 --> 01:01:48.797 and stick it in their local 01:01:48.817 --> 01:01:50.717 debt data center, but be only for them. 01:01:50.858 --> 01:01:51.318 You know what I mean? 01:01:51.358 --> 01:01:51.719 Like you could, 01:01:51.978 --> 01:01:52.800 you could buy a King notes 01:01:52.940 --> 01:01:55.262 S three rack or not S three at AWS rack. 01:01:55.282 --> 01:01:55.443 Right. 01:01:55.802 --> 01:01:56.804 And just stick it in there and just, 01:01:56.824 --> 01:01:57.965 you could run EC two and S 01:01:57.985 --> 01:02:00.927 three instances that are in AWS portal, 01:02:00.947 --> 01:02:02.489 but it's only in your DC, 01:02:02.608 --> 01:02:04.050 in your locked cabinet. 01:02:05.009 --> 01:02:05.748 It's like literally a 01:02:05.829 --> 01:02:07.389 private AWS cloud that you 01:02:07.429 --> 01:02:09.851 manage through the same portal. 01:02:09.891 --> 01:02:13.152 So like large US banks do that a lot. 01:02:13.572 --> 01:02:17.213 They'll buy literally AWS racks, 01:02:17.554 --> 01:02:18.355 put those in, 01:02:18.394 --> 01:02:19.855 and they had their own DC 01:02:20.034 --> 01:02:21.235 that only has their own ship, 01:02:21.255 --> 01:02:22.817 but it's all managed in the AWS portal. 01:02:22.856 --> 01:02:24.958 So you can do all the same stuff, 01:02:25.217 --> 01:02:26.177 but you have a hundred 01:02:26.197 --> 01:02:26.858 percent control of that. 01:02:26.878 --> 01:02:27.938 There's nobody next to you. 01:02:28.259 --> 01:02:29.960 It's a hundred percent your processes. 01:02:30.440 --> 01:02:31.260 It's fucking awesome, right? 01:02:31.280 --> 01:02:32.221 It's fucking cool. 01:02:33.161 --> 01:02:33.442 Um, 01:02:33.461 --> 01:02:35.123 so there's a bunch of those there as 01:02:35.163 --> 01:02:35.282 well. 01:02:35.302 --> 01:02:35.903 I'm like, is that what that is? 01:02:35.923 --> 01:02:38.425 He's like, no, no, that's all football. 01:02:39.047 --> 01:02:40.708 So I guess when, when, uh, 01:02:40.728 --> 01:02:41.668 there's streaming football 01:02:41.688 --> 01:02:42.369 games or something that 01:02:42.389 --> 01:02:43.490 that might be a place that they do, 01:02:43.510 --> 01:02:43.891 you know, 01:02:43.911 --> 01:02:44.972 either delivery or they might 01:02:45.012 --> 01:02:45.572 actually use it as a 01:02:45.632 --> 01:02:46.994 jumping point to send video 01:02:47.074 --> 01:02:47.775 out or something like that. 01:02:48.403 --> 01:02:49.063 It's fucking crazy. 01:02:49.405 --> 01:02:50.005 There's so much of that 01:02:50.045 --> 01:02:51.005 infrastructure that's out 01:02:51.025 --> 01:02:53.367 there that is so much larger. 01:02:53.427 --> 01:02:54.467 Like I'm watching a football 01:02:54.487 --> 01:02:55.268 game on browser. 01:02:55.288 --> 01:02:55.809 Who gives a shit? 01:02:56.329 --> 01:02:57.530 There is so much fucking 01:02:57.550 --> 01:02:58.570 infrastructure behind that. 01:02:58.710 --> 01:02:59.590 That's delivering that. 01:02:59.690 --> 01:03:00.452 That it's always, 01:03:00.952 --> 01:03:01.552 it's always like 01:03:01.632 --> 01:03:02.992 interesting to run into those things. 01:03:04.673 --> 01:03:05.054 I mean, 01:03:05.375 --> 01:03:08.436 just wandering around a DC is like 01:03:08.737 --> 01:03:10.018 pretty interesting in itself. 01:03:10.038 --> 01:03:11.318 It's like another world. 01:03:12.278 --> 01:03:15.280 It's like fucking, you know, it's, 01:03:17.521 --> 01:03:17.722 you know, 01:03:17.742 --> 01:03:20.244 when you like walk into the Lambo 01:03:20.284 --> 01:03:22.106 dealership and you're like, Oh, 01:03:22.126 --> 01:03:23.166 these are all really nice. 01:03:23.668 --> 01:03:24.668 And then you get back out 01:03:24.708 --> 01:03:26.889 and you're fucking like old Cutlass. 01:03:30.313 --> 01:03:31.393 You look at those Hester nodes, 01:03:31.414 --> 01:03:34.856 like the racks of nice shit. 01:03:34.876 --> 01:03:35.458 And then you're like, 01:03:35.478 --> 01:03:38.219 go back to your fucking like Honda jazz. 01:03:39.701 --> 01:03:40.402 Yeah. 01:03:41.302 --> 01:03:41.742 So yeah, 01:03:43.518 --> 01:03:46.862 I'm noticing there is more 01:03:47.021 --> 01:03:49.864 and more and more like 01:03:50.103 --> 01:03:51.804 micro tech stuff showing up in the, 01:03:52.126 --> 01:03:53.286 in the data centers that I've been. 01:03:53.347 --> 01:03:53.806 Oh really? 01:03:54.047 --> 01:03:54.367 No shit. 01:03:54.487 --> 01:03:55.047 Oh yeah. 01:03:55.168 --> 01:03:56.108 Yeah, man. 01:03:56.208 --> 01:03:58.831 Uh, like I don't know what, 01:03:59.411 --> 01:04:01.413 what Cisco and that do, but, um, 01:04:02.960 --> 01:04:03.661 because I've never used it 01:04:03.822 --> 01:04:05.762 but microtech have like 01:04:05.782 --> 01:04:07.344 this multi-shazzy link 01:04:07.384 --> 01:04:08.965 aggregation um which is 01:04:09.306 --> 01:04:11.047 pretty pretty easy to 01:04:11.666 --> 01:04:13.429 implement um well you know 01:04:13.668 --> 01:04:16.891 easy nothing's easy but you 01:04:16.911 --> 01:04:18.913 know not not too fucking 01:04:18.952 --> 01:04:21.474 crazy um seeing a lot of 01:04:21.494 --> 01:04:22.755 that man like two two 01:04:22.775 --> 01:04:23.956 routers two switches all 01:04:24.016 --> 01:04:25.137 over the place in there um 01:04:25.217 --> 01:04:27.079 microtech with an umbilical you mean 01:04:29.730 --> 01:04:30.309 An umbilical. 01:04:31.050 --> 01:04:32.851 Like between those switches 01:04:32.871 --> 01:04:33.652 and between those routers? 01:04:33.672 --> 01:04:34.213 Is that what you mean? 01:04:34.293 --> 01:04:35.414 Like for resiliency? 01:04:35.554 --> 01:04:37.934 Yeah, so like two inlet routers. 01:04:38.275 --> 01:04:39.815 So they'll have like two 01:04:40.016 --> 01:04:41.557 internet connections, one to each router. 01:04:42.697 --> 01:04:44.719 And then like multi-chassis 01:04:44.760 --> 01:04:45.619 lagged together. 01:04:45.679 --> 01:04:47.101 So there's like a joint 01:04:47.240 --> 01:04:49.422 between the two routers. 01:04:49.461 --> 01:04:51.063 So they work together. 01:04:51.764 --> 01:04:53.244 That's literally what failed 01:04:53.284 --> 01:04:54.184 in our DC two weeks ago. 01:04:55.025 --> 01:04:55.485 Oh, yeah. 01:04:55.786 --> 01:04:57.726 That's exactly what I'm talking about. 01:04:57.887 --> 01:04:59.788 And then they're 01:04:59.907 --> 01:05:05.869 cross-connected with LACPs to switches, 01:05:05.929 --> 01:05:07.731 and then every chassis has 01:05:09.411 --> 01:05:10.992 two connections, one to each switch. 01:05:11.452 --> 01:05:11.672 Right. 01:05:12.213 --> 01:05:15.833 And it's all fucking... But 01:05:16.295 --> 01:05:17.614 I've seen a lot of that. 01:05:17.735 --> 01:05:18.454 And it's a good 01:05:18.474 --> 01:05:20.416 implementation on microtech, 01:05:20.436 --> 01:05:22.297 and you can use BGP and 01:05:22.317 --> 01:05:23.336 that type of shit. 01:05:23.396 --> 01:05:24.757 I've never used it. 01:05:25.077 --> 01:05:25.818 I've not used it. 01:05:26.018 --> 01:05:27.318 Probably will at some point, 01:05:27.579 --> 01:05:32.940 but I don't know if we'll use it. 01:05:33.059 --> 01:05:36.559 I have tried it before and I hated it. 01:05:36.980 --> 01:05:37.780 It's interesting. 01:05:38.581 --> 01:05:38.760 I mean, 01:05:39.101 --> 01:05:40.860 I shouldn't because Cisco iOS has 01:05:40.920 --> 01:05:41.621 its own language. 01:05:41.760 --> 01:05:43.001 Like you have to know that language, 01:05:43.061 --> 01:05:43.262 right? 01:05:43.382 --> 01:05:43.961 And MicroTik, 01:05:44.001 --> 01:05:44.981 you have to know the language. 01:05:46.081 --> 01:05:47.043 So MicroTik, 01:05:47.063 --> 01:05:51.182 you can program it via CLI or 01:05:51.222 --> 01:05:52.563 you can use their Winbox. 01:05:53.023 --> 01:05:53.184 Right. 01:05:53.344 --> 01:05:53.503 And... 01:05:55.295 --> 01:05:58.777 like the the um you know how 01:05:58.838 --> 01:06:00.579 the cli is structured is 01:06:00.659 --> 01:06:04.019 how the the um ui is 01:06:04.039 --> 01:06:05.400 structured as well right so 01:06:06.041 --> 01:06:06.800 you know if you're like 01:06:06.840 --> 01:06:10.202 slash ip it has all of the 01:06:11.123 --> 01:06:14.164 um you know other options or other 01:06:16.344 --> 01:06:17.545 let's say folders in there, 01:06:17.726 --> 01:06:19.007 are like other things when 01:06:19.047 --> 01:06:21.728 you click on IP in the UI. 01:06:21.929 --> 01:06:22.989 It's got all the same shit. 01:06:23.429 --> 01:06:26.271 So you can pretty easily slash, slash, 01:06:26.291 --> 01:06:27.632 slash into the thing that 01:06:27.653 --> 01:06:28.653 you want and then set the 01:06:28.693 --> 01:06:30.675 options you want if you know the CLI. 01:06:30.715 --> 01:06:31.956 And I mean, 01:06:32.137 --> 01:06:33.757 their documentation is quite good. 01:06:35.199 --> 01:06:36.019 When you're reading through 01:06:36.039 --> 01:06:36.860 the documentation, 01:06:36.880 --> 01:06:39.963 it will give you the UI and then the... 01:06:41.844 --> 01:06:42.925 Except for like, you know, 01:06:42.965 --> 01:06:44.126 they've got case studies 01:06:44.146 --> 01:06:45.407 where they're like, you know, 01:06:46.088 --> 01:06:47.869 this is how this will work 01:06:48.048 --> 01:06:49.070 in this scenario. 01:06:49.150 --> 01:06:49.791 And then here's all the 01:06:49.811 --> 01:06:50.992 commands that you would do 01:06:51.012 --> 01:06:52.353 to make this type of thing work. 01:06:52.373 --> 01:06:53.313 They'll describe it and then 01:06:53.333 --> 01:06:53.753 they'll give all the 01:06:53.773 --> 01:06:54.954 commands because it just 01:06:54.974 --> 01:06:56.135 takes up too much room to 01:06:56.175 --> 01:06:57.117 put like all of the 01:06:58.257 --> 01:06:59.519 UI screenshots, right? 01:06:59.920 --> 01:07:01.702 But it's the same anyway, more or less. 01:07:02.063 --> 01:07:03.264 I mean, 01:07:03.485 --> 01:07:04.686 from a quality and from a 01:07:04.766 --> 01:07:05.748 reliability perspective, 01:07:05.768 --> 01:07:06.688 there's no issues at all. 01:07:06.849 --> 01:07:07.829 I don't have any issues with that. 01:07:08.471 --> 01:07:10.213 I know it's well-loved. 01:07:10.373 --> 01:07:11.414 I know that it's extremely 01:07:11.454 --> 01:07:12.755 stable and it's extremely 01:07:12.815 --> 01:07:14.197 well-architected and engineered. 01:07:14.847 --> 01:07:16.088 I think you have to get into 01:07:16.108 --> 01:07:18.329 that ecosystem to really know it. 01:07:18.409 --> 01:07:19.909 And depending on what you're doing, 01:07:20.250 --> 01:07:21.911 that either makes sense or it doesn't. 01:07:21.931 --> 01:07:22.632 You know what I mean? 01:07:22.652 --> 01:07:22.771 Well, 01:07:22.791 --> 01:07:24.432 you have to spend the time learning it. 01:07:24.672 --> 01:07:26.074 Yeah, you do. 01:07:26.313 --> 01:07:26.675 But, I mean, 01:07:26.715 --> 01:07:28.416 it's the same with any ecosystem, right? 01:07:28.615 --> 01:07:29.836 It doesn't matter how well you know. 01:07:30.056 --> 01:07:32.697 But it is actually, you know, 01:07:32.797 --> 01:07:33.878 it's a big advantage if you 01:07:33.958 --> 01:07:36.360 understand networking and a 01:07:36.380 --> 01:07:37.300 lot of other stuff. 01:07:37.320 --> 01:07:39.021 But you could replace that 01:07:39.041 --> 01:07:39.762 with a – you could 01:07:39.922 --> 01:07:41.184 fat-finger yourself through 01:07:41.304 --> 01:07:44.336 a PFSense structure and get – 01:07:43.465 --> 01:07:44.545 get the entire way there 01:07:44.565 --> 01:07:46.326 yeah and yeah and not and 01:07:46.346 --> 01:07:48.146 not yourself over because 01:07:48.166 --> 01:07:49.027 there's some safeguards in 01:07:49.047 --> 01:07:50.588 there but but you don't 01:07:50.608 --> 01:07:51.188 like there's a lot of 01:07:51.208 --> 01:07:52.387 automatic things that 01:07:52.427 --> 01:07:53.048 happen yeah you don't 01:07:53.068 --> 01:07:53.909 understand the same level 01:07:53.929 --> 01:07:54.989 that you have to understand 01:07:55.009 --> 01:07:55.630 that microtech right 01:07:55.670 --> 01:07:56.690 because you even said I 01:07:56.710 --> 01:07:57.150 think you were doing 01:07:57.190 --> 01:07:58.690 something like you you you 01:07:58.731 --> 01:07:59.610 said it was fine you went 01:07:59.650 --> 01:08:00.972 home you're like ah and you 01:08:00.992 --> 01:08:01.811 gotta turn around and drive 01:08:01.851 --> 01:08:03.012 back right big that that is 01:08:03.112 --> 01:08:05.213 a that is uniquely like a 01:08:05.273 --> 01:08:06.554 cisco microtech type of 01:08:06.594 --> 01:08:08.094 experience where there's no 01:08:08.233 --> 01:08:09.195 there's no guard rails 01:08:10.025 --> 01:08:11.166 know what I mean like like 01:08:11.186 --> 01:08:11.907 you could you could build 01:08:11.967 --> 01:08:12.849 totally unroutable 01:08:12.889 --> 01:08:14.170 situation things like yeah 01:08:14.250 --> 01:08:15.570 unrouting nothing yeah yeah 01:08:15.590 --> 01:08:17.332 like you're totally on your 01:08:17.393 --> 01:08:21.015 own right the yeah so 01:08:22.597 --> 01:08:25.400 learning marketeek is like 01:08:25.461 --> 01:08:26.462 sometimes you just have to 01:08:26.481 --> 01:08:29.524 try right yeah and and I'm 01:08:29.564 --> 01:08:30.985 trying that's like off to 01:08:31.006 --> 01:08:32.247 the side in the data center 01:08:32.327 --> 01:08:36.051 and then you know if it doesn't work 01:08:37.329 --> 01:08:38.911 there is there are I'll talk 01:08:38.930 --> 01:08:39.952 about the mechanisms that 01:08:40.051 --> 01:08:41.072 you can avoid this with but 01:08:41.573 --> 01:08:43.033 you know if uh if you 01:08:43.054 --> 01:08:43.974 change something and it 01:08:44.094 --> 01:08:45.635 just it doesn't work it'll 01:08:45.654 --> 01:08:47.176 just it and you have to go 01:08:47.195 --> 01:08:48.556 there right and plug into 01:08:48.577 --> 01:08:50.238 the thing into like an 01:08:50.318 --> 01:08:51.798 unconfigured port to be 01:08:51.838 --> 01:08:52.979 able to recover it it's 01:08:53.000 --> 01:08:54.079 like you can't do it 01:08:54.119 --> 01:08:56.041 remotely yeah it's like you 01:08:56.121 --> 01:08:57.801 ran go go yourself that 01:08:57.822 --> 01:08:58.823 shell and things like all 01:08:58.863 --> 01:09:00.064 right sir I'm gonna run it 01:09:00.104 --> 01:09:03.126 yeah I'm out come on come on pick me up 01:09:04.039 --> 01:09:05.921 So I've probably driven back 01:09:05.942 --> 01:09:06.842 to the data center ten 01:09:06.903 --> 01:09:07.863 times just to unfuck 01:09:07.882 --> 01:09:10.645 something that I fucked. 01:09:10.666 --> 01:09:11.806 Just on, you know, 01:09:12.506 --> 01:09:14.389 in the testing ground type of scenario. 01:09:14.448 --> 01:09:15.449 The glorious life of the 01:09:17.470 --> 01:09:18.492 data center operator. 01:09:19.052 --> 01:09:19.693 That's what you're getting into. 01:09:19.712 --> 01:09:20.613 Yeah, exactly. 01:09:21.217 --> 01:09:23.239 The front door, like, you know, 01:09:23.260 --> 01:09:24.681 the doorman really knows me well now. 01:09:24.702 --> 01:09:25.943 He's like, hey, what's going on? 01:09:25.983 --> 01:09:26.963 Come to unfuck some shit? 01:09:29.987 --> 01:09:31.287 So there is a thing, though. 01:09:31.307 --> 01:09:31.889 There's a button, 01:09:32.088 --> 01:09:33.751 top right-hand corner or 01:09:33.791 --> 01:09:35.332 top left-hand corner in the old Winbox. 01:09:35.953 --> 01:09:37.715 And you can press it. 01:09:38.015 --> 01:09:39.055 It says safe mode. 01:09:39.435 --> 01:09:40.356 So if you apply a 01:09:40.396 --> 01:09:41.798 configuration and then... 01:09:43.143 --> 01:09:44.724 the um router does not 01:09:44.783 --> 01:09:47.846 receive any more like into 01:09:48.146 --> 01:09:50.948 um interaction yeah and 01:09:50.988 --> 01:09:52.710 winbox disconnects it'll be 01:09:52.751 --> 01:09:53.650 like well they fuck 01:09:53.671 --> 01:09:54.692 something so we'll like 01:09:54.912 --> 01:09:56.533 undo that is that true yeah 01:09:56.592 --> 01:09:58.795 oh that's nice yeah I never 01:09:58.914 --> 01:10:01.037 click it I was gonna say 01:10:01.117 --> 01:10:02.217 and yet you've gone ten 01:10:02.257 --> 01:10:04.059 times to the data center 01:10:04.078 --> 01:10:05.100 because it's not like it 01:10:05.119 --> 01:10:07.381 won't be so it won't be like um 01:10:10.453 --> 01:10:11.734 You want to be like, you press the button, 01:10:11.793 --> 01:10:12.033 right? 01:10:12.814 --> 01:10:14.694 And then everything you do 01:10:15.314 --> 01:10:16.496 until you unpress the 01:10:16.536 --> 01:10:18.256 button will revert if you 01:10:18.277 --> 01:10:18.817 break something. 01:10:19.037 --> 01:10:20.478 So you have to be like constantly on, off, 01:10:20.557 --> 01:10:22.198 on, off, on, off. 01:10:22.217 --> 01:10:24.158 You're like, right, exactly. 01:10:24.279 --> 01:10:24.439 Yeah. 01:10:24.639 --> 01:10:25.939 So when you're like doing a 01:10:25.979 --> 01:10:27.100 lot of configuration, 01:10:27.140 --> 01:10:28.381 trying some things out and 01:10:28.421 --> 01:10:31.882 then it messes up and then 01:10:32.703 --> 01:10:34.423 reverts all of that configuration, well, 01:10:34.444 --> 01:10:35.684 I just wasted half a fucking day. 01:10:35.724 --> 01:10:36.805 That's so funny, dude. 01:10:37.529 --> 01:10:37.668 Yeah, 01:10:37.708 --> 01:10:38.909 you'd think it would just revert the 01:10:38.949 --> 01:10:40.211 last thing that you did, but no. 01:10:40.230 --> 01:10:41.832 Well, 01:10:41.851 --> 01:10:43.313 you could probably make incremental 01:10:43.332 --> 01:10:45.014 saves in there, and then... Oh, 01:10:45.033 --> 01:10:46.113 does that require... What's 01:10:46.134 --> 01:10:47.095 the reload process? 01:10:47.215 --> 01:10:48.475 It doesn't reboot or anything, right? 01:10:48.496 --> 01:10:49.496 It just reloads the tables? 01:10:49.555 --> 01:10:50.476 No, it just reverts it, 01:10:50.497 --> 01:10:52.158 and then that's it. 01:10:56.560 --> 01:10:58.782 Because when you're dealing 01:10:58.801 --> 01:11:00.643 with VLANs and stuff, 01:11:00.662 --> 01:11:02.564 there's a lot of nuance in VLANs. 01:11:02.904 --> 01:11:03.345 Oh, for sure. 01:11:03.465 --> 01:11:04.426 And in the... 01:11:05.666 --> 01:11:06.426 In MicroTik, 01:11:06.466 --> 01:11:07.905 they've got a lot of hardware 01:11:07.926 --> 01:11:08.587 offloading now, 01:11:09.167 --> 01:11:10.766 depending on your switch chips and stuff. 01:11:10.787 --> 01:11:13.046 So that's mostly what I've 01:11:13.067 --> 01:11:14.747 been messing around with is 01:11:14.768 --> 01:11:15.667 the hardware offload. 01:11:15.807 --> 01:11:18.368 And the way that you do the 01:11:18.448 --> 01:11:19.948 VLANs with hardware 01:11:19.988 --> 01:11:21.408 offloading is completely 01:11:21.448 --> 01:11:22.208 different to the way you 01:11:22.229 --> 01:11:23.849 would do it without hardware offloading. 01:11:25.609 --> 01:11:27.489 So, which is, 01:11:28.850 --> 01:11:30.729 I guess that's just how it is. 01:11:30.750 --> 01:11:31.729 Do you really need that even 01:11:32.411 --> 01:11:34.890 at those twenty five gigabit speeds? 01:11:36.052 --> 01:11:36.432 Oh, man. 01:11:37.434 --> 01:11:40.756 So if you use the CPU to route, 01:11:42.257 --> 01:11:44.319 like the CPUs in microtech stuff, 01:11:44.420 --> 01:11:46.381 when they've got like these 01:11:46.400 --> 01:11:48.882 bulletproof fucking switch chips, 01:11:48.962 --> 01:11:52.546 like the CPU is dog shit. 01:11:52.645 --> 01:11:54.568 It's like a fucking hundred hertz potato. 01:11:54.967 --> 01:11:55.247 Like... 01:11:56.208 --> 01:11:57.029 It's not. 01:11:57.449 --> 01:11:58.130 They're usually like an 01:11:58.170 --> 01:11:59.350 eight hundred Hertz ARM chip, 01:12:00.811 --> 01:12:02.032 but they do not cope well 01:12:02.192 --> 01:12:03.472 if you don't use your 01:12:03.493 --> 01:12:06.173 hardware offloading for workloads. 01:12:07.875 --> 01:12:08.935 They'll handle some stuff, 01:12:09.176 --> 01:12:12.118 but even if you have to 01:12:12.177 --> 01:12:13.478 look at the block diagram 01:12:13.738 --> 01:12:16.159 when you're buying MicroTik switches. 01:12:16.939 --> 01:12:18.301 so bring up the block 01:12:18.320 --> 01:12:19.520 diagram think about what 01:12:19.541 --> 01:12:21.301 you want to do and see if 01:12:21.502 --> 01:12:23.143 this particular product is 01:12:23.182 --> 01:12:24.143 the right one for you 01:12:24.684 --> 01:12:26.524 because for example like if 01:12:26.545 --> 01:12:29.966 you buy um if you buy a 01:12:30.006 --> 01:12:31.226 hundred gig switch right 01:12:31.327 --> 01:12:32.948 and it's got a and it's got 01:12:32.988 --> 01:12:34.667 a little um ethernet port 01:12:34.707 --> 01:12:35.548 over here and you'd be like 01:12:35.729 --> 01:12:36.849 oh yeah I can like have an 01:12:36.908 --> 01:12:39.149 uplink to my uh to my 01:12:40.230 --> 01:12:41.771 internet router to grab the 01:12:41.811 --> 01:12:43.351 internet and then you know 01:12:43.471 --> 01:12:44.453 all of those things that 01:12:44.912 --> 01:12:45.953 want to talk to each other 01:12:45.993 --> 01:12:47.212 on the hundred gig network, 01:12:47.353 --> 01:12:49.453 fine for storage or whatever, 01:12:49.913 --> 01:12:50.875 we'll be able to get like a 01:12:51.015 --> 01:12:52.095 one gig connection to the 01:12:52.136 --> 01:12:52.836 internet or something. 01:12:52.876 --> 01:12:53.036 Right. 01:12:53.055 --> 01:12:53.695 Yeah. 01:12:53.775 --> 01:12:54.197 But, 01:12:54.237 --> 01:12:57.198 but that one gig connection might be 01:12:57.578 --> 01:12:58.498 on a completely different 01:12:58.538 --> 01:13:00.038 path outside of the switch chip. 01:13:00.118 --> 01:13:00.699 And it will be, 01:13:00.760 --> 01:13:02.440 it'll be off the side of the CPU. 01:13:03.201 --> 01:13:04.582 So any, then all, 01:13:04.822 --> 01:13:06.021 all internet traffic then is, 01:13:06.162 --> 01:13:07.382 is being routed through that, 01:13:07.563 --> 01:13:10.243 through the CPU and not the switch chip. 01:13:10.283 --> 01:13:11.425 So, you're not, 01:13:11.444 --> 01:13:12.104 you're not doing a good job 01:13:12.125 --> 01:13:13.525 selling that all infrastructure. 01:13:13.546 --> 01:13:13.626 Yeah. 01:13:14.570 --> 01:13:15.831 but that's not the way you would do it. 01:13:15.871 --> 01:13:16.150 Right. 01:13:16.251 --> 01:13:17.351 So the way you would do it 01:13:17.391 --> 01:13:18.171 is you'd have a bunch of 01:13:18.192 --> 01:13:19.811 VLANs on your hundred gig network. 01:13:20.011 --> 01:13:21.912 And then one of those VLANs would be on, 01:13:22.113 --> 01:13:23.372 on your switch. 01:13:23.552 --> 01:13:23.833 Sure. 01:13:23.873 --> 01:13:24.974 Yeah. 01:13:25.014 --> 01:13:26.514 So you wouldn't use that port, 01:13:26.573 --> 01:13:29.034 but if you're like, you know, you just, 01:13:29.135 --> 01:13:30.475 Oh fucking, I don't know. 01:13:30.534 --> 01:13:31.475 And I'll just plug this in. 01:13:31.496 --> 01:13:32.435 Cause that seems like a good 01:13:32.475 --> 01:13:34.036 option for this dumb shit I want to do. 01:13:34.716 --> 01:13:37.056 Then, um, you know, 01:13:37.077 --> 01:13:38.597 you'd probably do that and you'd be like, 01:13:39.257 --> 01:13:40.738 why is my CPU or is pegged? 01:13:42.738 --> 01:13:43.219 What's happening? 01:13:43.238 --> 01:13:43.918 Why is it all flow? 01:13:46.359 --> 01:13:48.480 But if you configure everything correctly, 01:13:48.801 --> 01:13:49.962 then your CPU will 01:13:49.981 --> 01:13:51.023 basically be doing nothing 01:13:51.342 --> 01:13:52.063 the entire time. 01:13:52.082 --> 01:13:54.524 It's just all switch chip. 01:13:54.824 --> 01:14:00.630 And those fucking things are stout, man. 01:14:01.470 --> 01:14:04.613 They definitely match the 01:14:04.653 --> 01:14:07.194 switch chips to what you're 01:14:07.213 --> 01:14:08.655 getting out of that. 01:14:08.676 --> 01:14:10.237 You'll be able to saturate every port. 01:14:10.777 --> 01:14:13.800 sure yeah yeah oh yeah yeah 01:14:13.819 --> 01:14:14.579 there's so little overhead 01:14:14.619 --> 01:14:15.240 right you're not like I 01:14:15.301 --> 01:14:16.161 mean all these things are 01:14:16.180 --> 01:14:17.141 the same way right like 01:14:17.181 --> 01:14:17.962 they're so fucking 01:14:18.002 --> 01:14:20.885 optimized for that oh yeah 01:14:20.944 --> 01:14:21.826 the ports that are in it 01:14:22.225 --> 01:14:23.186 and they're like this is 01:14:23.327 --> 01:14:24.448 the switch chip we need for 01:14:24.467 --> 01:14:26.409 that yeah like what we do 01:14:26.430 --> 01:14:27.711 on ios is exactly the same 01:14:27.751 --> 01:14:29.452 like it's it's it's fucking 01:14:29.492 --> 01:14:30.193 miserable but it's 01:14:30.372 --> 01:14:31.833 miserable because it is so 01:14:31.934 --> 01:14:34.595 fucking optimized yeah like 01:14:34.615 --> 01:14:36.057 there's there's literally 01:14:37.179 --> 01:14:38.199 billions and trillions and 01:14:38.260 --> 01:14:41.021 whatever number of, of, uh, 01:14:41.042 --> 01:14:41.823 packets flowing through 01:14:41.842 --> 01:14:44.003 these things globally in really, 01:14:44.524 --> 01:14:45.926 really high critical 01:14:45.966 --> 01:14:50.128 locations that have been, um, 01:14:50.149 --> 01:14:50.908 that have been optimized 01:14:50.948 --> 01:14:52.229 for those specific situations. 01:14:52.289 --> 01:14:52.511 Right. 01:14:52.530 --> 01:14:54.152 So like it's, it's, 01:14:54.952 --> 01:14:56.052 it's a high learning curve and there's a, 01:14:56.252 --> 01:14:56.953 there's a high amount of, 01:14:57.493 --> 01:14:58.034 you have to know what the 01:14:58.055 --> 01:14:58.954 fuck you're doing because 01:14:59.216 --> 01:15:00.115 the stakes are so high in 01:15:00.136 --> 01:15:01.077 some of those types of situations. 01:15:01.157 --> 01:15:02.158 Right. 01:15:02.198 --> 01:15:02.997 So I get it. 01:15:04.219 --> 01:15:04.880 Um, 01:15:07.400 --> 01:15:07.520 Yeah. 01:15:07.541 --> 01:15:09.182 Allura, by the way, still not moving. 01:15:10.042 --> 01:15:10.264 Really? 01:15:10.283 --> 01:15:11.184 Yeah. 01:15:12.005 --> 01:15:14.546 It stopped there two hours ago. 01:15:15.507 --> 01:15:16.048 Somebody just said, 01:15:16.088 --> 01:15:17.449 will future upgrades take 01:15:17.470 --> 01:15:20.573 this long or is it an exception? 01:15:21.033 --> 01:15:22.395 I fucking hate whinging. 01:15:22.515 --> 01:15:23.395 Welcome to Cosmos. 01:15:24.582 --> 01:15:25.082 Nobody knows. 01:15:26.802 --> 01:15:27.144 Why? 01:15:28.163 --> 01:15:28.885 It's what you're doing. 01:15:29.324 --> 01:15:31.025 They're doing a migration. 01:15:31.065 --> 01:15:32.806 They're doing a pretty large migration. 01:15:33.467 --> 01:15:34.889 Yeah. 01:15:35.708 --> 01:15:36.369 We must be close. 01:15:36.470 --> 01:15:39.551 I do have... I have thirty-four peers. 01:15:40.731 --> 01:15:41.052 Actually, no. 01:15:41.073 --> 01:15:41.773 I have twenty-seven peers. 01:15:41.792 --> 01:15:42.393 That's my limit. 01:15:45.494 --> 01:15:46.555 So I have enough peers. 01:15:46.576 --> 01:15:47.617 I haven't brought up TM Top 01:15:47.636 --> 01:15:49.198 or anything else like that. 01:15:49.257 --> 01:15:50.338 Or PV Top. 01:15:50.378 --> 01:15:52.399 Shout out to a friend of the show, 01:15:52.439 --> 01:15:52.720 BlockPay. 01:15:56.609 --> 01:15:58.390 So anyway, I don't know what's going on. 01:15:58.591 --> 01:16:00.072 I know. 01:16:00.131 --> 01:16:01.873 He's a good dude. 01:16:01.974 --> 01:16:04.574 One of my favorite people in this industry, 01:16:04.614 --> 01:16:06.055 for sure. 01:16:06.076 --> 01:16:06.657 Friend of the show. 01:16:06.676 --> 01:16:08.278 So I guess this Allura is 01:16:08.337 --> 01:16:10.078 just taking a while because 01:16:10.118 --> 01:16:12.000 everyone's probably got potatoes. 01:16:17.110 --> 01:16:17.912 I thought it was at the head 01:16:17.952 --> 01:16:19.931 of that list kenny from the 01:16:19.971 --> 01:16:21.011 allure team said they were 01:16:21.032 --> 01:16:22.552 a couple couple when we're 01:16:22.573 --> 01:16:23.313 doing like this thing where 01:16:23.332 --> 01:16:24.132 it's going through the 01:16:24.153 --> 01:16:25.394 number of pages associated 01:16:25.413 --> 01:16:27.154 to a what was that that was 01:16:27.213 --> 01:16:29.475 a um some sort of account 01:16:29.574 --> 01:16:30.515 migration right is that 01:16:30.534 --> 01:16:31.296 what it was I don't know 01:16:31.315 --> 01:16:31.975 what the hell it was 01:16:32.055 --> 01:16:33.676 exactly but and there's a 01:16:33.695 --> 01:16:34.337 bunch of pages through 01:16:34.356 --> 01:16:35.457 there and we got to nine 01:16:35.476 --> 01:16:36.237 hundred ninety one pages 01:16:36.277 --> 01:16:36.936 and they were a couple 01:16:36.957 --> 01:16:37.637 ahead of me and I was like 01:16:37.717 --> 01:16:38.537 motherfucker I was like 01:16:38.557 --> 01:16:40.559 how'd that happen usually 01:16:40.599 --> 01:16:41.939 run the fastest hardware on this shit 01:16:43.453 --> 01:16:44.295 Although this node is doing 01:16:44.335 --> 01:16:45.315 like twenty five other things. 01:16:45.336 --> 01:16:46.877 So maybe that has something to do with it. 01:16:46.896 --> 01:16:49.920 And we like good hardware on this. 01:16:49.979 --> 01:16:51.900 And I was still at least one 01:16:51.940 --> 01:16:53.021 hundred pages behind you. 01:16:53.203 --> 01:16:54.342 So, yeah, I don't know. 01:16:54.363 --> 01:16:54.944 I don't know why. 01:16:55.283 --> 01:16:56.385 Maybe maybe I don't know 01:16:56.404 --> 01:16:57.246 when it started earlier. 01:16:57.265 --> 01:16:57.685 I don't know. 01:16:58.908 --> 01:16:59.568 What are you running it on? 01:17:00.189 --> 01:17:01.869 This is, uh, these are all seven, nine, 01:17:01.890 --> 01:17:02.470 fifty X's. 01:17:02.970 --> 01:17:03.810 Yeah, this is seven, nine, 01:17:03.871 --> 01:17:04.492 fifty X reading. 01:17:04.511 --> 01:17:06.472 I don't know. 01:17:06.592 --> 01:17:07.434 I don't know, man. 01:17:07.453 --> 01:17:07.573 Angry. 01:17:07.594 --> 01:17:07.833 I don't know. 01:17:07.854 --> 01:17:08.435 I don't know. 01:17:08.574 --> 01:17:09.534 Hard drive configuration. 01:17:09.574 --> 01:17:11.837 Probably raid zeros for life. 01:17:14.899 --> 01:17:15.118 Right. 01:17:15.198 --> 01:17:15.800 Zero, baby. 01:17:16.239 --> 01:17:17.480 Can't run raid zero with L 01:17:17.621 --> 01:17:19.042 or it doesn't do remote signing. 01:17:20.882 --> 01:17:23.984 What do you mean? 01:17:24.024 --> 01:17:24.465 I don't care. 01:17:24.484 --> 01:17:25.725 I have a copy of the key. 01:17:25.966 --> 01:17:27.186 As long as I can destroy the box, 01:17:27.207 --> 01:17:27.828 I don't give a shit. 01:17:28.368 --> 01:17:29.408 It's a test net, motherfucker. 01:17:29.748 --> 01:17:31.451 It doesn't do D-Gen. 01:17:31.671 --> 01:17:33.712 Horcrux. 01:17:33.733 --> 01:17:34.253 Don't believe so. 01:17:34.608 --> 01:17:35.188 I don't believe so. 01:17:35.268 --> 01:17:36.488 I haven't done anything with it yet. 01:17:36.609 --> 01:17:37.548 Although my signing on 01:17:37.569 --> 01:17:38.970 Allura is like unbelievable. 01:17:38.989 --> 01:17:39.529 I don't even know why. 01:17:39.590 --> 01:17:40.310 I think it's because I have 01:17:40.331 --> 01:17:41.251 good peers and I've never 01:17:41.291 --> 01:17:42.091 restarted this box. 01:17:42.112 --> 01:17:45.073 I think that's all it is. 01:17:45.773 --> 01:17:47.253 Your signing, my signing. 01:17:47.713 --> 01:17:48.475 There's only like two other 01:17:48.515 --> 01:17:49.996 nodes that are like at 01:17:50.176 --> 01:17:50.815 ninety nine percent. 01:17:50.895 --> 01:17:52.256 Everybody else is at forty percent. 01:17:52.697 --> 01:17:53.796 Like it's a fucking 01:17:53.817 --> 01:17:54.778 bloodbath on this chain, 01:17:54.818 --> 01:17:55.877 except for like a couple of us. 01:17:55.898 --> 01:17:56.998 I don't know. 01:17:57.038 --> 01:17:57.719 I don't know. 01:17:57.738 --> 01:17:58.600 It has to be peering because 01:17:58.619 --> 01:18:00.980 it can't be just it has to be peering. 01:18:01.569 --> 01:18:02.990 I feel like I miss a decent 01:18:03.011 --> 01:18:03.771 amount on Laura. 01:18:03.931 --> 01:18:04.891 You miss more than I do. 01:18:06.033 --> 01:18:06.413 Hat tip. 01:18:07.094 --> 01:18:09.576 Um, but, but compared to the rest, 01:18:09.996 --> 01:18:11.176 oh my God, it's not even, 01:18:11.577 --> 01:18:12.797 you're not even in the same ballpark. 01:18:12.858 --> 01:18:14.939 Like you are not so bad. 01:18:15.199 --> 01:18:16.020 It's not bad at all. 01:18:16.081 --> 01:18:16.421 Like it is. 01:18:17.779 --> 01:18:19.119 We're within a percent of 01:18:19.140 --> 01:18:20.060 each other or something like that, 01:18:20.079 --> 01:18:21.621 but everybody else is like, 01:18:21.820 --> 01:18:24.162 it is a bloodbath on that chain. 01:18:24.802 --> 01:18:25.143 Somebody from, 01:18:25.783 --> 01:18:26.823 was it Cosmo Station or something, 01:18:26.844 --> 01:18:28.045 built a small Grafana 01:18:28.064 --> 01:18:30.426 dashboard that showed last, 01:18:30.765 --> 01:18:32.466 whatever it was, thousand blocks. 01:18:32.487 --> 01:18:33.908 They're like, oh, look at that dashboard. 01:18:33.927 --> 01:18:35.007 And we're like, oh, fuck, 01:18:35.127 --> 01:18:35.769 don't look at it. 01:18:37.288 --> 01:18:38.689 It was a hundred percent. 01:18:38.949 --> 01:18:39.810 It was brutal. 01:18:40.793 --> 01:18:42.194 It looks so fucking miserable. 01:18:42.213 --> 01:18:43.534 It was just like, it was awful. 01:18:44.114 --> 01:18:44.635 I see red. 01:18:44.675 --> 01:18:45.215 I see red. 01:18:45.274 --> 01:18:47.956 I see red. 01:18:48.275 --> 01:18:52.497 Uh, uh, uh, so actually, well, I think, 01:18:52.657 --> 01:18:53.238 you know, 01:18:53.278 --> 01:18:55.318 something that affected you and I, uh, 01:18:56.878 --> 01:18:58.139 to some degree. 01:18:58.259 --> 01:18:58.319 Hmm. 01:18:59.944 --> 01:19:04.407 Asrock, Zero Day, IPMI. 01:19:05.307 --> 01:19:06.028 Ring any bells? 01:19:06.509 --> 01:19:06.748 Actually, 01:19:07.668 --> 01:19:08.909 they probably didn't send you 01:19:08.949 --> 01:19:10.371 that same email they sent 01:19:10.390 --> 01:19:11.131 me because they actually 01:19:11.171 --> 01:19:15.094 wanted a password for the IPMI. 01:19:15.293 --> 01:19:16.095 IP projects. 01:19:16.175 --> 01:19:16.935 I'm going to fucking name 01:19:16.956 --> 01:19:18.315 them because they've been total cunts. 01:19:19.077 --> 01:19:19.197 So... 01:19:22.596 --> 01:19:22.737 Man, 01:19:22.756 --> 01:19:24.118 I was badgering them so hard for a 01:19:24.158 --> 01:19:25.159 refund for the last month. 01:19:25.300 --> 01:19:27.122 So the backstory is that, 01:19:27.703 --> 01:19:28.484 did I bring this to you? 01:19:28.543 --> 01:19:29.506 I think I brought this to you, right? 01:19:29.525 --> 01:19:30.065 This is on me. 01:19:30.126 --> 01:19:30.266 No, 01:19:30.707 --> 01:19:32.548 I mentioned it to you and you hadn't 01:19:32.569 --> 01:19:33.170 had a problem yet. 01:19:33.189 --> 01:19:35.172 And I was like, man, this fucking box. 01:19:35.193 --> 01:19:36.654 No, no, I introduced them to you though, 01:19:36.694 --> 01:19:36.835 right? 01:19:36.895 --> 01:19:38.195 I said, I found this provider that blah, 01:19:38.216 --> 01:19:38.556 blah, blah. 01:19:39.556 --> 01:19:41.998 I think this is a... Yeah, maybe. 01:19:42.479 --> 01:19:42.819 I don't know. 01:19:42.840 --> 01:19:44.582 I think so. 01:19:44.601 --> 01:19:46.823 So one thing other good node 01:19:46.863 --> 01:19:48.404 operators do is that they 01:19:48.444 --> 01:19:50.386 find diversity in terms of 01:19:50.466 --> 01:19:51.347 bare metal providers 01:19:51.408 --> 01:19:53.109 because most of the 01:19:53.170 --> 01:19:54.230 ecosystems and a bunch of 01:19:54.251 --> 01:19:55.351 different ecosystems run on 01:19:55.452 --> 01:19:56.292 the lowest possible... 01:19:57.613 --> 01:19:59.015 configurations and not lowest, 01:19:59.055 --> 01:19:59.957 but also cheapest. 01:20:00.037 --> 01:20:01.217 And also there's a lot 01:20:01.618 --> 01:20:03.239 actual heavy amount of centralization. 01:20:03.279 --> 01:20:04.702 So that might be Hetzner is 01:20:04.742 --> 01:20:05.402 obviously a big one. 01:20:06.262 --> 01:20:07.304 OVH is a big one of the, 01:20:07.784 --> 01:20:08.364 I think a little bit of a 01:20:08.385 --> 01:20:08.985 better operator. 01:20:10.528 --> 01:20:10.707 There's, 01:20:10.768 --> 01:20:12.970 there's Contabo and like Vulture 01:20:13.069 --> 01:20:13.570 and things like that, 01:20:13.591 --> 01:20:15.252 where you have a pretty 01:20:15.332 --> 01:20:17.034 large organization that 01:20:17.055 --> 01:20:18.435 delivers bare metal nodes and 01:20:18.615 --> 01:20:19.837 globally at different at 01:20:19.917 --> 01:20:20.997 either a fixed price point 01:20:21.918 --> 01:20:23.300 um that's her I mean he's 01:20:23.340 --> 01:20:24.721 an extremely capable 01:20:24.822 --> 01:20:25.962 operator as well like we 01:20:25.983 --> 01:20:26.743 have no issues with them 01:20:26.764 --> 01:20:27.423 it's just the fact that 01:20:27.444 --> 01:20:29.006 it's so centralized that 01:20:29.326 --> 01:20:30.426 when we provide services 01:20:30.447 --> 01:20:32.228 for other teams we want 01:20:32.268 --> 01:20:33.269 diversity in that so like 01:20:33.288 --> 01:20:34.631 we use we use some smaller 01:20:34.671 --> 01:20:36.572 providers um that won't 01:20:36.612 --> 01:20:37.672 name because because I 01:20:37.693 --> 01:20:38.514 don't want everybody using 01:20:38.533 --> 01:20:39.635 them but we use some 01:20:39.694 --> 01:20:41.277 smaller providers in the us 01:20:41.317 --> 01:20:42.557 here we use some in europe 01:20:42.578 --> 01:20:43.698 we use some in asia pacific both 01:20:44.439 --> 01:20:46.501 in like North and Japan and 01:20:46.560 --> 01:20:48.182 also like Southeast Asia, like Singapore. 01:20:48.643 --> 01:20:49.703 And the reason for that is 01:20:49.743 --> 01:20:50.363 because we want to have 01:20:50.384 --> 01:20:52.105 diversity where if one of those goes down, 01:20:52.185 --> 01:20:53.106 like if Headstars goes down, 01:20:53.146 --> 01:20:54.427 which has happened before, 01:20:54.488 --> 01:20:55.448 they've had routing issues, 01:20:55.889 --> 01:20:56.729 you have a shitload of 01:20:56.770 --> 01:20:58.011 fucking VP that falls off 01:20:58.650 --> 01:21:00.453 and you can see who's running what. 01:21:00.912 --> 01:21:02.635 And also we want to have diversity where 01:21:03.255 --> 01:21:04.755 regardless of any issue if 01:21:04.814 --> 01:21:06.355 aws goes down or gcp or 01:21:06.395 --> 01:21:07.655 anything else that we're 01:21:07.796 --> 01:21:09.296 we're providing value right 01:21:09.315 --> 01:21:11.317 legal that's rcp rpc data 01:21:11.337 --> 01:21:12.337 or that's validator 01:21:12.356 --> 01:21:13.197 assigning or whatever else 01:21:13.237 --> 01:21:13.917 and so that's really 01:21:13.936 --> 01:21:16.037 important to us and so to 01:21:16.057 --> 01:21:17.358 that end one of the things 01:21:17.377 --> 01:21:17.997 that we're trying to find 01:21:18.037 --> 01:21:19.297 is another good provider in 01:21:19.358 --> 01:21:21.958 germany which which by far 01:21:22.019 --> 01:21:23.158 hessner and ovh are by far 01:21:23.198 --> 01:21:24.679 the largest I think and so 01:21:24.698 --> 01:21:25.099 we're trying to find 01:21:25.118 --> 01:21:25.918 somebody else that might 01:21:25.979 --> 01:21:28.220 have nodes within um 01:21:29.328 --> 01:21:31.189 Frankfurt or Hamburg or even 01:21:31.229 --> 01:21:32.789 Berlin that provide 01:21:33.689 --> 01:21:35.390 services in a distributed 01:21:35.430 --> 01:21:36.171 way that we like. 01:21:36.230 --> 01:21:38.011 And so we found this company, IP Projects. 01:21:38.631 --> 01:21:40.672 And specifically, specifically, 01:21:40.792 --> 01:21:43.193 seventy nine fifties at the 01:21:43.212 --> 01:21:44.873 time or Epic four thousands. 01:21:45.073 --> 01:21:45.293 Correct. 01:21:45.314 --> 01:21:45.694 Cause a lot, 01:21:45.753 --> 01:21:46.715 a lot of them also run like 01:21:46.835 --> 01:21:48.074 old Intel bullshit, right? 01:21:48.114 --> 01:21:49.416 Like they charge way too 01:21:49.456 --> 01:21:50.556 much for like this fucking. 01:21:50.577 --> 01:21:51.056 Twenty three, 01:21:51.157 --> 01:21:52.478 eighty eight fucking five 01:21:52.497 --> 01:21:54.498 hundred dollars a month. 01:21:54.559 --> 01:21:54.878 Exactly. 01:21:54.899 --> 01:21:55.519 There's some, 01:21:55.559 --> 01:21:57.140 some bullshit fucking dual 01:21:57.180 --> 01:21:58.600 CPU Xeon that doesn't, 01:21:59.240 --> 01:22:00.541 it has like really low 01:22:00.560 --> 01:22:01.462 clock speeds and it's 01:22:01.841 --> 01:22:03.323 fucking twenty seventeen hardware. 01:22:03.363 --> 01:22:04.582 Like nobody gives a shit about that. 01:22:04.863 --> 01:22:06.644 Like a seven forty. 01:22:07.185 --> 01:22:08.225 Plus we want like good data. 01:22:09.786 --> 01:22:10.466 Zero and gold. 01:22:10.485 --> 01:22:12.346 I want to know what data centers are in. 01:22:12.367 --> 01:22:13.547 We know what their pairing is. 01:22:13.587 --> 01:22:14.167 We want to know 01:22:14.887 --> 01:22:16.168 um what level data center 01:22:16.189 --> 01:22:16.988 that is I don't understand 01:22:17.009 --> 01:22:17.948 the support structure like 01:22:17.969 --> 01:22:18.729 there's a whole fucking 01:22:18.750 --> 01:22:19.409 checklist that we go 01:22:19.430 --> 01:22:21.369 through and ip projects was 01:22:21.430 --> 01:22:22.591 one of the ones that we we 01:22:22.610 --> 01:22:23.511 I kind of like because it 01:22:23.530 --> 01:22:25.051 was a smaller team good 01:22:25.091 --> 01:22:27.051 data centers um they were 01:22:27.072 --> 01:22:28.252 slow to provision things 01:22:28.292 --> 01:22:29.493 but doesn't I don't really 01:22:29.533 --> 01:22:30.453 care about that too much as 01:22:30.493 --> 01:22:31.514 long as like there's like 01:22:31.533 --> 01:22:32.474 they know what's going on 01:22:33.074 --> 01:22:33.935 and things like that and 01:22:33.954 --> 01:22:36.996 then both of us had issues 01:22:38.256 --> 01:22:39.096 that were a hundred percent 01:22:39.216 --> 01:22:39.837 exactly the same 01:22:41.457 --> 01:22:42.658 All of our issues have been the same. 01:22:42.698 --> 01:22:44.578 So it started off with the hard drives. 01:22:44.597 --> 01:22:45.878 They use consumer hard drives, 01:22:45.958 --> 01:22:46.759 which is bullshit. 01:22:46.800 --> 01:22:49.261 I don't really, honestly, 01:22:49.280 --> 01:22:50.100 I don't enjoy that, 01:22:50.280 --> 01:22:51.742 but if I know it's disposable, 01:22:51.782 --> 01:22:52.841 then I know what to put there. 01:22:53.682 --> 01:22:53.962 Yeah. 01:22:53.983 --> 01:22:55.182 But like, you know, you're, 01:22:55.262 --> 01:22:56.083 you're asking them to 01:22:56.144 --> 01:22:57.344 replace hard drives after 01:22:58.145 --> 01:23:00.605 three months because they 01:23:00.626 --> 01:23:02.067 just absolutely shot the shit, 01:23:02.386 --> 01:23:03.606 which they do not like doing. 01:23:04.688 --> 01:23:05.768 No, they do not. 01:23:05.967 --> 01:23:07.229 They're not doing anything. 01:23:08.786 --> 01:23:09.586 They don't like admitting 01:23:09.626 --> 01:23:10.787 they have issues and they 01:23:10.806 --> 01:23:11.667 don't like fixing them. 01:23:12.167 --> 01:23:14.149 We'll get to that part, yeah. 01:23:14.729 --> 01:23:16.510 So hard drive is always an issue there. 01:23:17.690 --> 01:23:18.731 And we were actually just 01:23:18.912 --> 01:23:20.813 starting to cancel servers 01:23:20.853 --> 01:23:21.993 because of the hard drives. 01:23:23.814 --> 01:23:25.296 Because they are like... 01:23:26.176 --> 01:23:27.717 They're not the cheapest. 01:23:27.757 --> 01:23:29.238 They are probably two X what 01:23:29.260 --> 01:23:32.143 you'd pay for a Hetzner box 01:23:32.162 --> 01:23:33.023 for the same thing. 01:23:33.042 --> 01:23:35.685 And I was like, well, Hetzner, 01:23:37.047 --> 01:23:38.048 while a lot of people use 01:23:38.088 --> 01:23:39.050 it and there's a little bit 01:23:39.069 --> 01:23:40.752 of stigma around using Hetzner, 01:23:41.011 --> 01:23:42.673 they're good to use for certain things. 01:23:42.694 --> 01:23:43.113 They're good to use. 01:23:43.654 --> 01:23:44.836 I use them all fucking day 01:23:44.876 --> 01:23:46.497 long for free RPC because 01:23:46.537 --> 01:23:47.497 people are just out there 01:23:47.518 --> 01:23:49.340 hammering that shit. 01:23:49.380 --> 01:23:50.440 I don't care about it too much. 01:23:50.461 --> 01:23:51.462 We use them for RPC as well, 01:23:51.481 --> 01:23:53.703 but I can't just rely on them. 01:23:53.764 --> 01:23:54.845 You can't just have them. 01:23:55.204 --> 01:23:57.667 You've got to have diversity 01:23:57.707 --> 01:23:58.787 for many different reasons. 01:23:58.929 --> 01:24:00.229 And one of those reasons, 01:24:00.270 --> 01:24:00.949 particularly if you're 01:24:00.970 --> 01:24:02.351 using these for validators, 01:24:02.431 --> 01:24:03.453 is you need to have your 01:24:03.493 --> 01:24:04.713 keys spread around as well. 01:24:05.494 --> 01:24:06.695 You can't just have all... 01:24:08.277 --> 01:24:09.637 You can't have your keys on 01:24:09.677 --> 01:24:12.899 a public server because if something, 01:24:12.998 --> 01:24:14.039 like you don't know a 01:24:14.060 --> 01:24:14.859 hundred percent what, 01:24:15.539 --> 01:24:16.119 you don't know if someone 01:24:16.140 --> 01:24:17.921 can just walk up to that 01:24:17.980 --> 01:24:19.042 server and plug into it. 01:24:19.101 --> 01:24:19.981 Like you don't know where it is. 01:24:20.341 --> 01:24:20.662 Spoiler alert. 01:24:21.622 --> 01:24:21.922 Yeah. 01:24:21.983 --> 01:24:23.444 And if you've, say if you've fucking, 01:24:23.564 --> 01:24:24.904 if you've left your, 01:24:25.824 --> 01:24:27.244 say you've used the IPMI, right. 01:24:27.265 --> 01:24:29.126 And you've stayed logged in. 01:24:29.246 --> 01:24:29.765 Don't do that. 01:24:29.886 --> 01:24:31.966 Always log out of your IPMI sessions. 01:24:32.967 --> 01:24:33.747 And then someone just goes 01:24:33.768 --> 01:24:34.729 and plugs into the box. 01:24:34.769 --> 01:24:37.170 Then they can just continue that session. 01:24:37.350 --> 01:24:37.510 Right. 01:24:38.109 --> 01:24:39.511 if they wanted if they 01:24:39.530 --> 01:24:41.492 wanted yeah because you 01:24:41.511 --> 01:24:42.452 don't know you don't know 01:24:42.471 --> 01:24:43.171 where the fuck it could be 01:24:43.212 --> 01:24:45.472 on someone's desk like it's 01:24:45.672 --> 01:24:46.833 some cloud shit somewhere 01:24:47.453 --> 01:24:49.395 it could be an installation 01:24:49.414 --> 01:24:50.676 in the middle of fucking 01:24:50.735 --> 01:24:52.636 pit street I don't know so 01:24:53.436 --> 01:24:55.118 so who the fuck knows so 01:24:55.177 --> 01:24:57.618 anyway um where was I going 01:24:57.639 --> 01:24:58.238 with that bullshit 01:25:00.612 --> 01:25:03.293 ransomware? 01:25:03.314 --> 01:25:04.073 Security keys. 01:25:04.354 --> 01:25:05.734 You need diversity around 01:25:05.774 --> 01:25:08.157 the place and key sharding 01:25:08.197 --> 01:25:09.796 and all that type of stuff. 01:25:10.257 --> 01:25:11.177 The moral of that story is 01:25:11.217 --> 01:25:11.878 don't leave anything 01:25:11.918 --> 01:25:12.939 important on a box that you 01:25:12.998 --> 01:25:14.279 rent publicly unless you 01:25:14.319 --> 01:25:16.640 have your own shit. 01:25:16.961 --> 01:25:17.822 If you do have keys, 01:25:17.881 --> 01:25:18.702 make sure they're sharded 01:25:18.721 --> 01:25:19.382 around the place 01:25:20.182 --> 01:25:22.185 um otherwise use specific 01:25:22.204 --> 01:25:23.286 key servers that you 01:25:23.326 --> 01:25:24.747 control somewhere or in 01:25:24.948 --> 01:25:25.929 something that you know has 01:25:25.948 --> 01:25:27.270 a lot better security like 01:25:27.609 --> 01:25:29.112 digital ocean or something 01:25:29.131 --> 01:25:30.792 that uses you know vms and 01:25:30.813 --> 01:25:32.494 that type of stuff um 01:25:34.055 --> 01:25:36.198 anyhow so not only the hard 01:25:36.217 --> 01:25:37.158 drives the hard drives was 01:25:37.198 --> 01:25:38.881 fucking annoying um but 01:25:40.221 --> 01:25:42.083 we had an issue where one of 01:25:42.145 --> 01:25:43.726 our boxes was ransomware. 01:25:44.006 --> 01:25:45.469 So this was an ongoing thing. 01:25:45.509 --> 01:25:48.011 We, we lost access to this box, right? 01:25:48.592 --> 01:25:51.476 Contacted the, you know, just disappeared. 01:25:51.617 --> 01:25:52.217 Stop reporting. 01:25:52.497 --> 01:25:53.880 Yeah. 01:25:53.979 --> 01:25:54.180 Right. 01:25:54.199 --> 01:25:54.961 Like what happened? 01:25:55.282 --> 01:25:55.702 Nobody knows. 01:25:55.742 --> 01:25:55.943 Yeah. 01:25:56.863 --> 01:25:58.925 So just disappeared. 01:25:59.625 --> 01:26:01.185 See you later. 01:26:01.225 --> 01:26:01.945 Stopped reporting. 01:26:01.966 --> 01:26:03.226 I'm just like, well, 01:26:03.247 --> 01:26:04.368 what the fuck happened to this thing? 01:26:05.328 --> 01:26:05.948 Obviously, 01:26:06.208 --> 01:26:08.590 it's everything that I know both 01:26:08.630 --> 01:26:10.791 SERP and iRun is redundant everywhere. 01:26:11.112 --> 01:26:12.112 So it didn't make a 01:26:12.171 --> 01:26:13.533 difference operationally. 01:26:13.733 --> 01:26:14.814 It was just fucking annoying. 01:26:16.534 --> 01:26:20.577 So into their web platform, onto the IPMI, 01:26:20.856 --> 01:26:22.118 can't log into the IPMI. 01:26:22.719 --> 01:26:23.840 okay, that's weird. 01:26:24.421 --> 01:26:26.023 So, um, you know, 01:26:26.082 --> 01:26:27.543 is there a motherboard issue? 01:26:27.644 --> 01:26:29.225 Is, is there a power issue? 01:26:29.286 --> 01:26:30.967 Like what's the deal that 01:26:31.268 --> 01:26:32.390 the screen will come up, 01:26:32.409 --> 01:26:33.390 but you can't log in. 01:26:33.711 --> 01:26:35.172 I haven't changed any credentials. 01:26:35.252 --> 01:26:36.134 So what's going on? 01:26:36.634 --> 01:26:38.195 Contact the, um, 01:26:39.750 --> 01:26:40.649 you know, the, 01:26:40.689 --> 01:26:42.850 the help team and they're like, Oh yeah, 01:26:42.890 --> 01:26:44.650 sometimes the motherboards just fuck out. 01:26:44.690 --> 01:26:45.831 So we'll replace a 01:26:45.871 --> 01:26:46.572 motherboard for you 01:26:46.931 --> 01:26:48.072 replaces the motherboard 01:26:48.112 --> 01:26:50.192 still can't log into SSH, but the, 01:26:50.212 --> 01:26:50.872 you know, you know, the, 01:26:51.092 --> 01:26:53.252 the machine's coming up, um, 01:26:53.672 --> 01:26:55.793 because you can, 01:26:56.194 --> 01:26:58.474 you can see the IPMI comes up. 01:26:58.734 --> 01:27:00.095 So the management is working, 01:27:00.854 --> 01:27:01.935 can't log into it, but you know, 01:27:01.975 --> 01:27:02.335 it's there. 01:27:02.475 --> 01:27:03.655 So, you know, the motherboard's alive. 01:27:04.275 --> 01:27:05.735 Um, and, um, 01:27:07.904 --> 01:27:09.244 in this case they changed 01:27:09.265 --> 01:27:11.126 the motherboard then reset 01:27:11.145 --> 01:27:13.228 the ipmi password then I 01:27:13.247 --> 01:27:15.069 could log in to the ipmi 01:27:15.088 --> 01:27:16.310 right so I could use like 01:27:16.390 --> 01:27:18.211 the remote session over 01:27:18.310 --> 01:27:21.453 ipmi like the kvm um which 01:27:21.493 --> 01:27:22.774 is just it's like a 01:27:22.814 --> 01:27:24.074 keyboard and mouse in a 01:27:24.354 --> 01:27:25.576 monitor remotely right 01:27:25.957 --> 01:27:27.056 directly so you're not over 01:27:27.197 --> 01:27:28.438 ssh or anything still 01:27:28.457 --> 01:27:29.439 couldn't ssh into the 01:27:29.479 --> 01:27:30.260 system which I thought was 01:27:30.279 --> 01:27:32.100 a little bit weird found out why 01:27:33.402 --> 01:27:36.967 log into the bloody, um, the KVM session, 01:27:37.509 --> 01:27:38.390 uh, 01:27:38.430 --> 01:27:41.234 over the IPMI fucking ransomware 01:27:41.295 --> 01:27:41.675 everywhere. 01:27:41.815 --> 01:27:46.483 So I raised this with, with the, um, 01:27:48.220 --> 01:27:49.641 through the ticket system with their help. 01:27:50.082 --> 01:27:51.122 And they say, not us. 01:27:52.524 --> 01:27:55.265 And I'm like, well, 01:27:55.286 --> 01:27:56.587 you're not even entertain 01:27:56.606 --> 01:27:58.688 the idea that you may have 01:27:58.728 --> 01:27:59.429 an issue somewhere, right? 01:27:59.488 --> 01:28:00.869 They're like, no, it's not us. 01:28:00.890 --> 01:28:01.250 It's you. 01:28:01.529 --> 01:28:01.750 I'm like, 01:28:02.149 --> 01:28:04.271 we've got our SSH fully locked down. 01:28:04.572 --> 01:28:06.372 The only way that there 01:28:06.393 --> 01:28:08.795 could be like a breach 01:28:09.055 --> 01:28:10.576 through SSH for us is a 01:28:10.655 --> 01:28:12.957 zero day attack on SSH. 01:28:13.457 --> 01:28:15.139 No one is using that shit on 01:28:15.219 --> 01:28:17.421 fucking a King nodes node, right? 01:28:19.542 --> 01:28:21.404 With no value on it whatsoever. 01:28:22.086 --> 01:28:26.430 So anyway, I'm like, you know, 01:28:26.449 --> 01:28:29.012 can someone gain access to the system? 01:28:29.052 --> 01:28:30.335 Like maybe I didn't log out 01:28:30.395 --> 01:28:32.737 of the session for some reason. 01:28:32.756 --> 01:28:33.637 Like I don't remember ever 01:28:33.757 --> 01:28:36.440 accessing over KVM previously. 01:28:36.782 --> 01:28:36.962 Right. 01:28:37.261 --> 01:28:38.182 And I was like, you know, 01:28:38.222 --> 01:28:39.823 could someone gain physical 01:28:39.923 --> 01:28:41.845 access and plug into it and then do this? 01:28:42.086 --> 01:28:43.306 And they're like, absolutely not. 01:28:43.806 --> 01:28:45.148 Started running off, 01:28:45.608 --> 01:28:46.809 running off stats of like 01:28:46.849 --> 01:28:48.270 forty five armed guards and 01:28:48.310 --> 01:28:49.171 fucking all the rest of the 01:28:49.190 --> 01:28:50.471 bullshit that they were going on with. 01:28:51.171 --> 01:28:54.595 And then I'm like, OK, so like, you know, 01:28:54.694 --> 01:28:55.836 is there maybe there's a 01:28:55.895 --> 01:28:57.417 vulnerability in the system 01:28:57.577 --> 01:28:58.797 where someone can get in 01:28:58.898 --> 01:29:01.760 through the through the. 01:29:03.445 --> 01:29:06.507 ipmi or you know this type 01:29:06.528 --> 01:29:07.368 of thing and they're like 01:29:07.387 --> 01:29:09.009 no absolutely not you got 01:29:09.168 --> 01:29:10.548 you got brute forced I'm 01:29:10.588 --> 01:29:12.689 like you can't even log 01:29:12.770 --> 01:29:14.030 into my system with a 01:29:14.091 --> 01:29:16.532 fucking password like it 01:29:16.551 --> 01:29:17.792 doesn't work that way like 01:29:17.832 --> 01:29:19.854 it's locked down brute 01:29:19.873 --> 01:29:20.953 force a four thousand 01:29:20.993 --> 01:29:22.435 ninety six bit key over and over 01:29:23.234 --> 01:29:25.817 yeah yeah I'm like it's 01:29:25.837 --> 01:29:26.557 fucking okay dude you know 01:29:26.578 --> 01:29:28.158 thanks yeah it's not me 01:29:28.198 --> 01:29:29.659 it's you and they're like 01:29:30.000 --> 01:29:31.001 no it's not and I'm like 01:29:31.060 --> 01:29:32.421 well okay you canceled that 01:29:32.462 --> 01:29:33.842 server I didn't even bother 01:29:33.863 --> 01:29:34.663 trying to recover I just 01:29:34.684 --> 01:29:35.703 started canceling fucking 01:29:35.724 --> 01:29:38.907 service yeah why and then um oh go ahead 01:29:39.685 --> 01:29:40.587 They went one by one. 01:29:40.726 --> 01:29:45.711 Eventually, all of my servers were gone. 01:29:46.532 --> 01:29:48.533 And I can only assume that 01:29:48.573 --> 01:29:50.414 they were all fucking ransomware. 01:29:50.635 --> 01:29:52.377 So I just turned them all 01:29:52.436 --> 01:29:54.738 off and fucking deleted them all. 01:29:54.819 --> 01:29:56.039 Tried to get a refund for the month. 01:29:56.060 --> 01:29:57.041 And they're like, 01:29:57.060 --> 01:29:58.061 we're not responsible for 01:29:58.101 --> 01:30:01.083 fucking zero day attacks. 01:30:01.404 --> 01:30:03.345 So they sent me an email. 01:30:04.226 --> 01:30:06.288 They're like, urgent, urgent. 01:30:06.588 --> 01:30:07.750 When did you get this today? 01:30:08.345 --> 01:30:09.664 Like yesterday or the day before? 01:30:09.725 --> 01:30:11.865 You forward that to me. 01:30:11.886 --> 01:30:13.445 Urgent. 01:30:13.985 --> 01:30:15.886 We need your IPMI password 01:30:15.926 --> 01:30:17.646 so that we can change your 01:30:17.667 --> 01:30:19.667 configuration so you're behind a proxy. 01:30:20.547 --> 01:30:22.207 And I'm like, dude. 01:30:22.247 --> 01:30:23.467 I thought it was behind a proxy. 01:30:24.148 --> 01:30:25.069 And they're like, 01:30:25.149 --> 01:30:26.189 what have you changed it to? 01:30:26.269 --> 01:30:27.969 And I'm like, like I said before, 01:30:28.029 --> 01:30:28.809 I've never changed a 01:30:28.929 --> 01:30:32.289 fucking password at all on IPMI. 01:30:32.310 --> 01:30:33.210 Your shit is owned. 01:30:33.470 --> 01:30:34.430 Your shit's fucked. 01:30:35.411 --> 01:30:36.171 You sort it out. 01:30:38.310 --> 01:30:39.072 So anyway, I'm like, 01:30:39.112 --> 01:30:39.893 you should give me a 01:30:39.953 --> 01:30:42.454 fucking refund because I 01:30:42.475 --> 01:30:43.256 haven't had any of my 01:30:43.475 --> 01:30:44.858 servers for a month from 01:30:44.917 --> 01:30:45.899 something you refused to 01:30:45.939 --> 01:30:46.859 fucking investigate. 01:30:49.141 --> 01:30:50.542 Is this an actual ASRock 01:30:50.582 --> 01:30:52.404 issue or this is an IP projects issue? 01:30:52.886 --> 01:30:55.127 They're saying that it's a 01:30:55.448 --> 01:30:58.091 zero day on the ASRock IPMI. 01:30:58.996 --> 01:31:01.577 Okay, but how would that be in the clear? 01:31:02.278 --> 01:31:03.238 That should not be in the clear. 01:31:03.377 --> 01:31:04.099 There's no reason for that 01:31:04.118 --> 01:31:04.658 to be in the clear. 01:31:04.958 --> 01:31:06.560 I don't know because they 01:31:06.619 --> 01:31:07.961 kept telling me that their 01:31:08.001 --> 01:31:09.381 fucking IPMI is behind a 01:31:09.461 --> 01:31:10.622 proxy and you have to go in 01:31:10.641 --> 01:31:12.143 and click the button on the website. 01:31:12.323 --> 01:31:13.583 Or they have one client that 01:31:13.623 --> 01:31:14.944 somehow gets into the proxy 01:31:15.043 --> 01:31:15.885 and then from there they 01:31:15.904 --> 01:31:16.625 can get to every other 01:31:16.664 --> 01:31:18.025 fucking IPMI IP address. 01:31:18.286 --> 01:31:19.127 Well, I don't know, 01:31:19.167 --> 01:31:20.889 because their solution that 01:31:20.929 --> 01:31:22.451 they were telling me was, oh, 01:31:22.470 --> 01:31:24.453 we've just put fucking, in one day, 01:31:24.472 --> 01:31:25.434 we've put three hundred 01:31:25.673 --> 01:31:27.076 IPMIs behind the proxy. 01:31:27.095 --> 01:31:29.217 I'm like, what the fuck was that before? 01:31:29.297 --> 01:31:30.479 Yeah. 01:31:30.538 --> 01:31:32.140 So you just opened the internet before? 01:31:32.641 --> 01:31:33.282 We only had three boxes. 01:31:33.301 --> 01:31:34.423 I wonder how that got hacked. 01:31:34.804 --> 01:31:36.024 We have three boxes there. 01:31:36.164 --> 01:31:36.706 One of those, 01:31:36.985 --> 01:31:37.707 exactly the same thing 01:31:37.726 --> 01:31:39.007 happened where it just went offline. 01:31:39.047 --> 01:31:39.868 We were arguing with them. 01:31:39.969 --> 01:31:41.189 I never got back into the box. 01:31:41.229 --> 01:31:42.189 They wanted to charge me 01:31:42.250 --> 01:31:43.631 sixty dollars over the 01:31:43.671 --> 01:31:46.554 weekend to reset the IP and my password. 01:31:46.573 --> 01:31:47.914 They sent me the same email. 01:31:47.935 --> 01:31:48.855 They're like, oh, yeah, 01:31:48.876 --> 01:31:50.237 there's a there's a fucking 01:31:50.256 --> 01:31:51.537 there's a there's a fee for 01:31:51.578 --> 01:31:52.078 like coming out. 01:31:52.099 --> 01:31:52.800 I'm like, all right, 01:31:52.819 --> 01:31:53.980 just do it when you get in the office. 01:31:54.661 --> 01:31:55.122 Yeah. 01:31:55.162 --> 01:31:55.962 So I told him to go fuck 01:31:55.981 --> 01:31:57.203 themselves and I canceled the box. 01:31:57.243 --> 01:31:58.984 And that's when Michael yelled at me. 01:31:59.005 --> 01:31:59.945 The CEO emailed me over 01:32:00.046 --> 01:32:00.786 email and all this kind of 01:32:00.805 --> 01:32:02.488 shit because blah, blah, blah. 01:32:03.027 --> 01:32:04.448 And then it happened to a second box, 01:32:04.467 --> 01:32:05.809 which is also just an RPC server. 01:32:05.828 --> 01:32:06.590 It wasn't the end of the world. 01:32:06.609 --> 01:32:07.211 I don't really care. 01:32:07.230 --> 01:32:08.511 There's nothing on there. 01:32:08.551 --> 01:32:10.533 That box, I was like, hey, 01:32:10.613 --> 01:32:12.015 this is during business hours. 01:32:12.475 --> 01:32:13.595 Like, I can't get into this thing. 01:32:13.695 --> 01:32:15.096 I never touch the IP of my password. 01:32:15.377 --> 01:32:15.757 There is no, 01:32:15.957 --> 01:32:17.658 we have no open SH ports at all. 01:32:17.739 --> 01:32:18.779 There's nothing open on our, 01:32:18.860 --> 01:32:19.541 the only thing that we have 01:32:19.600 --> 01:32:21.323 open on our nodes is P to P ports. 01:32:21.483 --> 01:32:22.604 Everything else is locked. 01:32:22.804 --> 01:32:23.363 There's nothing. 01:32:23.463 --> 01:32:25.006 Even RPC ports, those are, 01:32:25.065 --> 01:32:25.926 those are locked on IP 01:32:25.947 --> 01:32:27.067 addresses only to Cloudflare, 01:32:27.087 --> 01:32:27.627 other types of things. 01:32:27.988 --> 01:32:28.529 The only thing you can 01:32:28.590 --> 01:32:29.873 actually get to is a PDP port. 01:32:30.194 --> 01:32:31.679 So I'm guessing that's not the issue. 01:32:32.180 --> 01:32:33.082 Everything else is locked down. 01:32:33.101 --> 01:32:34.104 There's nothing you can get into. 01:32:35.037 --> 01:32:36.618 um so so I'm like it's 01:32:36.658 --> 01:32:37.939 impossible like you know 01:32:38.060 --> 01:32:39.520 blah blah so they want to 01:32:39.541 --> 01:32:40.581 charge me fifteen dollars 01:32:41.322 --> 01:32:42.181 or fifteen euros fifteen 01:32:42.202 --> 01:32:42.862 dollars I don't know what 01:32:42.881 --> 01:32:43.842 it was to reset that 01:32:43.863 --> 01:32:45.944 password I said do it I'll 01:32:45.984 --> 01:32:47.604 pay it put it in there add 01:32:47.625 --> 01:32:48.685 it to my bill do it right 01:32:48.725 --> 01:32:50.006 now so the guy went and 01:32:50.025 --> 01:32:51.027 reset the password sent it 01:32:51.046 --> 01:32:51.967 to me I got in there I said 01:32:51.988 --> 01:32:52.847 like a screenshot sent it 01:32:52.868 --> 01:32:53.988 back to him and said this 01:32:54.128 --> 01:32:55.829 thing's ransomware what the 01:32:55.930 --> 01:32:57.291 are you doing and they said 01:32:58.310 --> 01:32:58.390 you, 01:32:58.411 --> 01:32:59.792 I don't know what the hell they even 01:32:59.832 --> 01:33:00.252 replied to me. 01:33:00.332 --> 01:33:02.654 Like, like, like I have no open SSH ports. 01:33:03.314 --> 01:33:04.375 They never replied to me again. 01:33:04.414 --> 01:33:04.716 Actually. 01:33:04.735 --> 01:33:05.735 That's what they never replied. 01:33:05.917 --> 01:33:06.277 That was it. 01:33:06.317 --> 01:33:06.896 It was quiet. 01:33:07.037 --> 01:33:09.219 Like, so I, I didn't cancel that box yet. 01:33:09.279 --> 01:33:09.838 It's still running. 01:33:09.859 --> 01:33:11.000 Cause I have until the first, 01:33:11.060 --> 01:33:12.180 cause I'm waiting for the third one. 01:33:12.537 --> 01:33:13.016 The third one, 01:33:13.136 --> 01:33:15.158 I proactively went into the IPMI. 01:33:15.978 --> 01:33:17.057 I changed that password, 01:33:17.097 --> 01:33:18.259 which is in their portal 01:33:18.618 --> 01:33:19.658 from a seven character 01:33:19.698 --> 01:33:21.260 password to a thirty two 01:33:21.300 --> 01:33:24.461 character password and or more. 01:33:24.480 --> 01:33:25.301 I don't know what the hell it is. 01:33:25.680 --> 01:33:26.060 I said it. 01:33:26.201 --> 01:33:27.261 I just reset that IPMI 01:33:27.301 --> 01:33:28.021 password and nothing's 01:33:28.042 --> 01:33:28.702 happened with that box yet 01:33:28.721 --> 01:33:29.962 because I assume that 01:33:29.983 --> 01:33:31.042 they're brute forcing it. 01:33:31.403 --> 01:33:32.063 That's what I assumed. 01:33:32.182 --> 01:33:32.884 I assume they're brute 01:33:32.923 --> 01:33:34.583 forcing IPMI passwords is what it was. 01:33:34.604 --> 01:33:35.904 What my guess was after two 01:33:35.925 --> 01:33:36.885 of these fucking things happened. 01:33:37.913 --> 01:33:38.434 Um, 01:33:38.453 --> 01:33:39.715 so I just locked that down unless 01:33:39.734 --> 01:33:40.215 there's another, 01:33:40.534 --> 01:33:41.274 but there's also another 01:33:41.395 --> 01:33:42.295 admin box that they, 01:33:42.375 --> 01:33:43.655 or another login that they 01:33:43.756 --> 01:33:44.395 use internally. 01:33:44.796 --> 01:33:45.836 So they might be, who knows? 01:33:45.876 --> 01:33:48.077 It might be fucking IPP one, two, 01:33:48.096 --> 01:33:49.778 three for like fucking know 01:33:49.797 --> 01:33:50.478 their password is. 01:33:51.118 --> 01:33:52.537 So, um, 01:33:52.557 --> 01:33:53.998 so that one is also not that important. 01:33:54.019 --> 01:33:55.219 None of these are important to me, 01:33:55.538 --> 01:33:57.079 but it's still fucking annoying. 01:33:57.100 --> 01:33:58.319 It's fucking annoying. 01:33:58.699 --> 01:33:59.560 So I'm waiting for some new 01:33:59.581 --> 01:34:00.360 boxes to come up cause I'm 01:34:00.381 --> 01:34:01.560 moving this shit off and 01:34:01.581 --> 01:34:02.121 then I'm going to go, 01:34:02.220 --> 01:34:02.921 I'm gonna do the same thing. 01:34:02.962 --> 01:34:03.681 I'm gonna go after him and say, 01:34:03.702 --> 01:34:04.101 what the hell? 01:34:04.981 --> 01:34:05.962 I'm completely done I've 01:34:06.122 --> 01:34:07.103 cancelled every single one 01:34:07.222 --> 01:34:07.882 just turned them off well 01:34:07.903 --> 01:34:08.703 they haven't they haven't 01:34:08.743 --> 01:34:09.582 come back to me and said 01:34:09.622 --> 01:34:10.743 like oh by the way there's 01:34:10.764 --> 01:34:12.104 a like that's the other 01:34:12.123 --> 01:34:12.783 thing is like I haven't 01:34:12.804 --> 01:34:14.404 heard jack fucking shit 01:34:14.685 --> 01:34:15.824 about them oh yeah by the 01:34:15.845 --> 01:34:17.426 way there's a zero day here 01:34:17.466 --> 01:34:19.506 and this is our issue like 01:34:21.006 --> 01:34:22.186 I'll read you the email 01:34:22.226 --> 01:34:23.487 zero zero so fucking max 01:34:23.587 --> 01:34:32.010 you cock what is tell me 01:34:32.029 --> 01:34:32.590 what the email says 01:34:35.323 --> 01:34:35.904 All right. 01:34:35.925 --> 01:34:36.586 So they sent me this email. 01:34:36.645 --> 01:34:37.988 There's currently a critical 01:34:38.368 --> 01:34:38.929 vulnerability.