WEBVTT 00:00:00.000 --> 00:00:03.459 AWS can now reset failed Gateway Load Balancer 00:00:03.459 --> 00:00:06.339 connections instead of letting them hang for 00:00:06.339 --> 00:00:10.000 minutes. GitHub is getting ready to stop running 00:00:10.000 --> 00:00:14.019 jobs on stale self-hosted Actions runners. And 00:00:14.019 --> 00:00:17.760 a Linux desktop setup accidentally gave basically 00:00:17.760 --> 00:00:21.859 every user-space process a path to root through 00:00:21.859 --> 00:00:24.920 Docker. This week is mostly about failure modes 00:00:24.920 --> 00:00:28.359 hiding inside things we already trust. I'm Brian 00:00:28.359 --> 00:00:30.820 Teller from Teller's Tech, and this is Ship It 00:00:30.820 --> 00:00:50.380 Weekly. Welcome back to Ship It Weekly, the show 00:00:50.380 --> 00:00:54.700 about the DevOps, SRE, cloud, platform, and security 00:00:54.700 --> 00:00:57.259 stories that matter when you are the person keeping 00:00:57.259 --> 00:00:59.939 the thing running at three in the morning. For 00:00:59.939 --> 00:01:02.820 the weekly story list and source links, check 00:01:02.820 --> 00:01:06.659 out OnCallBrief.com. For past episodes and show 00:01:06.659 --> 00:01:10.379 notes, head over to ShipItWeekly.fm. This week, 00:01:10.439 --> 00:01:15.069 AWS Gateway Load Balancer gets TCP Reset to 00:01:15.069 --> 00:01:18.329 shorten recovery when a firewall appliance dies. 00:01:18.329 --> 00:01:22.590 Azure DevOps gets live migration tooling for 00:01:22.590 --> 00:01:26.670 moving enterprises to GitHub. GitHub is starting 00:01:26.670 --> 00:01:30.170 enforcement against stale self-hosted actions 00:01:30.170 --> 00:01:34.530 runners. And Omarchy shipped with a Docker configuration 00:01:34.530 --> 00:01:38.189 that effectively made normal desktop processes 00:01:38.189 --> 00:01:42.069 root-capable. Then we have a quick lightning round 00:01:42.069 --> 00:01:45.230 and a human closer about whether incident reviews 00:01:45.230 --> 00:01:49.030 should happen asynchronously first, live later. 00:01:49.230 --> 00:01:57.849 Let's get into it. First up, AWS Gateway Load 00:01:57.849 --> 00:02:02.030 Balancer now supports TCP Reset. This is one 00:02:02.030 --> 00:02:04.829 of those small-looking features that solves a 00:02:04.829 --> 00:02:08.349 pretty ugly failure mode. Gateway Load Balancer 00:02:08.349 --> 00:02:11.889 is commonly used to put firewalls, intrusion 00:02:11.889 --> 00:02:14.909 detection systems, and other network appliances 00:02:14.909 --> 00:02:18.830 inline with application traffic. When one of 00:02:18.830 --> 00:02:22.129 those appliances fail, new connections can usually 00:02:22.129 --> 00:02:26.009 move to a healthy target. Existing TCP connections 00:02:26.009 --> 00:02:29.550 are harder. They may keep trying to use the failed 00:02:29.550 --> 00:02:32.969 path until the connection times out. Depending 00:02:32.969 --> 00:02:36.610 on the application and TCP settings, that can 00:02:36.610 --> 00:02:40.349 mean 30 seconds. It can also mean several minutes. 00:02:40.710 --> 00:02:43.810 From the application's point of view, the network 00:02:43.810 --> 00:02:46.990 is not obviously broken. It is just hanging. 00:02:47.349 --> 00:02:51.270 With TCP Reset enabled, Gateway Load Balancer 00:02:51.270 --> 00:02:54.389 can actively terminate those affected connections. 00:02:54.949 --> 00:02:58.930 The client gets a reset immediately and can reconnect 00:02:58.930 --> 00:03:02.689 through a healthy appliance. That turns an ambiguous 00:03:02.689 --> 00:03:06.590 failure into an explicit one. And I think that 00:03:06.590 --> 00:03:10.289 is the useful lesson here. Fast failure is often 00:03:10.289 --> 00:03:13.789 better than slow uncertainty. A failed request 00:03:13.789 --> 00:03:17.289 gives the application something that it can react 00:03:17.289 --> 00:03:19.990 to. A connection sitting there doing nothing 00:03:19.990 --> 00:03:25.090 is much harder. Retries do not happen yet. Timeout 00:03:25.090 --> 00:03:29.289 budgets get consumed. Users wait. Threads stay 00:03:29.289 --> 00:03:33.349 blocked. And everything upstream looks degraded 00:03:33.349 --> 00:03:37.169 instead of clearly failed. We talk a lot about 00:03:37.169 --> 00:03:40.629 graceful failover, but sometimes graceful means 00:03:40.629 --> 00:03:45.110 being very direct. This path is dead. Start over 00:03:45.110 --> 00:03:48.009 somewhere healthy. This is especially important 00:03:48.009 --> 00:03:51.770 around stateful middleboxes because the backend 00:03:51.770 --> 00:03:55.569 application may be perfectly fine while the network 00:03:55.569 --> 00:03:58.680 path in front of it is not. The shorter you can 00:03:58.680 --> 00:04:01.939 make that ambiguity window, the faster the rest 00:04:01.939 --> 00:04:10.039 of the system can recover. Next, Microsoft has 00:04:10.039 --> 00:04:13.000 put Enterprise Live Migrations from Azure DevOps 00:04:13.000 --> 00:04:16.819 to GitHub into public preview. This is aimed 00:04:16.819 --> 00:04:20.100 at large migrations where shutting down engineering 00:04:20.100 --> 00:04:23.980 for a weekend is not really an option. The idea 00:04:23.980 --> 00:04:27.000 is that repositories remain writable in Azure 00:04:27.000 --> 00:04:30.199 DevOps while changes continuously synchronize 00:04:30.199 --> 00:04:33.980 into GitHub Enterprise Cloud. Teams keep working 00:04:33.980 --> 00:04:37.139 during most of the migration. Then, when it is 00:04:37.139 --> 00:04:39.879 time for the final cutover, you stop writes, 00:04:40.139 --> 00:04:43.000 sync the remaining changes, and switch over. 00:04:43.240 --> 00:04:46.319 Microsoft says the final downtime can typically 00:04:46.319 --> 00:04:50.240 stay under about 30 minutes. The migration tooling 00:04:50.240 --> 00:04:53.560 can also convert Azure DevOps branch policies 00:04:53.560 --> 00:04:57.680 into GitHub rulesets. And importantly, Azure 00:04:57.680 --> 00:05:01.399 pipelines can continue running while repositories 00:05:01.399 --> 00:05:04.879 move. That matters because repository migration 00:05:04.879 --> 00:05:09.579 is rarely just copying Git objects. The hard 00:05:09.579 --> 00:05:12.620 part is everything attached to the repository. 00:05:13.000 --> 00:05:18.660 Permissions, branch protections, CI/CD. webhooks, 00:05:18.660 --> 00:05:23.839 secrets, bots, service connections, release workflows, 00:05:24.240 --> 00:05:27.319 and all of the little assumptions around URLs 00:05:27.319 --> 00:05:31.319 and identities that accumulate over 10 years. 00:05:31.759 --> 00:05:35.339 A migration can be technically successful and 00:05:35.339 --> 00:05:38.600 still be operationally terrible if engineers 00:05:38.600 --> 00:05:42.139 lose half a day every time you move another batch 00:05:42.139 --> 00:05:46.000 of repositories. Live synchronization changes 00:05:46.000 --> 00:05:49.660 that equation. It lets the migration behave more 00:05:49.660 --> 00:05:52.839 like a controlled transition instead of a hard 00:05:52.839 --> 00:05:56.839 stop. But I would still treat this like any other 00:05:56.839 --> 00:05:59.939 production migration. Inventory what depends 00:05:59.939 --> 00:06:03.680 on Azure DevOps before you move anything. Test 00:06:03.680 --> 00:06:07.660 policy conversions. Validate pipelines. Know 00:06:07.660 --> 00:06:10.759 what happens to automation that references old 00:06:10.759 --> 00:06:15.120 URLs. And have a rollback path for the cutover 00:06:15.120 --> 00:06:18.470 itself. The goal is not just moving the code. 00:06:18.769 --> 00:06:22.250 The goal is moving the development system around 00:06:22.250 --> 00:06:25.230 the code without everybody discovering the missing 00:06:25.230 --> 00:06:33.730 pieces on Monday morning. Third, GitHub is getting 00:06:33.730 --> 00:06:36.730 serious about stale self-hosted Actions runners. 00:06:37.089 --> 00:06:40.129 This one is worth checking if you manage your 00:06:40.129 --> 00:06:43.649 own runner fleet. GitHub is rolling out minimum 00:06:43.649 --> 00:06:47.050 version enforcement. Self -hosted runners need 00:06:47.050 --> 00:06:51.610 to be at least version 2.329.0 to register. 00:06:51.910 --> 00:06:55.370 And runners also need to stay within 30 days 00:06:55.370 --> 00:06:58.649 of current releases to continue executing jobs. 00:06:59.129 --> 00:07:02.509 GitHub is starting runtime brownouts on September 00:07:02.509 --> 00:07:05.949 9th. Full enforcement is planned for September 00:07:05.949 --> 00:07:10.269 25th. The obvious action is to update your runners. 00:07:10.839 --> 00:07:13.480 The more interesting question is why they are 00:07:13.480 --> 00:07:16.660 stale in the first place. A lot of teams treat 00:07:16.660 --> 00:07:19.500 self-hosted runners like infrastructure appliances. 00:07:20.160 --> 00:07:24.060 Build the image. Deploy the runner. Forget about 00:07:24.060 --> 00:07:27.540 it until something breaks. But the runner is 00:07:27.540 --> 00:07:31.000 executing arbitrary CI workloads with access 00:07:31.000 --> 00:07:35.259 to repositories, credentials, networks, artifact 00:07:35.259 --> 00:07:39.949 stores, and often cloud permissions. It is part 00:07:39.949 --> 00:07:43.449 of your security boundary. It is also a client 00:07:43.449 --> 00:07:47.189 of GitHub's service. And that client has to evolve 00:07:47.189 --> 00:07:51.350 with the platform. If your upgrade process requires 00:07:51.350 --> 00:07:54.810 somebody remembering to rebuild an AMI every 00:07:54.810 --> 00:07:58.110 few months, enforcement like this is eventually 00:07:58.110 --> 00:08:01.350 going to catch you. This is where ephemeral runners 00:08:01.350 --> 00:08:05.089 are really attractive. Start from a maintained 00:08:05.089 --> 00:08:08.649 image. Run the job. Destroy the environment. 00:08:09.129 --> 00:08:13.110 Regularly rebuild the base image and roll forward 00:08:13.110 --> 00:08:16.509 automatically. That does not eliminate maintenance. 00:08:16.850 --> 00:08:20.310 It makes maintenance part of the lifecycle instead 00:08:20.310 --> 00:08:23.810 of an exception to it. And if you cannot automatically 00:08:23.810 --> 00:08:27.449 replace a runner fleet today, this brownout is 00:08:27.449 --> 00:08:31.050 probably a good excuse to figure out why. Because 00:08:31.050 --> 00:08:34.649 September 25th is a much nicer time to discover 00:08:34.649 --> 00:08:37.470 that problem than during a production deployment. 00:08:42.200 --> 00:08:46.379 Fourth, Omarchy fixed a configuration that effectively 00:08:46.379 --> 00:08:50.539 gave user-space applications a path to root through 00:08:50.539 --> 00:08:54.039 Docker. The default user was part of the Docker 00:08:54.039 --> 00:08:57.519 group. That is common on Linux systems because 00:08:57.519 --> 00:09:00.000 it lets you run Docker commands without sudo. 00:09:00.159 --> 00:09:03.679 It is also basically root access. The Docker 00:09:03.679 --> 00:09:07.139 daemon runs as root. If your user can talk to 00:09:07.139 --> 00:09:10.379 the Docker socket, you can generally start a 00:09:10.379 --> 00:09:13.220 privileged container, mount the host filesystem, 00:09:13.460 --> 00:09:17.220 and modify the machine. So anything running as 00:09:17.220 --> 00:09:20.399 that user inherits a very powerful capability, 00:09:20.799 --> 00:09:25.379 a browser extension, an IDE plugin, a compromised 00:09:25.379 --> 00:09:29.220 development tool, a coding agent, malware running 00:09:29.220 --> 00:09:32.500 in the session. If it can reach the Docker socket, 00:09:32.840 --> 00:09:37.000 the jump from normal user to root may be trivial. 00:09:37.600 --> 00:09:42.259 Omarchy fixed this in version 4.0.1. But the 00:09:42.259 --> 00:09:44.879 bigger point is not really about Omarchy. The 00:09:44.879 --> 00:09:47.679 Docker group has been a known privilege boundary 00:09:47.679 --> 00:09:51.860 for a long time. What has changed is how much 00:09:51.860 --> 00:09:54.639 software we now run inside developer sessions. 00:09:55.259 --> 00:09:59.840 IDE extensions, package managers, AI coding tools, 00:10:00.240 --> 00:10:04.539 local agents, browser automation, dev containers, 00:10:05.019 --> 00:10:09.460 CLI plugins. A modern workstation is a pretty 00:10:09.460 --> 00:10:13.000 busy environment. Giving every one of those processes 00:10:13.000 --> 00:10:16.600 indirect root access because Docker is more convenient 00:10:16.600 --> 00:10:20.360 without sudo is worth thinking about. Convenience 00:10:20.360 --> 00:10:23.840 is fine. Just name the privilege correctly. If 00:10:23.840 --> 00:10:26.620 membership in a group means the user can become 00:10:26.620 --> 00:10:30.519 root, treat that group like root access. Because 00:10:30.519 --> 00:10:41.519 an attacker absolutely will. Quick lightning 00:10:41.519 --> 00:10:46.120 round. First, AWS Lambda now supports fuller 00:10:46.120 --> 00:10:50.159 IAM resource-based policies. You can define 00:10:50.159 --> 00:10:54.120 multiple principals and actions and use normal 00:10:54.120 --> 00:10:57.759 IAM condition keys for cross-account access. 00:10:58.259 --> 00:11:00.720 That should make larger Lambda authorization 00:11:00.720 --> 00:11:04.899 models cleaner than some of the older one permission 00:11:04.899 --> 00:11:09.320 at a time workflows. Second, AWS is warning about 00:11:09.320 --> 00:11:12.840 PostgreSQL upgrades that can fail because of 00:11:12.840 --> 00:11:16.919 circular role memberships. RDS and Aurora upgrades 00:11:16.919 --> 00:11:21.799 from PostgreSQL 14 or earlier to 15 and newer 00:11:21.799 --> 00:11:25.159 can hit problems if roles like rds_ 00:11:25.159 --> 00:11:28.919 superuser and the built-in read-all or write- 00:11:28.919 --> 00:11:32.419 all roles end up referencing each other. AWS 00:11:32.419 --> 00:11:35.539 has a pre-upgrade query to find the problem. 00:11:35.820 --> 00:11:39.279 Not exciting. Very useful before your maintenance 00:11:39.279 --> 00:11:42.500 window. Third, a researcher released a proof 00:11:42.500 --> 00:11:46.860 of concept called FalconFlank, claiming a CrowdStrike 00:11:46.860 --> 00:11:50.139 privilege escalation path. Right now, I would 00:11:50.139 --> 00:11:53.000 treat this one carefully. There is public research 00:11:53.000 --> 00:11:57.480 and a PoC, but no CVE, affected version matrix, 00:11:57.820 --> 00:12:01.299 or CrowdStrike advisory yet. Interesting enough 00:12:01.299 --> 00:12:04.399 to watch though. Not enough evidence to panic. 00:12:04.620 --> 00:12:09.220 And fourth, SonicWall SMA1000 appliances have 00:12:09.220 --> 00:12:12.779 zero-day vulnerabilities being actively exploited. 00:12:13.000 --> 00:12:15.840 If those appliances are in your environment, 00:12:16.080 --> 00:12:19.940 this is firmly in the patch-now category. Internet- 00:12:19.940 --> 00:12:23.080 facing remote access infrastructure is not where 00:12:23.080 --> 00:12:25.799 I want to wait and see how exploitation develops. 00:12:34.480 --> 00:12:37.379 The human closer this week is about incident 00:12:37.379 --> 00:12:40.679 reviews, specifically whether distributed teams 00:12:40.679 --> 00:12:43.840 should start them asynchronously instead of immediately 00:12:43.840 --> 00:12:47.600 scheduling another meeting. One SRE team wrote 00:12:47.600 --> 00:12:51.120 about reconstructing a 37-minute outage almost 00:12:51.120 --> 00:12:55.299 entirely async. People added logs, timelines, 00:12:55.799 --> 00:13:00.440 screenshots, metrics, code references, and corrections 00:13:00.440 --> 00:13:03.929 directly into the review document. That part 00:13:03.929 --> 00:13:07.129 worked really well. It gave people time to check 00:13:07.129 --> 00:13:10.110 evidence instead of relying on memory in a meeting. 00:13:10.230 --> 00:13:13.110 Someone could make a claim, and another engineer 00:13:13.110 --> 00:13:16.590 could verify it. The timeline became much more 00:13:16.590 --> 00:13:20.370 precise, and people in different time zones could 00:13:20.370 --> 00:13:23.330 contribute without everybody needing to be online 00:13:23.330 --> 00:13:27.450 at once. But there was a downside. Disagreements 00:13:27.450 --> 00:13:31.159 became long comment threads. Questions about 00:13:31.159 --> 00:13:34.960 causality got buried. And action-item ownership 00:13:34.960 --> 00:13:39.279 became fuzzy. That sounds pretty familiar. Async 00:13:39.279 --> 00:13:42.480 communication is really good at collecting information. 00:13:42.820 --> 00:13:45.879 It is not always good at resolving ambiguity. 00:13:46.399 --> 00:13:49.879 A document can tell you what happened. A comment 00:13:49.879 --> 00:13:53.159 thread can show that three people disagree about 00:13:53.159 --> 00:13:56.340 why it happened. At some point, putting those 00:13:56.340 --> 00:13:59.320 three people in the same conversation is probably 00:13:59.320 --> 00:14:02.690 faster. The model they landed on was basically 00:14:02.690 --> 00:14:06.870 async first, live later. Use the document to 00:14:06.870 --> 00:14:10.570 gather evidence and build the timeline. Then 00:14:10.570 --> 00:14:14.110 hold a short live review only for the things 00:14:14.110 --> 00:14:17.590 that actually need discussion. Disputed causes, 00:14:17.990 --> 00:14:22.269 tradeoffs, decisions, and ownership. I like 00:14:22.269 --> 00:14:25.289 that a lot because the goal of a postmortem 00:14:25.289 --> 00:14:28.419 is not having a postmortem meeting. The goal 00:14:28.419 --> 00:14:31.340 is understanding the incident well enough that 00:14:31.340 --> 00:14:34.080 the organization learns something from it. Sometimes 00:14:34.080 --> 00:14:37.440 meetings help with that, but sometimes they just 00:14:37.440 --> 00:14:40.559 force 12 people to watch one person scroll through 00:14:40.559 --> 00:14:43.600 Datadog. Async evidence gathering also changes 00:14:43.600 --> 00:14:46.759 who gets heard. In a live review, the people 00:14:46.759 --> 00:14:49.559 who remember the incident fastest or speak the 00:14:49.559 --> 00:14:52.419 most confidently can shape the narrative early. 00:14:52.740 --> 00:14:56.429 In a shared document, somebody can go back, check 00:14:56.429 --> 00:14:59.830 the logs and say, actually, that happened four 00:14:59.830 --> 00:15:03.009 minutes later. Or this alert fired before the 00:15:03.009 --> 00:15:06.570 deployment. Or the database was already degraded 00:15:06.570 --> 00:15:09.950 before the API started failing. That is useful. 00:15:10.129 --> 00:15:13.909 But the async part still needs structure. Someone 00:15:13.909 --> 00:15:17.610 owns the timeline. Someone identifies unresolved 00:15:17.610 --> 00:15:20.730 questions. Someone turns the discussion into 00:15:20.730 --> 00:15:24.250 actions. And someone actually owns those actions 00:15:24.250 --> 00:15:28.259 afterward. Otherwise, you just traded a bad meeting 00:15:28.259 --> 00:15:32.320 for a very detailed document nobody closes. The 00:15:32.320 --> 00:15:34.980 thing I like most here is that it treats communication 00:15:34.980 --> 00:15:39.700 style as part of incident response design. Distributed 00:15:39.700 --> 00:15:42.659 teams do not have to recreate the same meeting- 00:15:42.659 --> 00:15:45.679 heavy process companies used when everybody 00:15:45.679 --> 00:15:50.419 sat in one office. Use async for the things async 00:15:50.419 --> 00:15:54.379 does well. Use live conversation for the things 00:15:54.379 --> 00:15:57.779 humans resolve better together. And keep both 00:15:57.779 --> 00:16:01.700 focused on learning, not blame. Because the best 00:16:01.700 --> 00:16:04.580 incident review is not the one with the most 00:16:04.580 --> 00:16:07.759 complete document. It is the one that actually 00:16:07.759 --> 00:16:11.860 changes how the next incident goes. That's it 00:16:11.860 --> 00:16:15.320 for this week's Ship It Weekly. We covered Gateway 00:16:15.320 --> 00:16:20.120 Load Balancer TCP Reset, Azure DevOps Live Migrations 00:16:20.120 --> 00:16:23.759 to GitHub, GitHub's self-hosted Actions runner 00:16:23.759 --> 00:16:27.279 enforcement, and the Omarchy Docker privilege 00:16:27.279 --> 00:16:32.679 issue, plus Lambda IAM policy improvements, Postgres 00:16:32.679 --> 00:16:36.559 SQL upgrade traps, the FalconFlank research, 00:16:36.919 --> 00:16:40.879 and SonicWall zero-days. Follow or subscribe 00:16:40.879 --> 00:16:44.220 wherever you are watching or listening. You can 00:16:44.220 --> 00:16:47.639 find the weekly story list and source links at 00:16:47.639 --> 00:16:51.649 OnCallBrief.com. and past episodes and show 00:16:51.649 --> 00:16:55.690 notes at ShipItWeekly.fm. I'm Brian Teller from 00:16:55.690 --> 00:16:58.509 Teller's Tech. Thanks for listening. And remember, 00:16:58.750 --> 00:17:02.269 recovery gets easier when systems fail clearly, 00:17:02.490 --> 00:17:05.569 migrations happen deliberately, and the humans 00:17:05.569 --> 00:17:08.730 reviewing incidents have enough space to figure 00:17:08.730 --> 00:17:10.309 out what actually happened.