AWS can now reset failed Gateway Load Balancer
connections instead of letting them hang for
minutes. GitHub is getting ready to stop running
jobs on stale self-hosted Actions runners. And
a Linux desktop setup accidentally gave basically
every user-space process a path to root through
Docker. This week is mostly about failure modes
hiding inside things we already trust. I'm Brian
Teller from Teller's Tech, and this is Ship It
Weekly. Welcome back to Ship It Weekly, the show
about the DevOps, SRE, cloud, platform, and security
stories that matter when you are the person keeping
the thing running at three in the morning. For
the weekly story list and source links, check
out OnCallBrief.com. For past episodes and show
notes, head over to ShipItWeekly.fm. This week,
AWS Gateway Load Balancer gets TCP Reset to
shorten recovery when a firewall appliance dies.
Azure DevOps gets live migration tooling for
moving enterprises to GitHub. GitHub is starting
enforcement against stale self-hosted actions
runners. And Omarchy shipped with a Docker configuration
that effectively made normal desktop processes
root-capable. Then we have a quick lightning round
and a human closer about whether incident reviews
should happen asynchronously first, live later.
Let's get into it. First up, AWS Gateway Load
Balancer now supports TCP Reset. This is one
of those small-looking features that solves a
pretty ugly failure mode. Gateway Load Balancer
is commonly used to put firewalls, intrusion
detection systems, and other network appliances
inline with application traffic. When one of
those appliances fail, new connections can usually
move to a healthy target. Existing TCP connections
are harder. They may keep trying to use the failed
path until the connection times out. Depending
on the application and TCP settings, that can
mean 30 seconds. It can also mean several minutes.
From the application's point of view, the network
is not obviously broken. It is just hanging.
With TCP Reset enabled, Gateway Load Balancer
can actively terminate those affected connections.
The client gets a reset immediately and can reconnect
through a healthy appliance. That turns an ambiguous
failure into an explicit one. And I think that
is the useful lesson here. Fast failure is often
better than slow uncertainty. A failed request
gives the application something that it can react
to. A connection sitting there doing nothing
is much harder. Retries do not happen yet. Timeout
budgets get consumed. Users wait. Threads stay
blocked. And everything upstream looks degraded
instead of clearly failed. We talk a lot about
graceful failover, but sometimes graceful means
being very direct. This path is dead. Start over
somewhere healthy. This is especially important
around stateful middleboxes because the backend
application may be perfectly fine while the network
path in front of it is not. The shorter you can
make that ambiguity window, the faster the rest
of the system can recover. Next, Microsoft has
put Enterprise Live Migrations from Azure DevOps
to GitHub into public preview. This is aimed
at large migrations where shutting down engineering
for a weekend is not really an option. The idea
is that repositories remain writable in Azure
DevOps while changes continuously synchronize
into GitHub Enterprise Cloud. Teams keep working
during most of the migration. Then, when it is
time for the final cutover, you stop writes,
sync the remaining changes, and switch over.
Microsoft says the final downtime can typically
stay under about 30 minutes. The migration tooling
can also convert Azure DevOps branch policies
into GitHub rulesets. And importantly, Azure
pipelines can continue running while repositories
move. That matters because repository migration
is rarely just copying Git objects. The hard
part is everything attached to the repository.
Permissions, branch protections, CI/CD. webhooks,
secrets, bots, service connections, release workflows,
and all of the little assumptions around URLs
and identities that accumulate over 10 years.
A migration can be technically successful and
still be operationally terrible if engineers
lose half a day every time you move another batch
of repositories. Live synchronization changes
that equation. It lets the migration behave more
like a controlled transition instead of a hard
stop. But I would still treat this like any other
production migration. Inventory what depends
on Azure DevOps before you move anything. Test
policy conversions. Validate pipelines. Know
what happens to automation that references old
URLs. And have a rollback path for the cutover
itself. The goal is not just moving the code.
The goal is moving the development system around
the code without everybody discovering the missing
pieces on Monday morning. Third, GitHub is getting
serious about stale self-hosted Actions runners.
This one is worth checking if you manage your
own runner fleet. GitHub is rolling out minimum
version enforcement. Self -hosted runners need
to be at least version 2.329.0 to register.
And runners also need to stay within 30 days
of current releases to continue executing jobs.
GitHub is starting runtime brownouts on September
9th. Full enforcement is planned for September
25th. The obvious action is to update your runners.
The more interesting question is why they are
stale in the first place. A lot of teams treat
self-hosted runners like infrastructure appliances.
Build the image. Deploy the runner. Forget about
it until something breaks. But the runner is
executing arbitrary CI workloads with access
to repositories, credentials, networks, artifact
stores, and often cloud permissions. It is part
of your security boundary. It is also a client
of GitHub's service. And that client has to evolve
with the platform. If your upgrade process requires
somebody remembering to rebuild an AMI every
few months, enforcement like this is eventually
going to catch you. This is where ephemeral runners
are really attractive. Start from a maintained
image. Run the job. Destroy the environment.
Regularly rebuild the base image and roll forward
automatically. That does not eliminate maintenance.
It makes maintenance part of the lifecycle instead
of an exception to it. And if you cannot automatically
replace a runner fleet today, this brownout is
probably a good excuse to figure out why. Because
September 25th is a much nicer time to discover
that problem than during a production deployment.
Fourth, Omarchy fixed a configuration that effectively
gave user-space applications a path to root through
Docker. The default user was part of the Docker
group. That is common on Linux systems because
it lets you run Docker commands without sudo.
It is also basically root access. The Docker
daemon runs as root. If your user can talk to
the Docker socket, you can generally start a
privileged container, mount the host filesystem,
and modify the machine. So anything running as
that user inherits a very powerful capability,
a browser extension, an IDE plugin, a compromised
development tool, a coding agent, malware running
in the session. If it can reach the Docker socket,
the jump from normal user to root may be trivial.
Omarchy fixed this in version 4.0.1. But the
bigger point is not really about Omarchy. The
Docker group has been a known privilege boundary
for a long time. What has changed is how much
software we now run inside developer sessions.
IDE extensions, package managers, AI coding tools,
local agents, browser automation, dev containers,
CLI plugins. A modern workstation is a pretty
busy environment. Giving every one of those processes
indirect root access because Docker is more convenient
without sudo is worth thinking about. Convenience
is fine. Just name the privilege correctly. If
membership in a group means the user can become
root, treat that group like root access. Because
an attacker absolutely will. Quick lightning
round. First, AWS Lambda now supports fuller
IAM resource-based policies. You can define
multiple principals and actions and use normal
IAM condition keys for cross-account access.
That should make larger Lambda authorization
models cleaner than some of the older one permission
at a time workflows. Second, AWS is warning about
PostgreSQL upgrades that can fail because of
circular role memberships. RDS and Aurora upgrades
from PostgreSQL 14 or earlier to 15 and newer
can hit problems if roles like rds_
superuser and the built-in read-all or write-
all roles end up referencing each other. AWS
has a pre-upgrade query to find the problem.
Not exciting. Very useful before your maintenance
window. Third, a researcher released a proof
of concept called FalconFlank, claiming a CrowdStrike
privilege escalation path. Right now, I would
treat this one carefully. There is public research
and a PoC, but no CVE, affected version matrix,
or CrowdStrike advisory yet. Interesting enough
to watch though. Not enough evidence to panic.
And fourth, SonicWall SMA1000 appliances have
zero-day vulnerabilities being actively exploited.
If those appliances are in your environment,
this is firmly in the patch-now category. Internet-
facing remote access infrastructure is not where
I want to wait and see how exploitation develops.
The human closer this week is about incident
reviews, specifically whether distributed teams
should start them asynchronously instead of immediately
scheduling another meeting. One SRE team wrote
about reconstructing a 37-minute outage almost
entirely async. People added logs, timelines,
screenshots, metrics, code references, and corrections
directly into the review document. That part
worked really well. It gave people time to check
evidence instead of relying on memory in a meeting.
Someone could make a claim, and another engineer
could verify it. The timeline became much more
precise, and people in different time zones could
contribute without everybody needing to be online
at once. But there was a downside. Disagreements
became long comment threads. Questions about
causality got buried. And action-item ownership
became fuzzy. That sounds pretty familiar. Async
communication is really good at collecting information.
It is not always good at resolving ambiguity.
A document can tell you what happened. A comment
thread can show that three people disagree about
why it happened. At some point, putting those
three people in the same conversation is probably
faster. The model they landed on was basically
async first, live later. Use the document to
gather evidence and build the timeline. Then
hold a short live review only for the things
that actually need discussion. Disputed causes,
tradeoffs, decisions, and ownership. I like
that a lot because the goal of a postmortem
is not having a postmortem meeting. The goal
is understanding the incident well enough that
the organization learns something from it. Sometimes
meetings help with that, but sometimes they just
force 12 people to watch one person scroll through
Datadog. Async evidence gathering also changes
who gets heard. In a live review, the people
who remember the incident fastest or speak the
most confidently can shape the narrative early.
In a shared document, somebody can go back, check
the logs and say, actually, that happened four
minutes later. Or this alert fired before the
deployment. Or the database was already degraded
before the API started failing. That is useful.
But the async part still needs structure. Someone
owns the timeline. Someone identifies unresolved
questions. Someone turns the discussion into
actions. And someone actually owns those actions
afterward. Otherwise, you just traded a bad meeting
for a very detailed document nobody closes. The
thing I like most here is that it treats communication
style as part of incident response design. Distributed
teams do not have to recreate the same meeting-
heavy process companies used when everybody
sat in one office. Use async for the things async
does well. Use live conversation for the things
humans resolve better together. And keep both
focused on learning, not blame. Because the best
incident review is not the one with the most
complete document. It is the one that actually
changes how the next incident goes. That's it
for this week's Ship It Weekly. We covered Gateway
Load Balancer TCP Reset, Azure DevOps Live Migrations
to GitHub, GitHub's self-hosted Actions runner
enforcement, and the Omarchy Docker privilege
issue, plus Lambda IAM policy improvements, Postgres
SQL upgrade traps, the FalconFlank research,
and SonicWall zero-days. Follow or subscribe
wherever you are watching or listening. You can
find the weekly story list and source links at
OnCallBrief.com. and past episodes and show
notes at ShipItWeekly.fm. I'm Brian Teller from
Teller's Tech. Thanks for listening. And remember,
recovery gets easier when systems fail clearly,
migrations happen deliberately, and the humans
reviewing incidents have enough space to figure
out what actually happened.