This week, Amazon EKS gets Kubernetes version
rollbacks, which sounds like an undo button,
but rollback is not the same thing as readiness.
A new supply chain report called Cordyceps found
exploitable GitHub Actions patterns across major
open source projects, which is another reminder
that CI/CD YAML is code with credentials. Tenet
Security showed how a fake Sentry error could
hijack AI coding agents into running
attacker-controlled code. So logs, errors, and telemetry
are not just evidence anymore. They may also
be instructions. And CloudWatch can now create
alarms directly from log queries. Useful, yes,
but also a very fast way to turn noisy logs into
noisy pages. Put those together, and the theme
this week is pretty clear. The safety nets are
getting better, but the blast radius is getting
wider. I'm Brian Teller from Teller's Tech, and
this is Ship It Weekly. Welcome back to Ship
It Weekly, the show about the DevOps, SRE, cloud,
platform, and security stories that actually
matter when you are the person who has to keep
the thing running at 3 a.m.. If you're new here,
follow or subscribe wherever you are watching
or listening. And if you want the weekly story
list and source links, check out OnCallBrief
.com. For past episodes, full show notes, and
more from the show, head over to ShipItWeekly
.fm. This week, we start with EKS version rollbacks
because Kubernetes upgrades finally get a safety
net in a place where a lot of teams have wanted
one. Then, Cordyceps and why GitHub Actions workflows
need to be treated like production code. After
that, agentjacking, where fake telemetry can
become a way to steer AI coding agents. Then,
CloudWatch log query alarms and the line between
better signals and easier noise in the lightning
round ECS Service Connect zone-aware routing
etcd 3.7, GitHub innersource advisories, secret
scanning metadata and CloudWatch application
signals service events and we close with a simple
reminder safety nets are good but they do not
replace ownership let's get into it First up,
Amazon EKS now supports Kubernetes version rollbacks.
And this is a big one. For a long time, Kubernetes
control plane upgrades have felt like a one-way
door. You do the checks, you read the release
notes, you look at your add-ons, your nodes,
your admission controllers, your autoscaling,
your old helm charts, and whatever legacy thing
nobody wants to touch. Then eventually, you upgrade.
and hope the cluster is still boring. EKS is
changing that with version rollback. If you upgrade
an EKS cluster and hit a problem, AWS now lets
you roll the Kubernetes version back to the previous
version within a seven-day window. That gives
teams a real recovery path when an upgrade exposes
a compatibility issue. That is useful, especially
if you run a lot of clusters. have strict change
windows, or operate in environments where Kubernetes
upgrades are already surrounded by approvals
and nervous Slack threads. But this does not
make upgrades free. It does not make your workloads
compatible. It does not fix every add-on problem.
It does not replace staging. And it does not
mean your whole cluster magically goes back in
time like nothing happened. AWS also added rollback
readiness checks, which is important because
rollback has its own requirements. Your nodes
matter. Your add-ons matter. Your disruption
budgets matter. And if you use EKS Auto Mode,
node rollback becomes part of the story too.
That means rollback can take time. And in an
incident, people hear rollback and assume fast.
But safe rollback is not always instant rollback.
The takeaway is pretty straightforward. Use this
feature. Put it in your upgrade runbook. Test
the path. Know what it does and does not cover.
But do not treat the rollback button like permission
to skip upgrade discipline. A rollback window
is a safety net. It is not a test plan. EKS got
an undo button. That is great. Just make sure
that someone still owns the upgrade. Second story,
Novee Security published research called Cordyceps,
and this one lands directly in the your CI/CD
is production code bucket. The report looks at
exploitable GitHub Actions workflow patterns
across high impact open source repositories.
Not one weird repo, not one random bad workflow.
A repeated class of CI/CD mistakes. Novee says
that they scanned around 30,000 high-impact
repositories and confirmed more than 300 fully
exploitable attack chains. The patterns include
command injection, artifact poisoning, broken
trust boundaries between workflows, and untrusted
pull request or comment data making its way into
privileged automation. That last part matters
because most teams still treat workflow YAML
like config. It gets copied from another repo, tweaked
until the build is green. Reviewed quickly. Then
forgotten. But workflow YAML can run commands, read
secrets, publish packages, sign artifacts, assume
cloud roles, push containers, cut releases. That
is not just automation. That is a privileged
execution path. The nasty part is that these
problems are often not obvious from one line.
A low-trust workflow produces an artifact. A
high-trust workflow consumes it later. A branch
name lands in shell. A pull request title becomes
command input. A cache crosses a privilege boundary.
Each piece looks normal enough. Together, you
accidentally gave untrusted input a path into
trusted automation. And now, add AI coding agents
to the picture. If agents are generating CI/CD
configuration, they can reproduce insecure workflow
patterns very quickly across a lot of repos.
The fix is not to stop using GitHub Actions. The
fix is to review workflow logic like code with
production authority, threat model your triggers,
separate trusted and untrusted workflows, be
careful with pull_request_target, watch shell
interpolation, limit token scopes, treat artifacts
and caches as trust boundaries, and stop assuming
YAML is harmless because it looks boring. Cordyceps
is a good reminder that your pipeline is part
of your supply chain. And attackers already know
that. Third story. Tenet Security published
research on agentjacking. The short version:
They showed how a fake Sentry error report could
influence AI coding agents into running
attacker-controlled code. That should make platform and
SRE teams pause. Because this is not only about
one agent or one vendor. The bigger issue is
that operational context is changing. Logs, errors,
stack traces, issue descriptions, pull request
comments, incident notes, runbooks. That stuff
used to be mostly passive. A human read it, decided
what mattered, ignored the nonsense, and took
action. Now, agents are reading that same material,
and sometimes they are acting on it. In Tenet's
example, the attacker injects a crafted error
event. The event looks like normal diagnostic
context. An AI coding agent is asked to investigate
or fix the issue, and the agent treats malicious
content like instructions. That is the trust
boundary. Not did the attacker compromise the
server, but can the attacker put text somewhere the
agent will trust? That is a much lower bar and
a pretty uncomfortable one. A lot of our systems
were built assuming that logs and error text
were just evidence, not instructions. Maybe we
sanitize them for display. Maybe we redact secrets.
But we were not always asking whether a string
in telemetry could become part of an agent's
decision loop that has to change if an agent
can read from observability tools open repos
run commands install packages access cloud credentials
or modify code then untrusted context becomes
a security problem the control is not the agent
is smart the control is boundaries can it read
without executing Can it propose without running?
Can it separate trusted remediation guidance
from user-controlled error content? Can you
audit what it saw and what it did? And are there
live credentials sitting nearby when it makes
a bad decision? The takeaway is simple. Do not
wire agents into developer and incident workflows
and only ask about productivity. Ask about authority.
What can it touch? What can it run? What does
it trust? And what happens when that context
is hostile? Agentjacking is not just an AI story.
It is a trust boundary story. The text around
your systems can now cause action. So your trust
model has to include the text. Fourth story,
Amazon CloudWatch now supports creating alarms
directly from log queries. This is one of those
features that sounds small, but it changes how
teams build alerts. Before this, alerting on
logs usually meant extra glue, metric filters,
custom metrics, another system. Some pipeline
where a log pattern turns into a metric and then
the metric turns into an alarm. Now CloudWatch
can create alarms from log queries directly.
Those alarms can use normal CloudWatch alarm
actions, including SNS and EventBridge. That
is useful. A lot of incidents show up first in
logs. A specific exception. A repeated auth failure.
A weird timeout. A dependency error. A line that
says the thing is broken before the normal metric
catches up. So yes, I like this feature. But
I also fear it a little. Because easier alert
creation can become easier pager noise. Someone
sees an error. Someone says we should alert on
that. Nobody checks how often it happens. Nobody
asks whether it is user impacting. Nobody asks
who owns it. Nobody checks whether the log message
changes every other deploy. And suddenly your
on-call rotation is getting paged by a regular
expression with abandonment issues. The right
use case is targeted signal. Known failure mode.
Clear owner. Clear threshold. Clear action. Clear
reason why a human needs to be interrupted. If
the alarm fires and the on-call has no idea
what to do, you probably made a search shortcut,
not an alert. So this is a good CloudWatch feature.
It removes glue. It shortens the path from log
signal to operational response. Just be careful.
The fastest way to create an alarm is not always
the fastest way to create reliability. Sometimes
it is just the fastest way to create noise. Quick
lightning round. First, ECS Service Connect now
supports zone-aware routing. AWS says It is
enabled by default for new and existing services.
But existing services need a one-time redeployment
to pick up the behavior. The idea is to prefer
same-AZ traffic when possible. That can help
with latency, cross-AZ cost, and failure domain
behavior. Service-to -service networking is
never just connectivity. It is cost, latency,
and resilience. all tangled together. Second,
etcd 3.7 is out. Not the flashiest headline,
but etcd is still the backing store under Kubernetes.
This release includes RangeStream for streaming
large result sets in chunks, plus performance
work, cleanup of old v2 store remnants, Protobuf
updates, and dependency updates. For most managed
Kubernetes users, This is not an immediate action
item, but it is a good reminder that Kubernetes
reliability still depends on the boring distributed
database underneath it. Third, GitHub innersource
security advisories are now generally available.
This gives GitHub Advanced Security customers
advisory workflows for internal packages and
internal repos. That matters because not every
vulnerable dependency is public. Internal libraries,
shared actions, platform modules, and private
packages need disclosure and remediation workflows
too. Fourth, GitHub secret scanning added extended
metadata and multipart validation. For some
secrets, the token itself is not enough to validate
the risk. You may need a workspace URL, host,
endpoint, project, or other related field. That
sounds small. But during incident response, it
matters. Finding a secret is step one. Figuring
out whether it is real, live, owned, expired,
scoped, and safe to revoke is where the pain
starts. Fifth, CloudWatch Application Signals
now captures service events automatically. AWS
says that it can capture exception snapshots,
latency event snapshots, and deployment events.
for instrumented applications that is useful
because during an incident teams do not only
need to know the service is slow they need to
know what changed more context near the signal
is good as long as it does not turn into another
wall of events that nobody owns the human closer
this week is about safety nets because that is
the shape of the episode. EKS gets rollback.
CloudWatch gets easier log-based alarms. ECS
routing gets more zone-aware. GitHub gets better
advisory and secret scanning workflows. etcd
gets reliability and performance improvements.
Even the AI agent security story is really about
figuring out where the new guardrails need to
go. And I like all of that. I like rollback buttons.
I like better alerts. I like scoped workflows.
I like dependency and secret cleanup that does
not require five spreadsheets and a prayer. But
safety nets can trick teams into thinking the
hard part is solved. It usually is not. A rollback
button does not tell you whether the rollback
is safe. An alarm does not tell you whether the
signal is worth waking someone up. A scanner
finding a secret does not tell you how to rotate
it without breaking production. An AI agent finding
a fix does not mean it should run commands on
your laptop. And zone-aware routing does not
mean you understand your failure domains. Tools
give you better options. They do not own the
system for you. That part is still on us. So
take the safety nets. Use them. Be glad they
exist. But do not let them replace ownership.
Because in production, the worst failures are
usually not the ones where you had no tools.
They are the ones where everyone assumed the
tool meant the thinking was done. That's it for
this week’s Ship It Weekly. We covered EKS version
rollbacks, Cordyceps, and CI/CD supply chain risk,
agentjacking through fake telemetry. CloudWatch
log query alarms, ECS zone-aware routing, etcd
3 .7, GitHub advisories, secret scanning, and
CloudWatch Service Events. If this episode was
useful, follow or subscribe wherever you are
watching or listening. If you are on YouTube,
hit subscribe. If you are in a podcast app, follow
the show there. And if you know someone dealing
with Kubernetes upgrades, GitHub Actions security,
AI Coding Agents, or Alert Noise, send them this
one. It genuinely helps the show grow, and it
helps me keep making this for the people who actually
live with these systems. You can find the weekly
brief at OnCallBrief.com and the full show notes,
links, and past episodes at ShipItWeekly.fm.
I'm Brian Teller from Teller's Tech. Thanks for
listening. And remember, safety nets are good.
but they do not replace ownership.