---
# The Last Hurdle: Absorbing The Patches

**URL:** https://crunchtools.com/absorbing-the-patches/
Date: 2026-09-09
Author: fatherlinux
Post Type: post
Summary: Discovery and remediation are hard, but absorbing security updates is the final hurdle. What I'm seeing in 30 Red Hat Enterprise Linux customer meetings.Continue Reading "The Last Hurdle: Absorbing The Patches" →
Categories: Articles
Tags: AI/ML, RHEL, Security, Systems Administration
Featured Image: https://crunchtools.com/wp-content/uploads/2026/09/gemini_gen_20260909_015631_7b54f06c.png
---

I read [jyn's post about having a year to fix security everywhere](https://jyn.dev/a-year-to-fix-security/) this morning, and I've been thinking about it all day. It mirrors something I've been hearing in discussions with Red Hat Enterprise Linux (RHEL) customers. The argument is essentially: open weight models, good enough to find and exploit real vulnerabilities, are now cheap enough that one person with $5,000 to $10,000 can run one continuously, day and night, finding new, undisclosed CVEs. This is way cheaper than doing it with frontier models today, and it means the window is closing a lot faster than we're updating our infra to handle patching at this scale. I agree with that threat assessment, I agree with the recommendations, and I would rather amplify the post than nitpick at its edges. What I can add is the view from a different seat, because over the last three to six months I've sat in somewhere between twenty five and thirty customer conversations where the patching discussion moved in interesting directions, especially as we talk about RHEL, Satellite, UBI, and Red Hat Hardened Images. To be clear, my perspective comes from tons of conversations with customers, partners, sales people, other product managers, security professionals, and of course, engineers on things like glibc, the Linux kernel, and of course the Hummingbird/Red Hat Hardened Images team.

The conversations have focused more and more on absorbing the patches as the final hurdle, and that nobody should mistake it for an easy one. Discovery is still a hard problem, and so is remediation in the sense of writing a correct fix that doesn't break three things downstream on its way out the door. The piece that gets the least attention of the three is the plumbing that carries a finished fix from the vendor into production across an entire fleet or estate. The *jyn* piece gets to the same place from a different perspective, pointing out that findings are getting very cheap while fixes aren't, and that deployment is the hard part rather than the patch itself. I'm watching customers reach that conclusion on their own, from the inside, and they're pulling some *classique* methodologies back off the shelf to deal with it.

## Air Gaps Showed Up Where I Did Not Expect Them

Air gapped environments used to be a conversation I only had with federal government accounts. The usual suspects, like DoD, NATO, defense contractors, or an occasional national lab. Those customers had their own genuine security reasons, as well as policy and compliance reasons. Before Mythos, I think many people treated air-gapped environments as a cost they paid because somebody made them, not because it bought them much beyond what a decent firewall did.

That has changed. I'm now having air gap conversations with private industry, with companies that have no compliance requirement pushing them there and no classified workload to protect. They're getting to it on their own, because they ran the math on what an over the air update path looks like when Mythos, Fable, Astra, GLM 5.3 and friends are on the other end, trying to find a path in… The *jyn* piece recommends that updates should be frequent but require physical access. I'm not making a policy argument here, I'm reporting what enterprises are already investigating.

## CI/CD Stopped Being a Feature Play

For most of the last decade, when I talked with a team about continuous integration and continuous deployment, the driver was feature velocity. Product teams wanted to ship faster than their competitors and the pipeline was how they did it. Security was in there somewhere, usually as a scanning stage somebody bolted on after an audit went badly, but nobody was building a pipeline because they were scared.

They are now. The teams I'm talking to are looking at CI/CD as survival infrastructure, as the thing that determines whether they can take security updates at the rate the updates are arriving. That turns out to be a different design conversation than the one I'm used to having. When the goal is velocity, you optimize for how quickly a developer can get a change into production, and when the goal is absorption you optimize for how quickly an upstream fix can cross your entire fleet without a human making a judgment call at every hop. You start caring a great deal more about rollback, about test coverage on the boring paths, and about whether you actually know what you're running. It doesn't hurt that containers, Kubernetes, and bootc (aka [image mode for RHEL](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux-10/image-mode)) are amazing pieces of technology for automating updates and rollbacks. It also doesn't hurt that Red Hat has played a leadership role in these technologies for over a decade.

I'm following this advice with my own self-hosted infrastructure (something I thought I'd never do again, but my oh my has Mythos and agents changed my calculus), which is probably the most honest thing I can tell you about how seriously I'm taking this technology wave. My WordPress sites (yes, you can pry WordPress out of my cold, dead hands) currently take over the air updates directly from WordPress.org, which is exactly the pattern I just described customers backing away from. I have a Claude Code terminal working on moving them to a model where the site gets rebuilt and redeployed from a known source and runs in a read only container. I got hacked this summer through an unpatched backup plugin, and one of the first things that attacker did was turn off my auto updates, which I wrote up in [my own Mythos moment](https://crunchtools.com/my-personal-mythos-moment/). So this isn't a theoretical preference on my part. I plan on writing more about how the Linux and WordPress ecosystems have commonalities in how they're fighting back against the bad guys, but that's for a future article.

## Live Patching Left the Boring Workloads Behind

Live patching, meaning applying a kernel fix without rebooting the machine, or applying a user space patch without restarting a service, has been around a long time and had a fairly predictable customer profile. SAP shops. Oracle databases on RHEL. Large, expensive, stateful workloads where a reboot meant a maintenance window somebody had to negotiate weeks in advance, and where the business cost of that window was large enough to justify the extra machinery. Boring workloads, in the best sense of the word.

The requests I'm fielding now come from a completely different crowd. People running modern cloud native workloads, in containers, on OpenShift and Kubernetes, are asking about live patching, and on its face that's a strange thing to want. The whole promise of that architecture was that individual nodes are cattle and you roll them without thinking twice about it. My read is that the reboot was never actually free, and that the assumption you can roll your entire fleet on short notice holds up fine when you do it once a quarter and starts coming apart when the fix cadence goes up by an order of magnitude. When you have to absorb a reboot every day, or multiple times per day, the cost of the rolling restart stops being noise and starts affecting your resource planning, and error budgets.

## The Skeptical Objection?

Often, when I lay this out, there are skeptical objections, and they're legitimate. I got one most recently from a buddy on Facebook, in a thread I was mostly typing into with two thumbs, which isn't where I usually do my best thinking. His position was that none of the stuff discussed in the *jyn* article, nor anything I've mentioned above, addresses the underlying asymmetry and cost of modernization. The core of the argument is that, because defenders have to find and fix everything while an attacker only needs one thing to work, and attackers can operate at machine scale and figure out what to do with the compromised systems later. While I have nitpicks with the asymmetry part of this argument, reflected in years and years of study of war (yes, I've read The Book of Five Rings, and The Art of War many times) as well as the results of the Ukraine war after approximately 4.5 years of fighting at the time of this writing, I think it's better to focus on the expense part of the argument.

The categorical fix everybody points to, is using memory safe language like Rust, but that requires a software modernization push that most organizations can't pay for in budget, people, downtime, or sheer organizational capacity to absorb change. Even with an AI tailwind, it's expensive. Worse, nobody has agreed on what good enough even looks like. So the cost stays tilted toward the attacker's advantage, for as long as it takes to get all of this sorted out, coordinated across industries and across borders.

I know how a rebuttal to that will likely sound, coming from somebody who works for a vendor, so be patient with me. The argument deserves something better than a brochure answer. The argument is not lazy and the failure mode it describes is real. Every shop I've ever worked with has something running in their estate that nobody wants to touch, on a version nobody wants to say out loud in a meeting. Where I think it goes wrong is in the assumption that your organization has to be the one doing the modernizing…

## Nobody Rewrote Every Android App

There is a wonderful example in the Android ecosystem. It's the one where somebody ran the experiment at scale and then published the numbers. In 2019, [memory safety defects were 76 percent of everything Google tracked in Android, and by 2024 that was down to 24 percent](https://security.googleblog.com/2024/09/eliminating-memory-safety-vulnerabilities-Android.html). In 2025 it [fell below 20 percent for the first time](https://security.googleblog.com/2025/11/rust-in-android-move-fast-fix-things.html). The absolute count dropped from [more than 220 in 2019 to a projected 36 by the end of 2024](https://security.googleblog.com/2024/10/safer-with-google-advancing-memory.html), and it managed that while the codebase was getting bigger rather than smaller.

Google never tried to rewrite Android, the ecosystem didn't try to rewrite all of the apps either. This has a profound effect on the modernization objection. They started writing the new code in Rust, left the existing C++ sitting exactly where it was, and let the difference in defect density drag the average down year over year. Their own measurement puts [Rust at roughly 0.2 memory safety defects per million lines of code against something closer to 1,000 for C and C++](https://security.googleblog.com/2025/11/rust-in-android-move-fast-fix-things.html), which they describe as a more than 1000x reduction. Even assuming their methodology is generous to itself, three orders of magnitude leaves a lot of room to be wrong and still come out ahead.

Focus on the cost for a second, the bigger part of the argument above IMHO. Google also measured a [4X lower rollback rate on Rust changes and about 25% less time spent in code review](https://security.googleblog.com/2025/11/rust-in-android-move-fast-fix-things.html). My guess is that the safe thing turned out to be the cheaper thing to ship, and if that holds up anywhere outside of Google it reframes the entire argument. You don't have a spending problem, you have a sequencing problem about when you write the new code and what you write it in.

## Killing the Class Instead of the Instance

With AI, constitutions, spec files, etc the most important rule is to quash classes of problems. If you come up with a new logging methodology, every other piece of software in your fleet can inherit it with a single change to the constitution. You must think in “classes of problems” instead of individual problems.

In the post-Mythos world, this is supremely important because the economics of finding bugs came apart underneath us. When a model can work through a codebase and surface thousands of exploitable issues for a couple thousand dollars in API credits, or a [Framework desktop in the $3000 range](https://frame.work/products/desktop-diy-amd-aimax300/configuration/new), patching instances is a treadmill running faster than Usain Bolt. What a model can't do is find a buffer overflow in code where writing a buffer overflow isn't possible in the first place, and once a class is off the table it stays off, no matter how much compute somebody points at it next year.

Patching a single CVE buys you that CVE and nothing else. Killing the class gets rid of every bug in the class… period. Great, one for the good guys. Woo hoo.

## What Is Actually Shipping

The rebuttal I usually get at this point is that it all sounds fine in theory but what's really moving? Ahem, quite a lot is moving.

In December of 2025, Miguel Ojeda landed a commit titled [“rust: conclude the Rust experiment”](https://github.com/torvalds/linux/commit/9fa7153c31a3e5fe578b83d23bc9f185fde115da), stripping the experimental language out of the kernel documentation on the grounds that the 2025 Linux Kernel Maintainers Summit had deemed the experiment concluded. [Android's Binder IPC driver](https://rust-for-linux.com/android-binder-driver) is written in it, the Apple GPU driver in Asahi was written in it from scratch, and so are Nova, [Tyr for Arm Mali](https://rust-for-linux.com/tyr-gpu-driver), and [pieces of the NVMe work](https://rust-for-linux.com/nvme-driver). Rust is still a small slice of the kernel by line count, and it will be for years yet, which is precisely the point. You don't need a rewrite to collect the benefit. You need the new code, and the exposed code, and the code chewing on bytes that came from a stranger, to be written in something that can't hold the bug.

Python is doing the same thing one layer up the stack. Emma Smith and the Rust for CPython group posted a [pre-PEP in November of 2025](https://discuss.python.org/t/pre-pep-rust-for-cpython/104906) and are [targeting Python 3.16](https://blog.python.org/2026/04/rust-for-cpython-2026-04/) as the first release with Rust in the interpreter itself. I want to be careful about the status here, because this is a proposal and not a done deal. The plan is a single extension module to start, the group's reference work is on zlib using the zlib-rs crate wrapped in PyO3, and the PEP still has to survive what the team itself expects to be a lengthy discussion before 3.16 beta 1 in May of 2027. Even if every bit of that lands, [Python 3.16 doesn't ship until October 5th of 2027](https://peps.python.org/pep-0826/).

Zlib isn't a random pick though. It's decompression, so it spends its whole life parsing bytes that somebody else handed you, which is precisely the surface where a memory safety bug turns into somebody's very bad afternoon. This methodology can and likely will be extended to other attack-rich surfaces like network facing libraries, or other parsing libraries. And this methodology goes beyond memory safety. It extends to threading as well. The pre-PEP makes that case directly, noting that with free-threaded Python becoming officially supported and more popular, ensuring the standard library is thread safe becomes critical, and Rust prevents data races at compile time. Somewhere between [a quarter and a third of third party Python extension modules are already written in Rust](https://discuss.python.org/t/pre-pep-rust-for-cpython/104906), a number that came out of the 2025 Language Summit, so the ecosystem got there well ahead of the interpreter, which is usually how these things go.

That's the mechanism, and it's not complicated. The fix arrives in the kernel, the language runtime, the base image, and the distribution, and your job is to consume the new thing on a cadence you were probably already committed to rather than to author any of it yourself. Nobody re-engineers the brakes on a dump truck they already own. The supplier ships a better part and you get it the next time the truck is in the shop. I spend my working life on the supply side of that trade, so weigh this accordingly, but I think the pipeline is a good deal further along than the panic suggests.

## The Defenders Got the Same Step Function

The other half of my buddy's argument was that AI is a step function for the attacker, and that if the capability leaks from state actors down to criminal organizations the whole balance tips over. He may well be right about the leak, and jyn's post is largely an argument that the leak has already happened.

The defensive side got the same step function though, and got it out in public. At DEF CON in August of 2025, the cyber reasoning systems in the DARPA AI Cyber Challenge [discovered 18 real, non-synthetic vulnerabilities in real open source software and provided 11 patches for them, submitting patches in an average of 45 minutes](https://aicyberchallenge.com/finals-winners-announcement/). All seven finalist teams released their cyber reasoning systems as open source afterward, and [OpenSSF picked the work up as OSS-CRS](https://openssf.org/blog/2026/04/02/from-aixcc-to-openssf-welcoming-oss-crs-to-advance-ai-driven-open-source-security/). Google's Big Sleep [found an SQLite flaw, CVE-2025-6965, that Google says was known only to threat actors and at risk of being exploited](https://blog.google/innovation-and-ai/technology/safety-security/cybersecurity-updates-summer-2025/). That's the scenario every security team assumes is playing out somewhere and can never prove, and that one resolved in our favor.

I got a much smaller taste of the same dynamic myself this summer. I hadn't hardened against the problem ahead of time and I'm not going to pretend otherwise, so what saved me was how fast I could recover once I understood what I was looking at, and the recovery went fast because I had the same class of tooling the bad guys were using.

## Protect the Agent From the World

There's one recommendation in jyn's post I want to sit with this one for a bit, because I think it's the thing our industry is getting most wrong right now. They say to sandbox the agents themselves, restrict credentials to narrow scopes, block requests at the firewall rather than in local configuration the model can override, and filter through proxies and trusted interfaces rather than trusting the agent to police itself.

Nearly everybody I talk to is worried about protecting their infrastructure from their agents. Very few of them are thinking about protecting their agents from the world. The models are reasonably well aligned and I run mine with permissions wide open more than I should admit, which I recognize isn't everybody's risk tolerance. Alignment isn't where the exposure is. I repeat, alignment isn't the problem. But… if your agent gets prompt injected by something it read on the open internet, you no longer have a well aligned assistant. And, if it has important credentials, you now have somebody else's assistant with your important credentials. Super bad.

I built my own answer to this, a filtering gateway that sits between my agents and everything they are able to reach, which I wrote up in [defending your agents against adversarial trajectory guidance](https://crunchtools.com/defending-against-adversarial-trajectory-guidance/). It isn't the only way to solve the problem and I'm not going to claim it's the best one. What I will claim is that a sandbox boundary is simply not enough. Repeat after me… Defense in depth.

## A Year Is Enough Time to Do Quite a Lot

I still don't think the balance is 80/20 in the attacker's favor. I think it's closer to 50/50, though I hold that number loosely, and I could be wrong about it. I think it sat strongly in the defender's favor for a very long time, long enough that we got comfortable and quit building like anybody was actually coming, and over the last couple of years it moved to something a lot closer to a coin flip. That's still bad, and I'm not trying to talk anybody down off a reasonable amount of concern, because most of us learned our lessons on this the hard way and have an incident we would rather not describe in detail. It's a very different problem than doom, though, and the difference matters a great deal when you're the one deciding whether to fund the boring work.

*jyn* says we have a year. I think that's about right, and I would point out that a year is enough time to get a great deal done if you start now and you spend it on absorption rather than on discovery. Get the pipeline built, find out what you're actually running, test the rollback until it's boring, and put a real boundary around your agents. None of that is exciting work, but all of it is good work.

I think the good guys are smarter than they look on the surface.

---

## Categories

- Articles

---

## Navigation

- [Home](https://crunchtools.com/)
- [Articles](https://crunchtools.com/category/articles/)
- [Events](https://crunchtools.com/category/events/)
- [News](https://crunchtools.com/category/news/)
- [Presentations](https://crunchtools.com/category/presentations/)
- [Software](https://crunchtools.com/software/)
- [Beaver Backup](https://crunchtools.com/software/beaver-backup/)
- [Check BGP Neighbors](https://crunchtools.com/software/check-bgp-neighbors-nagios/)
- [Chev](https://crunchtools.com/software/chev-check-vulnerabilities-script/)
- [Graph BGP Neighbors](https://crunchtools.com/software/grpah-bgp-neighbors/)
- [Graph MySQL Stats](https://crunchtools.com/software/graph-mysql-stats/)
- [Graph Sockets Pipes Files](https://crunchtools.com/software/graph-sockets-pipes-files/)
- [MCP Servers](https://crunchtools.com/software/mcp-servers/)
- [Petit](https://crunchtools.com/software/petit/)
- [Racecar](https://crunchtools.com/software/racecar/)
- [Shiva](https://crunchtools.com/software/shiva/)
- [About](https://crunchtools.com/about/)
- [Home](https://crunchtools.com)

## Tags

- AI/ML
- RHEL
- Security
- Systems Administration