A little less than three years ago, Red Hat shifted focus in the CentOS project, from the downstream rebuild to the upstream called CentOS Stream. I wrote a heartfelt response to try and explain it better: Before You Get Mad About The CentOS Stream Change, Think About… That article seemed to help dispel some of the conspiracy theories I’d seen floating around.
Recently, Red Hat made an announcement explaining that we’re going to stop doing work to make it easy for downstream rebuilds to claim patch-for-patch compatibility while contributing very little upstream to CentOS Stream. This led to a decent amount of discussion, and another response by my long-time, and well-respected colleague at Red Hat, Mike McGrath, essentially nudging downstream projects to contribute to CentOS Stream.
That brings us to today, and makes this recent Twitter thread and CentOS Stream Merge Request (MR) all the more important to discuss in a constructive way. I responded on Twitter but it’s difficult to do justice to a subject this complicated with 240 characters at a time so I wanted to respond in longer form here in this blog post.
Red Hat: Rebuilders contribute n̶o̶ ̶v̶a̶l̶u̶e̶ negative net value to RHEL. Contribute to CentOS Stream!
Rebuilder @AlmaLinux: okay we'll build off CentOS Stream. [Proceeds to submit patch to Fedora + Stream for a CVE against iperf3]
Red Hat: No, not like that! pic.twitter.com/lVXfLhHKrt
— Jeff Geerling (@geerlingguy) July 19, 2023
First off, I understand people’s frustration. I understand the cathartic feeling of finding a wrong, and righting it. I get it. I’m known to partake in a good rant from time to time. That said, I really do think this particular interaction is nuanced, and important, so it deserves a deeper dive. I think bullet points might be the easiest way to do this, and I’m going to word each of these in the most positive way that I can. Note, this is my “hot take” and I will definitely update or fix anything I’ve gotten wrong:
- This contribution by a member of the AlmaLinux community is forcing a tough conversation about what changes should or shouldn’t be accepted in CentOS Stream and therefore RHEL. There is also a healthy debate happening internally at Red Hat. This is good. This is a testament to RHEL’s upstream development happening in a transparent way, and improving RHEL through CentOS Stream.
- This is the first material contribution I’ve become aware of from the EL community (AlmaLinux, Rocky, Oracle, etc) which formed around the originally downstream rebuilds (now peers) of RHEL. This is good. But, this is just the beginning. This isn’t going to be easy (more on that later).
- Red Hat follows a Risk Based Approach to security fixes (the only sane model). Our colleague Vincent Danen explains it quite well in this article: Demystifying risk using CVEs and CVSS
- The community is learning some tough lessons about RHEL. I’ve learned these lessons over 12 years at Red Hat. RHEL currently follows a very risk-averse model where changes are typically ONLY accepted if there is customer need. The risk of unknown change is weighed quite heavily against any potential fix, especially if the fix is a low or moderate CVE which isn’t exploitable, or is only exploitable by an administrator (IMHO, likely for this CVE).
- Life cycle is also a large factor in this conversation. The Red Hat Enterprise Life Cycle page is key to this discussion and provides some history. Once RHEL reaches the maintenance phase, it’s quite difficult to get changes into RHEL. Stability is prioritized very highly.
- This change is for 9 which is the newest version of RHEL, and it’s for a CVE, not a feature. These are two things which I think will be considered when deciding whether to accept this change. I think it’s important to point out that this is not a feature, not a bug fix, but a security fix for a version of RHEL in the Full Maintenance Phase, so I understand why the contributor believes this should be accepted. I get it. The Red Hat bugzilla is littered with rejected requests for fixes for Low and Moderate CVEs because often, even a one liner fix, breaks people in production.
- You may be thinking, “if I were Red Hat, I would have been extra careful with the first contribution from AlmaLinux” – that’s a fair question. I’m not sure that’s as easy of a problem as people think. First, you have to know what organization people are from. Then, you have to have a company-wide strategy. Then you have to communicate it and train people, etc. These problems look quite simple and easy from the outside, but they’re not quite so simple when you have ~25,000 people. Some people pay more attention to internal communications, some to Social Media, etc.
- Tracking which organization a contribution comes from is a much harder problem than people realize. Today, not everybody uses an email address which connects back to the organization which they’re contributing from. The Fedora and CentOS Stream communities do not ask, and do not force people to list the organization they work for in their profile. This makes it difficult to collect data and do analysis. The upstream communities are thinking about how to solve this, but it’s not solved today.
- In this case, Michal Ruprich was indeed aware that this was an AlmaLinux contributor and how sensitive this was (I checked with him), but he treated this bug just like he would from any other contributor. No special exception was made for AlmaLinux, and an order from management never went out to make a special exception for AlmaLinux or any other contributor. He treated this merge request JUST like he would have if it was a Red Hat employee. Let that sink in. I think that’s fair. I don’t think Red Hat is trying to make special exceptions or to pander for marketing purposes.
- Culture, personality and relationships play a huge role in these interactions. We must understand this. I genuinely believe culture may have played a bit of a role in this as well. A diatribe about cultural differences between different parts of the world is beyond the scope of this post, but suffice to say, I beg people, please consider this with all interactions online.
To summarize, I think the process in CentOS Stream is working. It’s working as it has always worked, and does not treat anyone with special exceptions. There are new interactions happening between members of different communities, and I hope this continues! I think this merge request will continue to be evaluated. It may or may not get accepted, but I very much appreciate that it happened, and I hope we continue to see more interactions, and more learning!
As I’ve mentioned in other places, Red Hat is STILL learning about open source just like the rest of the world.
I think I understood your article, but the concept of customer is very restricted in this view. As a clone version customer, I believe I need to move away from this ecosystem. The organization where I work has a lot of Red Hat systems, and also has a bunch of clone systems. We also have Suse systems, which at the moment is more worth our investment.
It pains me to hear that, but I do understand. I would question whether Suse can actually offer as much as Red Hat, but again, I understand the sentiment.
These are all fair points, thanks for posting.
A while ago, and executive taught me a really important lesson. I was proposing what I thought was a great new feature / product direction. I was concerned with how we could do it, what it could do the the customers, how it would position the company etc.
He asked me something that I hadn’t considered, something that I hadn’t _had_ to consider before with any such proposal:
What if if works great? What if it becomes the most popular thing we make? What if it is super successful? Can we support it?
While I was mired in the technical details and the possibilities of what it could do, I hadn’t considered what we as a business would have to do for it.
This is an important thing. “Are we ready, do we have a plan? Do we know what it will me to be in the xxxx business 100%?’
I don’t understand how Red Hat could have made such a huge change to the new status quo post Centos, and not have planned and figured out how to handle it if people actually did what they wanted. If this is a _process_, and will take a while to work out, like from a starting at 0 level, why roll it out before it is at least remedially ready?
“What if alma and rocky do what we want? How is the process going to work? How do we set it up so that all parties understand how it works? how do we make this a positive?”
Red Hat is missing an opportunity here. How can we make this a positive? How can we give people a positive take away from this change from _their_ point of view, and end up with a better ecosystem for everyone instead it only looking like we are doing what is good for us?
– give a vision of how Stream contributions would work, how downstream can have a stake and some participation beyond MRs. A vision of a new world order where they can see a positive out of it
– marshal existing programs, come up with new ones (ISV etc ) that gives some palatable path for downstream users or businesses to become Red Hat users in a way other than “fuck you, pay me’ Maybe respecting business realities is a two-way street.
So first off, thank you for the thoughtful reply. Most people do not think this deeply about things. I think your challenge is a fair one. Let me counter with, why are we just figuring this out now if the downstreams were really contributing so much? Wouldn’t we, Red Hat have already learned to work with them better? I’d summarize it in a slightly more nuanced way than Red Hat didn’t have a plan for success:
1. I’m sad that this is the first material contribution to CentOS stream from a rebuilder. In theory, that’s what CentOS Stream was for, and there was public commitment that the rebuilders were indeed going to contributer there 2-3 years ago. We now have our answer. That was never happening. I’ve been snooping for two years, and the few PRs/changes I’ve actually been able to track back to the downstream contributors are rarely more than someting which removes branding. That’s not a material contribution IMHO. I feel harsh saying it out loud, but it’s something that a bunch of Red Hatters chatter about internally and are too scared to say publicly.
2. Now that the we do have one good example of a material contribution, much of the public commentary makes it clear that many/most of the rebuilders don’t actually understand how or why RHEL is built the way it is. Disciplined and very purposeful change is exactly why RHEL customers love it. That’s why people want a free-as-in-beer version. It’s also why Red Hat views free-as-in-beer versions which claim patch-for-patch compatibility as a net-loss for Red Hat.
3. Alma and Rocky can and perhaps ought to make these kinds of changes in a Special Interest Group (SIG) where they can change anything they want, in any way they want. Then, they don’t have to be bound by the same rules as RHEL, and if their opinions work out better for people, we can learn from that. This is going to sound harsh too, but they should already know this. CentOS Stream has been around for years. You could argue that Red Hat should tell them that, or should have had an FAQ ready to go, but I’ll be honest, these things are super surprising to us. At Red Hat, we actually assumed that rebuilders knew this stuff. You can call that arrogant, or maybe naive. I think it’s more naive, and learning, on both sides.
4. The community is skeptical of Red Hat’s intentions. I get that. I really do. But Red Hat is also skeptical of the downstream reabuild’s intentions. There is not trust on either side. I think Red Hat has cognitive dissonance about whether we really *do* want contributions. I think some people do, but others don’t. That’s my *personal* and honest opinion. I think Red Hat “could” be convinced that contributions will be material and valuable, but that hasn’t been proven yet.
5. Given #1-4, yeah I think your critique about not having a plan is fair. I think many people within Red Hat were skeptical that we’d ever see a material contribution. I’d say that many still are skeptical that this will be a trend, and more than just an example to hang Red Hat with.
6. On a personal level, I’m actually glad this bug was submitted. Selfishly, as the product manager for RHEL server, I kind of love that outside people are challenging my own engineers. In my experience, it’s ALWAYS good to get an outside perspective.
7. Given #2 and #6, I’m kind of excited. Who’s right? Maybe our policies on CVEs balance stability a little too much over security? Maybe they’re still just right? That’s a legitimate debate to have, and honestly, I’m quite excited that it’s happening.
8. The community is learning how RHEL is built, and perhaps Red Hat is being forced to reevaluate our policies. That doesn’t mean they’ll change, but I’m glad fresh blood are challenging them. That’s healthy. In my personal opinion, that’s real value from the community.
So, call me cautiously optimistic. Do I believe that 5-10% of RHEL 10 will get built by the Rocky, Alma and Oracle contributors? No way. Will the those contributors pick up 5-10% of the maintenance cost of RHEL over it’s lifecycle? no way. It won’t even be 1/10th of one percent. But will this new way of interacting make RHEL and all of the derivitives better? I think it might. And, that’s a good thing.
I hope these super honest opinions don’t come off as harsh, and I hope they inform you a bit about how it looks from somebody who’s collected a pay check from open source for 12 years. I’ve bought a house, I’m raising two children. We have two cars. We took a nice vacation this summer. I’d like that to continue 🙂 But, I’m also always looking for the win-win. I hope we can find it.
Thanks for this great article. It’s so easy to pick up on Red Hat these days so it’s refreshing to read a well reflected, fair opinion.