A little less than three years ago, Red Hat shifted focus in the CentOS project, from the downstream rebuild to the upstream called CentOS Stream. I wrote a heartfelt response to try and explain it better: Before You Get Mad About The CentOS Stream Change, Think About… That article seemed to help dispel some of the conspiracy theories I’d seen floating around.
Recently, Red Hat made an announcement explaining that we’re going to stop doing work to make it easy for downstream rebuilds to claim patch-for-patch compatibility while contributing very little upstream to CentOS Stream. This led to a decent amount of discussion, and another response by my long-time, and well-respected colleague at Red Hat, Mike McGrath, essentially nudging downstream projects to contribute to CentOS Stream.
That brings us to today, and makes this recent Twitter thread and CentOS Stream Merge Request (MR) all the more important to discuss in a constructive way. I responded on Twitter but it’s difficult to do justice to a subject this complicated with 240 characters at a time so I wanted to respond in longer form here in this blog post.
Red Hat: Rebuilders contribute n̶o̶ ̶v̶a̶l̶u̶e̶ negative net value to RHEL. Contribute to CentOS Stream!
Rebuilder @AlmaLinux: okay we'll build off CentOS Stream. [Proceeds to submit patch to Fedora + Stream for a CVE against iperf3]
Red Hat: No, not like that! pic.twitter.com/lVXfLhHKrt
— Jeff Geerling (@geerlingguy) July 19, 2023
First off, I understand people’s frustration. I understand the cathartic feeling of finding a wrong, and righting it. I get it. I’m known to partake in a good rant from time to time. That said, I really do think this particular interaction is nuanced, and important, so it deserves a deeper dive. I think bullet points might be the easiest way to do this, and I’m going to word each of these in the most positive way that I can. Note, this is my “hot take” and I will definitely update or fix anything I’ve gotten wrong:
- This contribution by a member of the AlmaLinux community is forcing a tough conversation about what changes should or shouldn’t be accepted in CentOS Stream and therefore RHEL. There is also a healthy debate happening internally at Red Hat. This is good. This is a testament to RHEL’s upstream development happening in a transparent way, and improving RHEL through CentOS Stream.
- This is the first material contribution I’ve become aware of from the EL community (AlmaLinux, Rocky, Oracle, etc) which formed around the originally downstream rebuilds (now peers) of RHEL. This is good. But, this is just the beginning. This isn’t going to be easy (more on that later).
- Red Hat follows a Risk Based Approach to security fixes (the only sane model). Our colleague Vincent Danen explains it quite well in this article: Demystifying risk using CVEs and CVSS
- The community is learning some tough lessons about RHEL. I’ve learned these lessons over 12 years at Red Hat. RHEL currently follows a very risk-averse model where changes are typically ONLY accepted if there is customer need. The risk of unknown change is weighed quite heavily against any potential fix, especially if the fix is a low or moderate CVE which isn’t exploitable, or is only exploitable by an administrator (IMHO, likely for this CVE).
- Life cycle is also a large factor in this conversation. The Red Hat Enterprise Life Cycle page is key to this discussion and provides some history. Once RHEL reaches the maintenance phase, it’s quite difficult to get changes into RHEL. Stability is prioritized very highly.
- This change is for 9 which is the newest version of RHEL, and it’s for a CVE, not a feature. These are two things which I think will be considered when deciding whether to accept this change. I think it’s important to point out that this is not a feature, not a bug fix, but a security fix for a version of RHEL in the Full Maintenance Phase, so I understand why the contributor believes this should be accepted. I get it. The Red Hat bugzilla is littered with rejected requests for fixes for Low and Moderate CVEs because often, even a one liner fix, breaks people in production.
- You may be thinking, “if I were Red Hat, I would have been extra careful with the first contribution from AlmaLinux” – that’s a fair question. I’m not sure that’s as easy of a problem as people think. First, you have to know what organization people are from. Then, you have to have a company-wide strategy. Then you have to communicate it and train people, etc. These problems look quite simple and easy from the outside, but they’re not quite so simple when you have ~25,000 people. Some people pay more attention to internal communications, some to Social Media, etc.
- Tracking which organization a contribution comes from is a much harder problem than people realize. Today, not everybody uses an email address which connects back to the organization which they’re contributing from. The Fedora and CentOS Stream communities do not ask, and do not force people to list the organization they work for in their profile. This makes it difficult to collect data and do analysis. The upstream communities are thinking about how to solve this, but it’s not solved today.
- In this case, Michal Ruprich was indeed aware that this was an AlmaLinux contributor and how sensitive this was (I checked with him), but he treated this bug just like he would from any other contributor. No special exception was made for AlmaLinux, and an order from management never went out to make a special exception for AlmaLinux or any other contributor. He treated this merge request JUST like he would have if it was a Red Hat employee. Let that sink in. I think that’s fair. I don’t think Red Hat is trying to make special exceptions or to pander for marketing purposes.
- Culture, personality and relationships play a huge role in these interactions. We must understand this. I genuinely believe culture may have played a bit of a role in this as well. A diatribe about cultural differences between different parts of the world is beyond the scope of this post, but suffice to say, I beg people, please consider this with all interactions online.
To summarize, I think the process in CentOS Stream is working. It’s working as it has always worked, and does not treat anyone with special exceptions. There are new interactions happening between members of different communities, and I hope this continues! I think this merge request will continue to be evaluated. It may or may not get accepted, but I very much appreciate that it happened, and I hope we continue to see more interactions, and more learning!
As I’ve mentioned in other places, Red Hat is STILL learning about open source just like the rest of the world.