Background
Recently, the Enterprise Linux (EL) supply chain has been pretty interesting. The announcement of CentOS Stream as an upstream for Red Hat Enterprise Linux (RHEL) in late 2020, the announcements of Alma Linux and Rocky Linux as downstream rebuilds, and the announcement of AWS Linux 2022 being built as a downstream of Fedora are all really big movements in this space. Even going back to 2019, the announcement of Red Hat CoreOS, with CRI-O, as an image-based version of RHEL 8 managed by OpenShift 4 was a completely groundbreaking way to think about managing an EL server. This makes the beginning of 2022 a great time to take a look at the EL supply chain, identify the major suppliers and consumers, and do a bit of public analysis.
I wrote this blog because I think most of the public perspectives, analysis and critiques, in blogs, on Twitter, and such, come from members of the upstream and downstream communities. While I think these community focused perspectives are very good, and important, I thought it might also be interesting to hear the perspective of somebody actually responsible for the business health of RHEL Server. I’ve seen a lot of wild theories on why Red Hat makes some of the decisions it does, and I think transparent articles like this might help quell those. For full transparency, I work at Red Hat and I’m the Product Manager for RHEL Server, so I have my expertise and my biases, but I try to be fair to each of the projects discussed. My daily focus is on paying RHEL customers, and I’m not shy about that, but these are my personal thoughts, and not those of Red Hat officially. I warn you to read this post with your own critical eye, but this is similar to the voice you’d get out of me if we sat down and had beer together.
OK, with that out of the way, let’s take a look at a drawing to help us break down the basic relationship between the major suppliers and consumers. But, before we do, let’s quickly define what a supplier and a consumer actually means in this context. In the Delicate Art of Product Management with Open Source, I propose that open source projects can be thought of as suppliers in an open source supply chain, much like the supply chain in any manufactured products. For example, Volkswagen buys fuel injectors from Bosch and Tesla buys tires from Michelin. In this same way, open source products and projects rely on suppliers, the difference with free and open source software is, you buy it with your time, instead of money. I think it’s a useful model for analyzing the EL space.
Before we dig into the state of EL in 2022, let’s go back a short three years and see what things looked like in 2019.
The State of Enterprise Linux in 2019
In 2019 things were fairly simple. Fedora was the place where innovation and development happened. New versions of upstream code were integrated, tested and used in Fedora. The focus has always been on innovation over a long life cycle..
Then, every 3-5 years, a new RHEL release would snap from an upstream version of Fedora. For example, RHEL 8 was based roughly on Fedora 28. This RHEL work would continue for a period of time, really behind mostly closed doors, and then it would be released to the world as a new major version, along with the full source code. Technically, releasing all the source code is not a legal requirement because many of the packages which are included in RHEL are released under less restrictive licenses which don’t require redistribution (Apache, BSD, MIT, etc). Red Hat releases the full source code because we think it’s the right thing to do.
When a new major version of RHEL is released, the Fedora project builds and maintains a community supported package repository called Extra Packages for Enterprise Linux (EPEL). Versions of EPEL align with major versions of RHEL and often enable RHEL customers to more easily run workloads which have dependencies which are not supported in RHEL. To be clear, these extra packages do not carry the same level of support as those that are in RHEL but they are quite usable and often at a quality level similar to Fedora. Many people rely on software in EPEL because it’s easier than building and maintaining packages themselves, and they’re OK with incurring the risk, because they can always build the packages themselves if necessary.
Once this new major version of RHEL is released, the downstream derivatives like CentOS Linux and Oracle Linux would begin their work and attempt to release as soon as possible. This is pretty much what the ecosystem looked like for RHEL 4, RHEL 5, RHEL 6, and RHEL 7.
Historically, one of the challenges with this ecosystem is that the RHEL derivatives really didn’t provide a lot of value for paying RHEL customers, which means it wasn’t sustainable open source. When a bug was filed in CentOS, or a patch offered, there was really no way to get the code into RHEL. A given major version of RHEL was, for all intents and purposes, a read-only open source project, similar to Google’s Android.
- RHEL Customers could file support tickets with Red Hat and have their voice heard, but there was no solid process for submitting patches or pull requests
- CentOS Linux users could file bugs with the CentOS Project, and they mostly went nowhere
- Oracle users, it seems, mostly waited for Red Hat patches to be released and incorporated into OEL, similar to CentOS Linux, though there seems to have been some cases where Oracle would do the work and patch something before CentOS Linux
- Contributors could submit patches to Fedora, but then they’d have to wait 3-5 years to see their changes land in the next version of RHEL.
As can be seen from the list above, this model from 2019 created misaligned incentives. Most users couldn’t really effect change in a given major version of RHEL or be good members of open source communities and contribute back. This model promoted usage, but not contribution. Looking forward to 2022, I’m happy to say that those incentives are looking a lot more aligned…
The State of EL in 2022
Fast forward to 2022, and the supply chain of upstream and downstream projects and products is quite a bit more complex. This is a testament to the health of the EL supply chain and I’m optimistic that this will promote contribution as well as adoption and growth of RHEL as a technology standard.
Fedora remains the source of innovation for the entire EL ecosystem. In general, packages start out in Fedora and are only included downstream once they are considered mature enough (stability, security, performance, customer need, etc). CentOS Stream is now the upstream for RHEL and development of the next version of RHEL (RHEL 9 at the time of this writing) happens in the open. This is a major change which gives contributors a chance to influence the next major version of RHEL, as well as submit patches during its life cycle. This is a critical difference which makes RHEL 9 very special in my mind.
The CentOS project can now welcome contributors from far and wide, and the changes in CentOS Stream benefit everyone, including RHEL users. This better aligns the incentives between Red Hat and CentOS, fostering a real community. This also gives the RHEL derivatives a chance to contribute to CentOS Stream for their own benefit, promoting sustainable open source (a hot topic nowadays given examples like the recent log4j exploit). If RHEL goes away, so do the RHEL derivatives. Let that sink in…
Now, let’s talk about some of the dynamics with the major suppliers and consumers in this supply chain.
The Major Suppliers
Fedora supplies CentOS Stream and EPEL, CentOS Stream supplies RHEL, and RHEL supplies the rest of the ecosystem. If users want to push the boundaries of what’s possible in Linux, they are welcome to contribute to Fedora, but if users just want to nudge the direction of RHEL, they contribute to CentOS Stream.
In 2022, there’s still an open question as to whether most non-paid users really want to consume CentOS Stream as a replacement for the original downstream CentOS Linux, but there’s definitely evidence of success if you search Reddit, HackerNews, Stack Overflow, Quora, etc. Knowing RHEL from the inside, I think CentOS Stream is viable for most development use cases, and light production use cases. If somebody needs a heavy production use case, making a bunch of money, I don’t know why they wouldn’t want to use RHEL (more on free RHEL later).
There’s also evidence of people not being satisfied with a continuously delivered distribution which releases patches asynchronously, even if those changes are tested, often minor, and generally non-disruptive compared to a standard Linux distribution. For those people, I’d first recommend checking out the 16 node, no-cost RHEL individual developer subscriptions. You can use these personal no-cost subscriptions even if you are part of a larger commercial or government organization. These 16 nodes can be used by the individual developer for demos, prototyping, QA, small production uses, and cloud access (aka at AWS, Azure, GCP and friends). You get full access to patches and with Simple Content Access, even subscription-manager is easier to use. For more information on how to get these subscriptions, check out the No-cost Red Hat Enterprise Linux Individual Developer Subscription FAQ. I have to think that this is a great option for original CentOS Linux users.
OK, with that out of the way, let’s move on to the major consumers in this supply chain.
The Major RHEL Consumers
To simplify this, I’m going to break this down into three classes of consumers:
- Red Hat Consumers
- Fedora Consumers
- RHEL Derivative Consumers.
Red Hat Consumers
First, there’s the Red Hat consumers like RHEL CoreOS and Red Hat Universal Base Image (UBI).
RHEL CoreOS isn’t a standalone operating system, it’s built from RHEL bits, but released, upgraded, and managed as part of OpenShift. As of 2022, it can be thought of as an image-based RHEL 8 container host, managed by OpenShift. A better understanding of how CoreOS works can be found in this article: Red Hat OpenShift Container Platform 4 now defaults to CRI-O as underlying container engine.
If you’re looking for an image-based version of RHEL that you build and release yourself, outside of OpenShift, then check out RHEL for Edge. This isn’t shown in the drawing above because it *is* RHEL. It’s built, managed, installed, and upgraded by the end user. It’s just a different deployment mechanism.
The next consumer of RHEL is Red Hat Universal Base Image (UBI), which is essentially a freely redistributable derivative of RHEL specifically designed for development and production use cases with containers. This makes it quite a good alternative to other RHEL derivatives, when your use case is limited to worrying about what’s in the container image (which is often true for developers). Really, UBI is three things. It’s a set of base images (micro, minimal, standard, init), a set of application images (python, ruby, node.js, httpd, nginx, etc) and a set of unrestricted RPM repositories which lets you update any UBI base image whenever and wherever you want. For more information, check out the UBI 8 Product Page, FAQ, and e-Book.
Of course, I believe that the UBI images are the highest quality and most flexible set of container images available in the market. They’re built on the same trusted foundation as Red Hat Enterprise Linux and they provide container developers/users with access to proven bits that work just as well in production as they do in development.
Fedora Consumers
This is a recent, and interesting breakthrough. Going forward, Amazon has announced that Amazon Linux 2022 and beyond will be derived from Fedora. I think we’re all still analyzing what this means, but it will be interesting to watch.
In particular, I’ll be watching to see if the Amazon Linux team contributes upstream, files bugs, submits patches and drives improvements in Fedora. This would be great as it would create more value for the entire EL supply chain, including future versions of RHEL. I’m hopeful, so let’s watch together.
RHEL Derivative Consumers
This includes the likes of Rocky Linux, Alma Linux, Oracle Linux, Cloud Linux, Springdale Linux and numerous others. I don’t proclaim to be an expert on any of these downstream rebuilds, but they do seem to all follow a pretty standard pattern. They take the RHEL source code, put it back together, and release Linux distributions with distinct “brands” and distinct “value propositions.”
The good news in 2022 is, the builders/contributors of these downstream RHEL rebuilds can now actively participate in CentOS Stream to drive changes which everyone can inherit. It seems like a lot of the leaders in those communities understand that, and are actively participating, including in the CentOS Special Interest Groups (SIGs).
I’m very hopeful and optimistic that this will create a much healthier relationship with RHEL. Now, their users, when they file bugs, submit PRs or write documentation, can drive improvements in RHEL and the RHEL ecosystem. I think this is a fair trade and quite exciting.
Conclusion
Looking back a year since the original announcement that CentOS would shift focus in December 2020, I think we’ve made a lot of progress. I think the upstream and downstream relationships are now solidifying and this is probably the most sophisticated supply chain for open source software in the world. I’m not aware of any other open source product that has created as much gravity as RHEL.
In fact, I’d argue that we are all still learning what this is, and what it can be, together. I think people forget that we’re not done learning about open source, open organizations, and how they scale. I think this is the largest scale example today, but might there still be larger supply chains in the future?
It’s exciting because I think there’s better potential for value creation in the RHEL supply chain, and therefore better value capture opportunities for everyone, including RHEL customers. Let’s look forward to another post in 2023 to see how things have evolved. In 2023, I’d like to analyze some of my predictions on how contributions are flowing, how they’re creating value, and how customers and users are capturing value.
I think this makes a good point (RedHat derivative users can’t contribute back to RHEL) and then misses how CentOS Stream doesn’t really solve that problem.
Insofar as CentOS stream moves RedHat development out into the open, that’s generally a good thing. However, that’s for the NEXT version of RHEL, not the current one. Derivative customers want the current one.
One tiny tweak to the CentOS “Stream” model that could probably make it everything everyone wants it to be: Long-term support releases that match the *current* RHEL releases, but remain very slightly upstream of them. This would mean CentOS Stream LTS releases would be upstream of the current RHEL release, and allow security or major bug-fix patches, which would eventually make it into RHEL after further testing. That’s the kind of useful feedback that the derivative releases want to be able to provide. If it tracked closely enough, this simple tweak would probably eliminate most of the need for the pure derivatives like Rocky.
The current CentOS stream model provides a useful way to influence the *next* RHEL release, but that isn’t the only release that users want to be able to use or contribute to.
Greg, I’m confused by your comment. CentOS Stream has major release numbers just like you describe. There is a CentOS Stream 9 to work on RHEL9.0GA (which is the released yet) just like you describe, but there’s also CentOS Stream 8 to work on RHEL 8.6 (the current version we’re all working on).
As far as I understand your comments about a small change, that *is* what CentOS Stream is today. It’s the upstream to the next dot release of a given major version (8, 9, eventually 10, etc). It’s just “slightly” upstream to RHEL as you describe (which is a great word choice IMHO).
There are essentially 4 batch releases which are 6 weeks apart in any given dot release. For example, we are planning on releasing RHEL8.5 batch release 2 in early February with Podman version 3.4. You can see the upstream branch as well as a package build CentOS Stream today (here).
It’s literally the next batch update being built and tested, so it’s fairly stable AND it’s super useful to help build RHEL. Even I use it, as the PM for RHEL Server (and still containers for now) to check out features and versions before they drop in RHEL. It’s actually easier/better/faster than messing around with internal build systems. This forces internal and external people to work together in a very natural way. Perhaps, I did not highlight this enough in the article and will add more of that information next year when I do the 2023 version 🙂
I still just can’t wrap my head around this change. I read your previous article as well….but what really changes with bringing CentOS contributors to Stream? I understand the cost issues raised by people using CentOS downstream and I’m unaware what payments Oracle provides Red Hat. They’re downstream as well and don’t appear to be affected by this. CentOS had been community-run, if it was such a costly operation to maintain why wasn’t it released back to the community instead of killing it? It wouldn’t have cost Red Hat anything(other than the money used to purchase CentOS and their trademarks) to drop their devs from working on CentOS, right?
That I believe is a large reason for the hate generated, in my opinion. It was like the writing was on the wall for CentOS. “Once Red Hat purchased CentOS it’s days were numbered” was the sentiment I got when that happened. Also, like in your previous article the fact that CentOS changed it’s support cycle for RHEL 8 would certainly and understandably cause some high amount of anger.
When RHEL 9 is released, there’s a bunch of paid and free contributors all contributing to the NEXT release right? RHEL 9.0 isn’t being affected by anything they do unless it’s a significant bug needing to be patched, it’s still a read-only Git repo for everyone involved (RHEL 9.0 in this case). What I interpret from this is Red Hat’s attempt to include the free labor from CentOS into their development cycle and calling it an incentive. Which it is for them. I also under this, this is still open-source software we’re talking about. Many projects are solely on free labor from willing contributors.
The next thing I need to read up on is what is the truly differs from CentOS, AlmaLinux, and RockyLinux before Red Hat owned CentOS in 2014? Right now, it just sounds like it’s still the same CentOS with two new names, so nothing truly has changed for users looking for a free and “stable” Red Hat-based OS option…except of course the CentOS developer base won’t be there yet.
Finally, I am a die-hard Red Hat fan. I also believe it’s a great and very stable OS. I also see the fact that it’s behind the ‘latest and greatest’ in Linux advancements. I just don’t see the point of doing this to CentOS.
I appreciate your comment. I think the biggest difference now with CentOS Stream is that Stream *is* the way to contribute back to RHEL. It’s got its own GitLab repository publicly, it’s own BZs, etc. For example CentOS 9 is already out. Historically, end users would have a really tough time helping modifying RHEL9 before it becomes Generally Available (GA). Basically, RHEL development is now in the open and it’s not just read only like it used to be. IMHO, that’s the most positive thing about Stream.
I think your comment about Alma and Rocky is pretty accurate. Those two, as well as vzlinux, CloudLinux, and others (probably 5-10 different rebuilds) are all quite similar to CentOS before Red Hat merged/acquired them. Now that Red Hat stopped building CentOS downstream, there are actually more choices, not less. And each of those downstreams have the ability to contribute to RHEL through CentOS Stream.
For the record, Oracle does NOT pay anything to Red Hat. There is NO business relationship there at all. From a Red Hat perspective, Oracles Linux is just like any other rebuild. Basically, we are neutral and doing our own thing that “we” think is better. No community relationship, no business relationship.
Hopefully that helps a little.
Hi Greg. Thank you very much for your blog. CircleLinux also a down-stream Distro. LOL !! thanks alot for rhel and centos stream.
I am a Release Engineer for CircleLinux, would like to contribute to RHEL via the centos stream. It is true that this change of enterprise-level Linux can bring more interesting changes to enterprise-level Linux. The downstream redistributions of RHEL are blooming everywhere, which further promotes the progress and prosperity of the ecology.
Yes we indeed should Pay some for opensource productions or software, Avoid the tragedy of fakerjs. LOL!!!
Thank you again ! Best Regards New Year !
Commenting a little bit late I’m afraid!
One thing that stands out in the 2021 diagram, is the relationship between CentOS and Openshift.
One of the stated aims of CentOS Stream was to allow projects downstream of RHEL to collaborate earlier in the process, and hence be more aligned with new RHEL releases as they appear.
From an outsiders perspective, I’ve noticed this happening with EPEL, but not with other projects (for example OpenShift’s upstream projects like OKD & CRI-O).
So I’m curious how well this is working out in practice with Stream 9 and the runup to RHEL 9?
That’s an interesting outside perspective, and I appreciate you sharing it! It can be very difficult to discern how things appear from the outside, or for people not following things super closely. I think the CentOS collaboration is actually working quite well in general for anybody that wants to collaborate with Red Hat. Internally to Red Hat, there are special cases where things need to run on RHEL bits. Then, there are even more special cases like OpenShift. OpenShift has deep connections into RHEL because it needs to control things like networking, and storage, etc. There’s also a huge partner ecosystem which does everything from needing to load kernel modules, to setting specific configuration files, etc. The relationship between OpenShift and RHEL has a very broad surface once the Operator Framework is relied on (and it’s become quite popular for automating these kinds of interactions).
Long story short, I’m not sure there’s been HUGE collaboration between CentOS Stream and OpenShift because they rely specifically on downstream RHEL for a lot of their capabilities. Furthermore, RHEL CoreOS can be thought of as a specific “spin” of RHEL specifically for OpenShift. Stated another way, OpenShift has their own branch of RHEL which is setup and released in a transactional way (OSTree), so CentOS Stream is not an easy way to collaborate, though there are definitely people from the CoreOS team involved in upstream Fedora CoreOS.
Does that help at all? Or, did I just create more confusion? 🙂
Ben, I’m working on the new version of this article for 2023, and was reading through the comments again. I wanted to share this artcle, and I plan to tackle this a bit in my next version: https://cloud.redhat.com/blog/okd-streams-building-the-next-generation-of-okd-together