Update January 3rd, 2022: Please see: The State of Enterprise Linux in 2022
Change is hard. Explaining that change in a way that makes sense is hard. Not getting frustrated when someone is explaining change to you, or while explaining it is hard. We are all human, so please be patient. But, before you get angry, please read this. I’ve been surfing Twitter, Reddit, and HackerNews threads just like many others. I’ve gathered some major buckets of complaints which seem to come up, and I’d like to address each of them.
Before I start, full disclosure, I work at Red Hat and I’ve been here 9.5 years. I’ve never missed a paycheck. Never. Not one time. It’s always been there on payday. That makes me happy, but just because you work somewhere doesn’t mean you check your brain at the door. I am constantly questioning what we are doing, and coming to my own conclusions. A work relationship is like any other relationship, and it must be evaluated from time to time. Like any rational human being, I do this every now and then. I’m also known for saying my mind, even when the truth is difficult to hear. I think this makes me a decent diplomat for tough conversations. That said, these are my opinions and thoughts, not Red Hat’s.
Let’s have a tough conversation here. There are a lot of complaints and some optimism about the CentOS to CentOS Stream change. Let’s dissect each of what I consider legitimate complaints:
#1 This Was Communicated Poorly
As I said in the introduction, messaging is hard. We are all human, even the people at Red Hat. Do I think it’s possible that this could have been communicated better? Sure. That said, I’ve seen a lot of people tossing hate toward Rich Bowen (CentOS Project shifts focus to CentOS Stream) as well as Chris Wright (CentOS Stream: Building an innovative future for enterprise Linux) who wrote the two main pieces which communicated this change.
That’s not fair. These are both stand up guys. I mean seriously. I met Rich many moons ago because I had read and used his Apache docs. I went up to him and said thank you with nervousness. This guy believes in open source, more than most. He not only uses, he contributes. Big time. Same with Chris. I’ve never seen anyone who gets open source more than Chris. Seriously, when you listen to him speak you can discern how deeply he thinks about it. He is committed to it. If Red Hat ever changed their stance on open source, I am 100% sure that both of these guys would leave. There’s no conspiracy here, not with these guys. I know them both personally, and I believe in both of them. When these guys get up and say something, it’s true to the best of their knowledge at the given time. But, the world changes, and so do strategies.
Furthermore, this is a big change, and many people were involved, from both the community and from Red Hat. In fact, I’m writing this blog because I have a bias towards contribution and I believe it’s all of our responsibility to try to communicate this change effectively. Will Red Hat live and learn from this messaging? Yes, I believe so.
My only ask is, please extend some goodwill to Red Hat and see how this change goes. Give it a chance. I believe it will actually be better in the long run, and more sustainable.
#2 Things Changed, I Don’t Like Change
This brings me to the next most common gripe I hear. I get it it. Please try to separate your anger that is a direct result of unwanted change, from the other things discussed in this blog. Every time I’ve ever went through a re-org at work, changed roles, changed companies, bought a house, had a child or remodeled a room in my house, I’ve sort of had this “who moved my cheese” feeling. Everything was one way, now it’s another. Being annoyed by change is legitimate, but ask yourself what the benefits might be?
Already, some people have proposed some benefits to this change, and I agree. I’m going to summarize the benefits I’ve seen from three different sources [1][2][3] and add some of my own thoughts:
- It makes RHEL development more transparent and reliable [1] – historically, once a RHEL release snapped from Fedora it became a black box. Red Hat tries to solve this by creating Alpha and Beta releases which we share with customers and partners. We spend a tremendous amount of time/energy/money (as a PM I have to answer bunches of questions about Alpha/Beta), but literally almost nobody uses them. It’s expensive and painful. CentOS Stream will change this. Now any CentOS Stream user will have access to and be able to participate in this process. This is amazing.
- It provides a way for ISVs and developers to contribute fixes and features [1] – Even when partners and customers would use Betas/Alphas it was difficult for them to contribute changes back. None of the branches/repositories were public. Remember, the source code at git.centos.org is basically read only, downstream code from RHEL. That’s how Red Hat complies with the GPL. Technically we go above and beyond because we are only legally required to provide code to customers, and not required to provide code for BSD/Apache/etc licensed code, only attribution. Nonetheless, this was not a community, this was a code repository. The RHEL code was more akin to Android’s read-only, versus Kubernetes’ or the Linux kernel’s read-write community. With CentOS Stream, this all changes.
- It provides a way for the community to provide feedback [1] – Partners have special people dedicated to them called partner managers, and historically these partner managers help partners provide feedback and drive changes to the RHEL repositories. Community CentOS users (and even Fedora users) never had a way to drive changes into RHEL minor releases. Fedora drives changes into major releases, but there was a black box with minor releases. This gave RHEL the perception of being insulated from the upstream community, and it was. With CentOS Stream, this all becomes transparent. It’s a noble goal, and I genuinely believe this is better.
- Moving to a rolling stream is a consequence of moving to a cloud native world [2] – this was an interesting one and I hadn’t thought about this. This is a consequence of the world of containers. I always understood the value of containers to control when you absorb updates into your application dependencies, but it also means that a rolling stream of OS updates just doesn’t matter anymore. Now that containers control when the updates occur, why even care about whether it’s a rolling stream or released with version numbers. Do a “podman build” after the minor release drops in RHEL and don’t update again until the next dot release – boom, you have something quite similar to downstream CentOS Stream. A rolling stream lets us move faster and freer.
- Don’t underestimate the value of working directly with Red Hat engineering [3] – As Neal points out, CentOS was always a community driven project. Red Hat had extended it’s brand to the CentOS brand, and perhaps mistakenly given the perception that this was a well funded project when in fact it never was. Like every community driven project I have ever seen, there are more wants and desired than time and energy. Maintenance of community driven projects is not wholly unlike non-profits where a sort of depression permeates because it feels like you are never making enough progress, and worse you feel like you’re not being appreciated by the people who benefit from the charity. CentOS stream will be directly integrated with Red Hat engineering providing a direct line to RHEL. These are people paid to do this work, in service of making RHEL better. This aligns incentives and makes it much more sustainable, but more importantly, it creates a natural communication channel between Red Hat engineering and the community which frankly never existed. This will keep RHEL engineering more in touch with the realities of user needs, and provide the community a sense of connectedness. This is a very positive move.
- Don’t underestimate the power picking up a thousand new smart, highly motivated power users with a bias towards contribution, not consumption [4] – I haven’t seen anyone highlight this value yet. There are thousands of Solutions Architects, Consultants, Engineers, Product Managers, Product and Technical Marketing Managers, and Evangelists at Red Hat. None of them had an incentive to talk about CentOS much less contribute to it. I haven’t used CentOS in 9 years because I like getting a pay check. I was annoyed when customers wanted to talk to me about CentOS. I never contributed. I didn’t hate it, but I always viewed it as a wash – it garnered users of Red Hat technology, but these users very rarely contributed. I’ve never even seen a CentOS user file a RHEL issue (aka Bugzilla) or ping me on Twitter for CRI-O, Podman, Buildah, Skopeo, or anything else I work on at Red Hat. These users weren’t even on my radar. About a month ago, I installed CentOS Stream so that our team could collaborate with a customer on some Podman development. I didn’t have to feel dirty about pointing them to a downstream rebuild. With an upstream rebuild, the feedback is all directly useful to RHEL – feature requests, bugs, etc. This is powerful.
[1]: Thoughts on CentOS Stream by Jim Perrin (@BitIntegrity)
[2]: Unpopular opinion: CentOS switching to Stream is not a bad thing at all by Kristian Köhntopp (@isotopp)
[3]: Don’t underestimate the value of a direct relationship with RHEL engineering by Neal Gompa (@Det_Conan_Kudo)
[4]: My own thoughts.
#3 The Life Cycle Commitment Was Changed
The complaint is that the CentOS wiki originally said that CentOS 8 would be supported until 2029, and that this seems particularly egregious since CentOS 6 was recently deprecated. I totally understand being upset by this if you already had your heart set on using this, but think about why you’re upset? Let’s delve into this from an open source product management perspective. To the best of my knowledge, I am the first person to attempt to talk about open source in the context of creating and capturing value (Blogs/Presentations: The Delicate Art Of Product Management With Open Source) – two concepts that are quite common in the start-up world.
Let’s talk about this in the first principles of product management. Something free was meeting your needs (market problem). That solution was not lowering the development cost of RHEL (bottom dark red block), and it was also putting price pressure on Red Hat’s ability to capture some of the value that we create. This is a problem long term. Here’s a dirty secret with technology companies, it’s not about making money, it’s about out growing your competitors. Making steady money is fine for a gravel company, but it’s death for a technology company. You have to grow or die. Red Hat reported 69 quarters of consecutive growth before being folded into IBM’s earnings, and continues to grow within the Cloud Business Unit at IBM. This isn’t coincidence. This isn’t luck. We have to keep focusing on growth. We grow or die, just like any technology company.
Everyone knows that CentOS puts price pressure on RHEL. Let’s all look each other in the eye and discuss it. Many CentOS users speculate that that’s why Red Hat made this change, but let me explain how this isn’t a conspiracy. If successful, CentOS will lower the development cost of RHEL by spreading it across the community. With more eyeballs, all bugs are small. But, it will not remove the price pressure. That’s right, if CentOS Stream is successful, we will still have to compete with ourselves to make money. Why? Because CentOS Stream will likely be ridiculously stable just like CentOS was. That’s fine. That means that as a Product Manager on the RHEL team, I need to be laser focused on providing my customers with what they need. Every. Single. Day. I have to be focused on velocity of the value our team (Podman and container images) provides to RHEL and OpenShift customers.
Challenge accepted. I’ll still make sure RHEL provides value that CentOS Stream doesn’t. For me, it has nothing to do with hamstringing CentOS Stream and everything to do with focusing on creating value in the Buyer’s Problems boxes (light red, and pink boxes in above drawing)!
Could Red Hat have waited until CentOS 9 to make this life cycle change? Sure, I agree that would have ruffled less feathers, but we need these people in the CentOS Stream community. Proprietary engineering is a zero sum game, and the only way to change that is to do community driven development where the cost is socialized among the people who benefit (see also: Why Red Hat is investing in CRI-O and Podman). Any complaints about the stability of CentOS Stream absolutely positively must be looked at in the light of spending money stabilizing downstream CentOS. Without downstream to worry about, upstream will get better. The zero sum portion of the cost of engineering CentOS moves upstream, and becomes socialized. That effectively doubles the chance that CentOS Stream will be great.
#4 The Marks Are Owned By Red Hat
This is a touchy subject. One could make the argument that Red Hat should have never touched CentOS. Over beer, I could be convinced of this argument. I was in sales at the time that Red Hat acquired the CentOS trademarks. Some people hated it, some loved it. I always felt neutral about it. I felt like it was the best existing development platform for OpenStack, which was a problem at the time. The Red Hat contingent working on OpenStack needed something more stable than Fedora because they weren’t interested in doing Operating System development, they were interested in building a platform on top of the operating system. That was a tough problem to solve. Fedora wasn’t the answer because Fedora isn’t the upstream for RHEL minor releases (aka dot releases). I’m sure some people thought about releasing an upstream to RHEL, but it would have taken years and OpenStack was moving at the speed of light. So, Red Hat acquired CentOS.
Again, let’s all look each other in the eye and have a talk like grown ups about this. Once Red Hat touched the CentOS brand, it could never set it free again. There would always and forever be a perception that “Red Hat owns it” – this would forever benefit the CentOS brand and forever incur a cost for Red Hat. Before you slam your laptop shut in anger, let me explain.
First let’s talk about the value that Red Hat gave to the CentOS brand. Before Red Hat acquired the marks, and started paying the contributors, there were always worries that the project might fail one day. Remember in 2009 when Greg Davis disappeared for a while and people freaked out? Yeah, that’s what I’m talking about. There were many similar incidences where updates were slow, rebuilds of new versions took longer than expect, etc, etc. But, when Red Hat took over, all of those worries went away. That made a lot of Red Hat customers feel a lot less worried about running development, test, and even production workloads on CentOS for things they didn’t feel they needed support on.
There’s a big problem that almost nobody understand. The RHEL update stream has a huge value proposition, and most people assume the CentOS update stream is built by the same people, or at least people that can communicate with the people that build it. Let’s be honest. But, the updates in CentOS were never produced by the RHEL engineering team. In fact the packager on my team never, ever talked to the CentOS people, and the CentOS people never talked to him. There was no transfer of knowledge going on. The CentOS updates were produced by a few community members who rebuilt them. Doesn’t matter, the world perceived it as “updates from Red Hat” – I know it with all of my heart. That could never be undone. Never. If we let the marks go, some other people would fire up a rebuild and capture that value in the community. It’s fair that Red Hat didn’t want to give up that value capture.
Second, when Red Hat acquired the CentOS brand, we automatically incurred a cost – forever. Even if it’s just answering questions about how we gave CentOS back to the community and don’t patch it. I remember when I was a Solutions Architect and the CentOS acquisition occurred, there was an expectation from my customers that I would tell them about the value of CentOS. If you think about that, it’s completely insane. Why would a sales person come in and tell you about the value you can capture with a free project? It’s fine to capture the value, but asking Red Hat about it is a cost we incur. Forever and ever. We will always have to answer questions. This is a weird one, and to be honest, I don’t think anybody could foresee how expensive that is. If there’s a cost incurred, there needs to be value captured. With CentOS upstream it’s probably net-neutral from a value perspective. It will lower RHEL development costs, but incur sales costs. That’s better than a net loss.
I genuinely understand why this one makes people mad, but I also understand why Red Hat would have heartburn giving these back to the community. Open source is a tough business.
#5 CentOS Stream Support is Only 5 Years
This is an interesting one. It’s a fair critique, and to be honest, one I hadn’t internalized this when we made this announcement. Let’s be fair, this is basically identical to what Canonical does with Ubuntu LTS. You pay for the later year updates. This has a very good logic to it. If you are running something for that long, it’s probably a cost/benefit analysis and not some upstream development. That means you’re making money off of it. In the parlance of product management, we discuss these concepts as creating value and capturing value. What you might not realize is that whatever workload you are running on Downstream CentOS is creating value for your business, especially in years 6-10. Moreover, you’re likely charging for the service you run on downstream CentOS and capturing value. This is a leaky value chain that isn’t sustainable.
I mean, I’ll admit it, I’m a lazy sysadmin at heart and have had blogs, wikis, and ticket systems run for ten years before, so this isn’t 100% true. Not everyone running old-ass shit is making money, but you get the point. This is a key area of value capture for RHEL and it doesn’t feel wrong to me to charge money here. It feels fair. Neither Ubuntu nor SUSE Enterprise Linux have freely available, highly promoted downstream rebuilds which are supported for 10 years, built and paid for by the companies that sell the upstream products. This actually feels like madness right?
Tangential to the life cycle complaint, that Red Hat should have waited until RHEL 9 to make this change, I get why people are mad about having an expectation changed. That said, providing maintenance for CentOS Stream 8 until 2024 is still pretty damn good for $0. Also, remember, when you run something for $0, you take full and complete responsibility for it. With open source, you have the power to do whatever you want with it. You’re not trapped. That’s the power of open source, and also the responsibility.
This five year support debate is really about value creation and value capture – what should be given away for free, and what should be charged for. That’s strictly a business decision, even though I know it’s annoying. I’m not some hardcore Libertarian that likes to get on the “responsibility soap box” but I see the problem in the CentOS space and it’s even more egregious in the Linux container space. Companies are consuming container images for $0 without realizing they need to have the expertise to build/rebuild them if the upstream project ever disappears or decides go a different direction. One time, I talked to a partner that wanted to move to UBI, but they didn’t know how to move a bunch of their software because they just consumed container images from upstream without knowing how to rebuild them. They were selling this solution to their customers. Let that sink in. This is a serious problem in the cloud native world. You must ruthlessly track and gauge the risk you incur with by consuming upstream dependencies.
When you consume community open source software like CentOS, you become a product manager like me whether you realize it or not. I don’t mean to be insensitive, but consuming CentOS for $0 instead of buying RHEL means you are responsible for the life cycle even if somebody else was making your life easier by doing it for you. You are only in a bind if you are paying $0 and have no contributors.
#6 CentOS Stream Will Be Less Stable Than RHEL
There’s a perception that CentOS Stream will be less “stable” than downstream CentOS. This could be rephrased as, “this pisses me off because I always relied on paying RHEL users to absorb these changes, test them, and file bugs for me!” In reality, this move basically swaps who gets changes first, but the changes themselves were never “unstable.” Assuming these changes are unstable is essentially a conspiracy theory. Here’s why.
If Red Hat was upsetting RHEL users with a bunch of unstable live testing, how would Red Hat even have paying customers? The fact is, every dot release, RHEL users have been absorbing these same changes without a problem for years and years and years. How is this possible? Do you want the blue pill or the red pill?
The truth is Neo, with RHEL, there never really were dot releases. Not in the sense of traditional Unix numbering. RHEL has always been quite similar to a rolling release within a major version. With the launch of RHEL 8, we made this even more explicit with Always Ready RHEL It’s just that few people understand it. For years, we’ve encourage customers and partners not to park on dot releases. We always tell them that there is forward compatibility built into the architecture, but they never listen. I genuinely think this comes from how other software vendors have trained them. The stability of RHEL has always come from it’s architecture, and now that’s verified with tests and gating.
First, let’s start with the architecture. Many vendors don’t adhere to Semantic Versioning. RHEL does. If you don’t feel like reading that Semantic Versioning (SemVer) page, it’s the three digit version number that is so commonly used in software. It works like this. Given a version number MAJOR.MINOR.PATCH (ex Podman 2.0.5):
- MAJOR version when you make incompatible API changes
- MINOR version when you add functionality in a backwards compatible manner
- PATCH version when you make backwards compatible bug fixes
Software in RHEL falls into four levels of compatibility. If you’ve never heard about this, or don’t understand, check out: Red Hat Enterprise Linux 8: Application Compatibility Guide. All of the key pieces of software in RHEL like the kernel, glibc, openssl, are in compatibility level 1 or 2 which means they are binary compatible within a major release. Stated another way, changes to compatibility level 1 or 2 software are in the category of patches, not major or minor changes (there are some minor exceptions when a rebate doesn’t break compatibility). The main reason the RHEL dot release increments is because some of the higher level pieces of software in compatibility levels 3 and 4 can change. RHEL dot releases are nothing more than a bunch of patches bundled up on a certain day, ran through quality engineering (QE) testing, and tagged together.
This doesn’t change with CentOS Stream. It’s architecture must adhere to these API/ABI promises to front run RHEL.
How could it not? CentOS Stream wouldn’t be an upstream for RHEL dot releases if it deviated from these rules. The stability of RHEL starts with the architecture, application compatibility promises, and back porting.
Second, let’s talk about testing. You can’t test your way out of bad code (design, bad programmers, etc). RHEL starts with very high quality code and limits the variance of any change which can be accepted into the pipeline. Each change builds off of the last stable version. This is the foundation which makes RHEL stable. Even with this filter in place, each change then goes through rigorous testing and Quality Engineering (QE).
Humans do QE which improves stability and gives developers feedback, while automated testing verifies that regressions haven’t popped up in the code. Automated testing doesn’t necessarily make software more stable per se, it allows teams to move faster while retaining the same stability. Stated another way, gating tests are designed to catch things that have broken before. They prevent major flaws from re-entering a piece of software before it’s committed, but let’s also be fair about the limits of CI/CD.
CI/CD has never caught more than a small percentage of bugs with any software (even with 100% code coverage). The magic of automated test plans is that teams can move quicker, break things, and fix things while retaining quality. They can add new tests quickly so that teams don’t break the same things over and over again. Automated testing works because there’s an implicit assumption that things break at the weakest link. Testing verifies that those weakest links haven’t cracked again. None of this changes with CentOS Stream. Brendon Conoboy does a great job of explaining the multiple tests and QE that CentOS Stream goes through in this post: How RHEL is Made. CentOS Stream is essentially always ready RHEL. Stef Walter walks through the vision and the journey it took to get to this place with this blog: CentOS Stream is Continuous Delivery.
There’s argument being made is that CentOS Stream won’t be stable because it isn’t tested, or it’s “beta RHEL” which is just patently false. Don’t listen to Brendan, Stef or myself, who all work in the RHEL team, listen to a CentOS contributor named Carl George: “Yes, it is. In fact, large portions are tested twice. Everything released into CS8 has already been through RHEL8’s internal tests as well as the public t_functional test suite.” CentOS Stream is tested quite rigorously before it’s released to users.
Historically, if a bug made it past RHEL gating tests, it was automatically, and naively picked up by downstream CentOS. Today, if a bug makes it past CentOS Stream gating, it’s likely going to get automatically and naively picked up in RHEL. This effectively swaps RHEL and CentOS in the update stream. If RHEL users weren’t getting unstable changea before, there’s no reason to believe CentOS Stream users will get unstable changes now.
The RHEL team doesn’t want public Beta testers with CentOS Stream, we want community members that will help drive the next cool RHEL feature. The RHEL Alphas/Betas are/were rarely used, and were never the key ingredient to stability. The stability has always come from a combination of the architecture, high quality engineering (open source), back ports, ABI/API compatibility for level 1 and 2 software, and verification tests. Our competitors have never been able to achieve the same thing because they refuse to do the same level of work. Back porting is. Not. Fun. Let’s all be honest. But, it’s super convenient to consume. CentOS Stream users will still be able to run a yum update with little worry of pulling in breaking changes. The stability of CentOS Stream should be effectively the same as downstream CentOS for most users. This should be mostly a non-event. Your anger likely stems from not understanding this.
#7 You’re Not Looking For Opportunities to Improve This Ecosystem
RHEL and CentOS have become tied at the hip from an ecosystem perspective. This especially became true when Red Hat acquired CentOS, but you’re not looking for opportunities to improve this. Let me provide an example from my perspective.
The Container Tools team in the Red Hat engineering (Podman, Buildah, Skopeo, CRI-O, Red Hat Universal Base Image) releases our main application stream (container-tools:rhel8) about once every 12 weeks. We move faster than most subsystem teams in RHEL. Our upstream projects are still young tools and adding features very quickly. Our customers are dying to get the latest versions of Podman. Often, our customers are happy to get access to new versions of Podman even just a weeks earlier, especially when there is some killer feature they want. Historically, I had to point our RHEL customers to Fedora (which I love), but the configuration is often different than RHEL (ex. cgroups V2 on by default). Worse, our team doesn’t commit to Betas. Since we release every 12 weeks, the version on the Beta would already be old. This means, my customers had to wait until the RHEL dot release, which is painful.
Before the release of RHEL 8.3, I was able to share Podman 2.2.5 configured on a box that looked pretty much identical to RHEL (no cgroups V2, same registry config, etc) weeks before it dropped in RHEL 8.3. This gave us user feedback that will be used for our next 12 week release, as well as RHEL 8.4. Historically, we had to wait for feedback until RHEL 8.4 dropped. This was very painful because the upstream Podman team is moving so fast and the version on was never up to date. With CentOS, our problem is solved for both our team and our customers.
Furthermore, let me propose an idea. A place every project struggles, is with docs. I am constantly giving user stories to our documentation people, and they are constantly having to stack rank them, prioritize them and complete the pieces that they can. Imagine if we can move our containers guide (BUILDING, RUNNING, AND MANAGING CONTAINERS) upstream and have the CentOS Stream community contribute docs? That would be amazing and socialize the development cost for my team. This is a concrete example of the types of ideas that will come down the pipe if we all work together.
Look for the opportunity, not the challenges.
Conclusion
This change will be difficult for some, and I have empathy for that. Know that we are listening. I’m listening and available on Twitter @fatherlinux. I’m sure there’s still going to be a lot more anger, but hopefully this perspective helps people see the benefits, and perhaps better understand why these changes were made. Give it a year, and lets see how this things turn out. It’s really up to all of us to make this work. I want to leave you with a something funny, and something sad.
Let’s start with funny. Apparently, somebody spent the time to make this website: centos.rip. I’m not going to lie, I literally laughed out loud when I saw it. I mean, like really hard. I’m fine with people trolling Red Hat about this change. All is fair in love and open source. This is funny.
Now, let’s talk about something sad. There’s a change.org petition (Do not destroy CentOS by using it as a RHEL upstream), which I think is on the line because this is for things like wildlife being killed or rain forest being destroyed, but fine. I don’t have any heartburn with people being angry and expressing it, but I do have a problem with people comparing this change to the Charlie Hebdo Shooting. The Je Suis CentOS meme I’ve seen floating around is just disrespectful to the people that were murdered. It feels like people are just trying to find things to go viral.
Right now, children are getting blown up in Syria. If you’re in an emotional state where you feel the urge to make comparisons between a free software change and mass murder, please spend your energy and money on something that really helps the world. I urge you to donate to The Free Burma Rangers. They deliver food and water to people in need, even in war zones where normal NGOs won’t and can’t operate. This is a problem on the scale of Charlie Hebdo, or global warming, or the deforestation of The Amazon. Don’t make these comparisons lightly.
Finally, I’m not trying to dismiss your anger, but moving to CentOS Stream is literally three commands (How do I migrate my CentOS Linux 8 installation to CentOS Stream?). I understand that it’s annoying to have to migrate servers, even if moving is super easy, but put this in context. This CentOS change will cause you to do something that you might not want to do, but there are a lot of potential positives and I hope you will focus on that and try to help make it better.
At least this change isn’t as bad as children being blown up in Syra? Your conclusion was amazing. I would just like to say that this may be the worst explanation from a company employee regarding a controversial decision that I’ve ever seen.
I was not making a comparison between this change and children getting blown up in Syria. I’m saying that people creating memes like Je Suis CentOS is just disrespectful to the people that died, and trying to remind people that problems that bad are happening right now, as we have the luxury to have this debate about a free software change. I have not problem with people being mad at Red Hat. I have a problem with righteous indignation stemming from a sense of entitlement with a blatant lack of self awareness for how lucky we are.
I’ve cleaned up the last paragraph to try and better reflect what I think. I wrote it quickly and I don’t think what I was trying to say was coming through.
The last paragraph is dirty pool. Almost all of our (professional computer touchers in the United States) troubles pale in comparison to those faced in war zones. Whatever pressures Red Hat is feeling also don’t matter in comparison.
Correct. That’s my point. Comparing a free software change to the Charlie Hebdo massacre is just disrespectful to the people that died. Troll Red Hat, fine. Get mad at Red Hat fine. You can’t control if you’re angry, but you an absolutely control how you respond to it. Have respect for real problems. This truly isn’t as bad as children getting blown up in Syria, so I’m encouraging people to realize when they make these comparisons. It’s a display of righteous indignation that seems to stem from a spoiled sense of entitlement. I absolutely appreciate that my children are not getting blown up, and I do donate money to that caus, so I figured I would use it as an opportunity to highlight it.
Thanks for taking the time to write this article. I am genuinely curious to know how much time it took to gather your thoughts, research the referenced articles, and then type/post. Basically how much time do you have in it total?
As far as an opinion on the article, it’s still sinking in. It was a good read, little dramatic, which helps communicate how personal it is.
“If successful, CentOS will lower the development cost of RHEL by spreading it across the community.”
Does Red Hat have resources (paid employees) who’s full time job is to contribute to CentOS Stream? If so, is that because it is upstream and can provide value to RHEL? Were there no resources assigned to CentOS because it was downstream and this provided no value or would result in a net-loss?
I have been closely following the Red Hat jobs/careers posting for several months and in hindsight can see that positions may have been posted to work on CentOS stream team, if there is one, with out explicitly mentioning it.
Are there other projects upstream or otherwise that Red Hat assigns employees to? If so, which ones? I’m part of the OKD community and believe there is at least one person fully assigned and maybe another partially assigned.
I’m not saying it’s a bad thing to have these folks in place, but more clarity/transparency from Red Hat would be beneficial, and help alleviate any future conspiracy theories.
I’ll try to address each of your questions. I work in the RHEL product team, so I understand how RHEL is built, tested, released and used by customers. Even with this knowledge, it took me a couple of days to gather my thoughts and publish. Even for people internal at Red Hat, even in the product teams, we each have to internalize this and figure out how it affects us and our products. Obviously, for me, this change is quite close to home.
Red Hat does pay tons of people for upstream projects, ranging from Kubernetes, to Fedora, CebtOS and thousands of smaller projects.
I’m not specifically aware if we are hiring more people for CentOS Stream, but I can find out. It definitely doesn’t seem unreasonable or unlikely.
Red Hat is pretty transparent about hiring people who’s job it is to work upstream. We even have an entire program office dedicated working on and in upstream communities.
Their old name was Open Source and Standards (OSAS). I think theyncha he’s their name, but like anyone, I struggle with change 🙂
The team is called “OSPO’, Open Source Program Office. I work there and we still have OSAS around (github repo, etc). Small team of 20 ful time person + 2 interns. And we are not even the only team dealing with community at RH.
Well, I do understand RHEL perspective.
It makes total sense.
FOR RHEL. NOT CENTOS.
You use centos as a test environment for RHEL features, so centos would also receive security patchs first, right? WRONG. You’re a guinea pig, no nice security patchs for you, only testing stuff.
If a stream rolling release is sooo good, RHEL would use it. You don’t make an incredible revolutionary product and hand over to the next guy. Just to the fall guy.
You rent a house today with a 10 year contract and next day the owner come and say GTFO. You can’t trust someone like that, you lose respect.
RHEL base, if much, bears 1% in the market share and will likely decrease and certainly lose relevance.
I just hope RHEL dies alongside CentOS stream.
When you rent a house, you are paying money. RHEL is a house. When you use CentOS, you are sleeping on your beat friend’s couch and eating his favorite pop tarts. If he asks you to throw money in for food every now and then, you don’t get mad. You don’t have a contract.
If RHEL and Stream dies, so does Rocky Linux. I don’t want to sound cold, but it is. This is business. People don’t buy software/support because they get an emotional reward from it. They do it because it provides concrete value.
IMHO, CentOS Stream will be better than CentOS and this is a “who moved my cheese” comment.
You do realise that this affects scientific communities as well who do not make a dime out of it?????
I do, and I am sympathetic. There are definitely more no/low cost options coming. Today, there are already ridiculously low priced options for HPC workloads.
Ah, “Who moved my cheese”, the textbook on managerial gaslighting. I’m sorry, but I think you’ll find people tend to not respond well to gaslighting when they’re not on your payroll, and it’s also not going to convince people of your arguments.
I mean sure, that’s one perspective. Here’s an alternative perspective. I worked as a contractor for NASA when I learned about Who Moved My Cheese. Let’s put it this way, it wasn’t really a profit driven enterprise. In fact, most days people were depressed because they couldn’t get things done. Most good ideas died in Bureaucracy. I think management was concerned about moral, and knew we needed to move faster (even in 2002ish when I first read that book). It was about letting the innovators move quicker, while nudging the slow movers along without letting kill the whole organization. There was a LOT of slacking going on at NASA in 1999 to 2006 when I worked there. A good friend of mine told me one day, “leave here, before you are ruined!”
It broke my heart really. I still love the mission. I was one of the 5% who pushed hard to move things forward. Mostly, we were fighting against lazy people, fear, and subconscious bias against change.
Or, maybe you’re right, the whole idea was to scam and really the 5% that wanted to do things were just gaslighting everyone. I went on to be quite successful in private industry, so I suspect you are wrong. My 2c.
Hi fatherlinux,
We are a small manufacturing company, and is using Centos and some ubuntu. I work as a system admin. The effect of this change for me is two: Use full ubuntu, or use AlmaLinux or RockyLinux. Actually this doesn’t affect me technically since if I have CentOS 7 running, I could just replace it with the two distro mentioned.
The change still doesn’t really matter for users technically, in my opinion.. Those users who are upset, were emotionally affected by it and is afraid and not comfortable to migrate to AlmaLinux or RockyLinux. Yes, change is hard, but the justification of that difficulty or hardness is justified, that in the software world, you do not change anything unless it is a really needed. Some of your end users felt that the change is not needed. However,
However, I do not agree with the analogy being used by you and the author above you(Felipe). A house for rent is not obliged to provide free rooms, couch or even food in a community. It is just business, people pay you rents or they don’t and they can’t sleep on couches and access the fridge to cook if they don’t pay you.
But in RH’s case, they are obliged by the GPL license to release all the source code as part of RH’s obligation to the GPL license they are adhering into, that given if the original source code is governed under GPL(regardless of who are the authors). And once they downloaded the code they will build in their own kitchens. RH’s cost is the storage for the repository of RHEL code in this case. RH may provide building tools and other free stuff for the community as mentioned here that RH has participated in thousands of open source projects. But the end game really is to monetize those projects by providing support. Everybody needs to eat. That’s the business of RH. So its a win-win actually. And if you provide the free tool such as CentOS, and you stop suddenly, what will you get in return?
I think, it was handed poorly, RH may stop “future” releases of CentOS, that would be fine, but stopping the development of a version that was already released publicly and is guaranteed to be supported for a certain period of time and you cut it off, RH is sending a bad image to the community that it will find difficult to recover. The people that will trust you from that point are those paid users.
As they said, you need exert 6 times more effort to win a customer back.
Regards,
-Allan
I totally understand your position. I do agree that we could have handled it better. Technically, not everything in RHEL is GPL so we don’t have to release, but we *do* go above and beyond and do release everything so that an Alma Linux and Rocky Linux are easier to build. We just don’t want to be in the business of building a downstream anymore. It’s expensive from a resources perspective and these small projects run on both less and more people than you’d think. It’s expensive than you think. The few people is actually the less expensive piece. Having thousands of engineers take their eye off the ball and focus on things that don’t contribute to the development of RHEL is extremely expensive. Now, with CentOS Stream, Red Hat engineers and community contributors are always focused on something upstream to RHEL. That means RHEL benefits from all of those users, conversations, modifications, tests, etc, etc. When CentOS was downstream, it was “just free RHEL” and that’s how people treated it. Nobody respected it. They just consumed. Now they are mad about the world changing. I hope they contribute to Alma and Rocky so that those projects continue to flourish.
Nope, if Redhat dies – AlmaLinux and Rocky Linux will continue to live.
What happened to Mandriva? Is OpenMandriva, Mageia and PCLinuxOS Dead?
NO and that’s how open source works.
Perhaps the truth is that RedHat isn’t cutting it and deserves to die so the future will rise.
How on earth would Alma and Rocky live on if Red Hat dies? Those two projects rebuild RHEL. If there’s no RHEL, there’s no Alma, and no Rocky. They would have to start from scratch and build a Linux distro. It would no longer be compatible with RHEL if there is no RHEL.
To be honest I quit reading midway, because first part was already quite “moot”.
The only reason why many companies use CentOS is to piggyback on Redhat’s stability. You change CentOS into something else, no enterprise user will be interested. You say its not fair, but eh I am pretty sure this is also an advantage for Redhat. We have no need for any support other than what is available publicly. But still we do have some products which we pay Redhat for. And dont forget as a CentOS user we also do bug reports for RHEl…
All the internal issues you mentioned, you could have left CentOS as it is and started Redhat Stream, or Fedora Stream and you would have achieved exactly the same. Because at the end that is exactly what CentOS is going to be. So be a man and just admit you guys killed CentOS for no good reason for its users.
I am not mad at Redhat nor IBM, this is not the first time a RHEL clone got killed and probably wont be the last.
I have one criticism towards users of CentOS, when Redhat acquired CentOS, that was the moment that the community should have came up with an alternative, because deep down you know this was going to be a conflict of interest. So I hope we all learned our lesson and dont let it happen again.
And there’s nothing to be angry about, there’s nothing Redhat nor IBM can do to prevent yet another RHEL clone.
disclaimer: I am responsible for a fleet of ~5000 baremetal servers with CentOS.
+1. My thoughts exactly. Why touch CentOS? Could of just created RHEL Stream instead and you would achieved the same goal.
Read my other comments. There is a cost zero sum game. The computer and human resources for downstream CentOS are significant and like the users complaining, we can’t just come up with new budget out of thin air to put the same investment in upstream.
If you can’t find budget why did RedHat buy CentOS in the first place?!?
We have the budget that we have. My point is, choosing between Upstream or Downstream is a zero sum game from a dollars perspective. Equally investing in both for 9 more years, just isn’t a viable option. I get that.
Hehe, and that was unpredictable when Redhat acquired CentOS? Hehe. But if what you say is true, it means Redhat/IBM decided CentOS as it is; is not worth the money for its users. I guess this is the real truth and its fair. But that also means it accepted that users will flock to something else and/or competition. That is mindblowing from my perspective.
Still a moot case.
Why in the first place buy CentOS if you do not want to maintain it anyway in the long term?
I read your words, but we all know it was incredible stupid. Rh will loose market share, for short term profit. Do you really think I will use a red hat product like openshift now? Maybe next month you will change the license. Rh is now not better than oracle, IBM and the rest. We (2000 bare metal server for public funded research) will find a way to use free software without annual fees.
The goal of this was never to force people to buy RHEL. There are several ways a company can make money. 1. trimming bottom line cost 2. Driving top line revenue and 3. Investing in innovation. This is about #3. My business unit doesn’t have increased sales goals because of this move. Think about how weird that sounds. Our sales goals are exactly what they were. This is a mid-year move that has nothing to do with driving revenue growth. I would say it’s about shifting costs from bottom line, to innovation.
I wanted to add, I am thankful for Redhat for all the things the company has done. I started using Redhat in 97 and pretty much during my whole career Redhat played a big role. But you cant blame people for feeling betrayed right now, starting by acquiring CentOS, getting acquired by IBM and then kill CentOS as it is. I wouldnt be surprise people will get suspicious now regarding the other redhat products. And this is sadly not a good thing for Redhat and us as users.
Alex,
I also started using RHL back in the 90s — before there was RHEL! We be old! 🙂 I wouldn’t be where I am today without Red Hat, so am also thankful.
This feels like a betrayal of trust b/c it’s taken right out of the Embrace-Extend-Extenguish playbook: embrace CentOS Board, extend with CentOS Stream, extinguish CentOS Linux. Whose to blame? That’s what I still want to know…
Jp
I think that’s a fair criticism, and I definitely understand the sentiment. I just don’t think it’s extinguished. In fact, I think it will burn much brighter, not be extinguished. Lets look back at this in a year and see where CentOS Stream goes.
I’ve only touched CentOS lightly a few times. My choice of server OS is OpenBSD. I’ve used Fedora frequently. But my Linux distro of choice is Slackware. Granted, it’s been at least four years since the 14.2 release. And CentOS users are complaining?!?? If there’s anyone who should squawk, it’s us Slackware users. The repositories are still active. But trying to stay true to Slackware is almost comparable to the Trump supporters not budging on the outcome of the 2020 Election.
All fair comments, and I see why people, especially in 2020, are suspect of changes like this. This was NOT driven by IBM. They literally play no tactical role in our product and project decisions.
This is all public knowledge, but IBM’s biggest push is on OpenShift to support a large ecosystem of software. They are not driving changes in CentOS, etc.
this line sums it up
Here’s a dirty secret with technology companies, it’s not about making money, it’s about out growing your competitors. Making steady money is fine for a gravel company, but it’s death for a technology company.
yes very very very true
Hi,
Thanks for the detail explanation and covering redhat. As one of the user correctly said, redhat would have created a RHEL stream as upstream project and leave the CentOS as it is.
So, surely it’s no secret from Redhat/IBM to get rid of a solid OS so that more RHEL subscriptions can be sold.
Thanks.
It’s more about moving the community that sticks around upstream. If there’s going to be hemmorhaging, it’s best to make the change fast and crisp. IMHO, I think it’s 6 of one and half a dozen of another. Waiting 9 more years to make this change wod eat up a ton of resources from Red Hat.
The correct time to announce this would have been at the release of RHEL 8, 14 months ago. If this is primarily a decision based around getting more upstream contributors, and not a post-COVID cash grab, surely it was already being planned then?
I can’t really deny that. I also think it would have been better. I cop to that being a mistake. Just know, it was an honest mistake, not on purpose to screw people over.
That sounds like you think red hat owns the community.
I was pretty clear that there is the perception that Red Hat owns it, so there is forever more an expense incurred for CentOS. This is true regardless of whether Red Hat continues to invest money in CentOS (value creation), or generate a wider RHEL ecosystem (value capture). I’d like to point out that #2 was not happening. Red Hat did a ton of investment in CentOS (value creation), and received nearly nothing in return (value capture).
The anecdotes about CentOS users claiming to move to RHEL are a triumph for the human spirit, but simply aren’t true at large enterprises with thousands of installs (the place where Red Hat makes money). Red Hat can’t thrive on one or two users going from CentOS to RHEL.
The biggest question still has not been addressed or answered. Would you utilize centos steam as a production product in your organization? Everything leans toward development and testing, but none of this inspires use of centos in a corporate production environment. So now companies using centos in production will hit a fork in the road: pay to switch to RHEL or switch to another product.
I can’t answer that for you. I never could. IMHO, running “production workloads” – aka workloads that either make money or save your company money – on CentOS was crazy. That hasn’t changed one way or another.
If you can’t answer that question after reading this blog, I’m genuinely concerned that you should have never a made a decision to run on CentOS.
That said, if you “though” CentOS was good enough before, you’ll likely think CebtOS Stream is good enough now. It’s three commands. Try it out. I think you’ll be surprised. Our customers have been absorbing this change first for years, and they never complained.
We use CentOS in production. We paid for RHEL for several years but never used support because we support ourselves. We would gladly pay a small sum to use RHEL without support but with updates (access to the repositories).
That’s already possible today. If you talk to a sales rep there are ridiculous deals for update only support and Developer subs that are “lightly supported” – that said, they are going to announce more streamlined versions of these. I don’t even know the details yet, but stay tuned (my focus is on Containers, not our SKUs and support).
How about for customers who dont utilize support, but have thousands of machines? A developer sub wont due. If I turn around and ask management for 4.8 million dollars so we can license everything, do you think I wouldn’t be laughed out of the room?
We use RHEL on our backend infra, but we dont use it in remote facilities because the sheer number of them would hurt us in licensing costs. Watch what happens, everyone will move to Rocky and/or Cloud Linux. The engineers who were running/learning your product to support customers around the world will start to fizzle out. You closed the doors to us at work, why would we experiment on our own time?
Why would running production workloads on CentOS be crazy? A lot of us are Red Hat Certified Engineers, do you think Red Hat support is more capable? Outside of that, there is no difference. Its a rebuild.
You’re assumptions are way off. Thousands of developer subs might cost a few thousand dollars a year even with today’s programs. The fact that you don’t know that is the one thing I’ll blame Red Hat for.
Furthermore there are all kinds of pricing models for HPC and remote fleets.
I respect RHCEs a lot. I did my RHCA back in the day, and I’m still proud of that. It’s not about being better then RH support or worse, it’s about the relationship that gives you access to drive change. Engineers are not always good at assessing tail risk, that’s why we have finance and procurement people in big companies.
This is one of the biggest issues with the messaging. of this change. Waiting to announce the new license options was a mistake. The CentOS announcement should have gone hand-in-hand with that change, IMHO.
Totally fair criticism. Totally fair.
That’s the thing – You don’t need to go through a sales rep to get a community supported version of Ubuntu. You just download it.
I know contracts are RH’s bread and butter but you have to understand that this is going to drive a lot of people out of your ecosystem. Startups, small shops, and non-profits are going to bail hard for Ubuntu, and when they get big enough to need and afford support, they’ll already be out the door. And as much as you might see Stream as a drop-in replacement for CentOS “classic” – I don’t think the majority of the CentOS userbase does. That’s the issue here. Hopefully Rocky fills that gap for you and allows people to stay in the ecosystem long enough to want to actually upstream things – by which point they’ll want some Stream and RHEL boxen, but it will come at the cost of a massive reputation hit.
Ask yourself a question. What happens when the money runs out. In December 1999, Mark Shuttleworth sold Thawt for $575M ($846M in 2019 dollars). If he had thrown that money into an S&P 500 Vanguard fund (VTI), it would be worth $3B today. Instead Mark Shuttleworth has a net worth of $500M accordinging to some sources, and as low as $300M according to some others. Ask yourself if the Ubuntu model is sustainable. Ask yourself.
That aside, Ubuntu LTS is only supported for 5 years without a subscription. The company is barely profitable on a good year, and not profitable on bad years. They’ve had huge layoffs which haven’t affected Mark Shuttleworth’s lifestyle but has surely affected the life of the people that got laid off. Suse just had lay offs right before Christmas, and barely a blip in the media (given how bad the rest of 2020 is, it’s kind of understandable).
Having a business model that is sustainable is nothing to feel ashamed about. Red Hat creates more value than it charges for. I feel like that’s totally ethical. I’ve published a lot more on my views of building an ethical product here: http://crunchtools.com/the-delicate-art-of-product-management/
All that said, we just never saw CentOS users become RHEL users. It VERY rarely happened. It was way more common for things to go the other way. The downstream nature of CentOS was never healthy. To be fair, we are ALL still learning here. An open source software ecosystem as big as RHEL has never been done before. It will be seen if this was a good move or a bad move, but overall I think it’s a good move. The right answer is to make RHEL more accessible, not to have downstream rebuilds. My 2c.
“I can’t answer that for you. I never could. IMHO, running “production workloads” – aka workloads that either make money or save your company money – on CentOS was crazy. That hasn’t changed one way or another.”
Sorry, wrong. That’s exactly where you took the wrong turn. And your statement being untrue, is exactly the reason why CentOS shines. That difference in understanding is exactly, why the community is so upset. CentOS is far better and far more reliable than your perception allows.
By leaps and bounds.
It’s exactly why Red Hat should reconsider. Because if not, you dumped your CentOS investement and strengthened anyone else, while weakening Red Hat. I am aware you disagree. That’s why I take it with your words: Let’s see in a year or two. Maybe we will all be surprised when fecals interfere with the autonomously rotating propeller behind a mesh.
Just my five Cents.
Maybe Rocky, may try CentOS Stream, will reinstall OpenBSD (the only OS seriously withstanding harsh changes 🙂 ).
I work at RH in the Community Infrastructure team, part of the Open Source Program Office. My team provides infrastructure for various upstream projects ( see osci.io ), and we will indeed replace our Centos Linux 8 by Centos Stream 8 (and to be clear, we decided to not run RHEL long time ago because we want to have the same “user experience” as community members, and we also run Fedora and Debian in production).
I already switched the Gluster.org infra already this morning ( cf https://lists.gluster.org/pipermail/gluster-infra/2020-December/006187.html ), and going to discuss the others in the various infrastructure in our team meeting tomorrow.
I know that others are also switching, like Remi, our php packager, from remirepo fame, is migrating his server to C8S: https://github.com/remicollet/remirepo/issues/163
This is good news. I explored osci.io and the linked projects such as gluster and ovirt. I could explore these projects and will recommend the use of them.
Life cycle commitment is meant the way the word means it. Red Had lost my trust !!!!
This was never a lifecycle commitment from Red Hat. This was a life cycle commitment from a community project, not Red Hat. To get a commitment, you also have to be committed. I’m sure you have a significant other and understand how this works.
This statement only means your article on CentOS Stream is as bogus as the EOL offered by RedHat for CentOS 8.
Since you talked about that person’s relationship I must remind you an important aspect in a healthy relationship. Trust. It is earned when action meets words.
That life cycle “commitment” was a page in a wiki written by a CentOS community member who may or may not have been paid by Red Hat, so it’s way more nuanced than your making it sound. That said, you can still move to Stream 8 for 4 more years of maintenance. Then, you can use the LEAP tool to upgrade to Stream 9, in place. I understand the trust thing, but imagine these two scenarios:
1. We move everyone over to Stream 12 months from now, and spend the next 8 years of the life cycle focusing on making it better
2. We wait nine more years, trying to support both, then move over in RHEL 9, and 10, and 11 which will all be in maintenance during the CentOS 8 “life cycle” as you refer to it.
Which one sounds like it will succeed. I’m not sure if you noticed, but RHEL sped up with the release of RHEL 8. It’s now moving way faster than it ever has. RHEL moved to a 3 year life between major versions come out. That means that there will literally be four concurrent releases, maybe 4 or 5 with EUS/ELS support. That’s an insane amount of maintenance. And, you expect us to extend that to the free downstream rebuild? For 8 more years? This is a zero sum game from an engineering perspective. If e do that, I predict that CentOS Stream would be under massive pressure and would probably have stability problems in some of the releases. I have a sigh of relief that we are moving to Stream 8 as soon as possible. This will conserve resources and give Red Hat the ability to move faster while at the same time provide stability. If we waited until 2029, we would die on the vine. This is a cloud native world, and we have to move faster.
What about this post?
https://blog.centos.org/2019/07/ibm-red-hat-and-centos/
Specifically, “”What does this mean for Red Hat’s contributions to the CentOS project?
In short, nothing.”
And…
“We will continue to execute the existing project roadmap.”
That’s not from some Wiki somewhere. That’s from the CentOS website *last year*.
Sorry, but his BS about “we never actually said that” is making your (you and Red Hat) reputation even worse.
Stop lying to us! Tell us you screwed us over. At least we’ll respect that.
This is a semi-fair criticism. The fair part is that the CentOS community said it would support CentOS 8 until 2029. The community perceived this voice as Red Hat’s voice. Furthermore, Red Hat creating CentOS Stream seems to have given some people the wrong impression, that Red Hat was investing more and more into CentOS. In fact, we doubled the CentOS brand and sandwiched RHEL between two pieces of CentOS bread. I get why that would be perceived the way it was.
The unfair part is, the world changes. We can argue about timing, but this had to change long term, and going until 2029 was not sustainable. Now you have until 2024 with Stream. I think that’s fair.
The other unfair part is, I haven’t seen a single Red Hatter complain about Rocky Linux or Cloud Linux. Not one. In fact, I’m fine with both. It creates healthy competition downstream from RHEL, and removes the responsibility (and cost) from Red Hat. It also creates healthy competition between CentOS Stream and Rocky Linux. Game on.
Users can and decide what they want. Faster stuff and equal stability? Or slower stuff and equal stability? Either way, I think the charge of a lack of stability is insane. The “development branch” of RHEL is more stable than any other Linux distro out there. At least that’s what we have to prove now.
As someone in the reddit comments outlined “implied contract”. There 100% was an implied contract based on the fact that CentOS and RHEL had a very established pattern of the same EOL date.
Correction, there was an inferred contract. Huge difference.
Interesting statement, who had the controlling share of CentOS since 2014? Was that not RedHat, so yes it was a commitment from RedHat.
Technically the board controlled it, but Red Hat still controlled resources:
https://lwn.net/Articles/839553/
That is somewhat disingenuous now isnt it. As part of the 2014 takeover RedHat have a greater than 50% share of the board seats, therefore RedHat control what the board vote for.
And if it is deemed the board are not in agreement then the Redhat Liaison can over rule the board.
What part of that makes it not RedHats decision?
@fatherlinux, thanks for the thoughtful post. I’ve been considering this situation for a day, and it seems to me that you and Red Hat have overlooked something important. You’re killing the goose that lays golden eggs, not just for Red Hat but for the entire RH ecosystem.
When I started my career, IBM and Microsoft were the default answers in most companies. Today, Red Hat is the default answer for almost any serious discussion. That didn’t happen by accident, and it didn’t happen solely because Red Hat’s solutions are technically superior.
It happened because of what CentOS is, and has done, for Red Hat.
The idea that a teenager could install an enterprise-grade OS on a computer in his bedroom, play with it, and learn it was critical to the success of Red Hat. Free-as-in-beer software attracted young developers to Red Hat, and that led to the much larger understanding of free-as-in-speech that underlies most modern software.
It also hooked them on the Red Hat environment, tools, and way of thinking. Many hobby toolsets are built and tested on CentOS. Some of those hobbies become major players, which in turn reinforce Red Hat’s position at the center of the ecology. Centos Stream is useful, but it cannot fill this role.
The pricing argument doesn’t hold water for me; those sales were never going to happen, and they won’t happen now. Either someone rebuilds CentOS under another name, or the hobbyists will use debian or Ubuntu.
Only time will tell us whether Red Hat’s prediction that the OS no longer matters will come true before the next generation of coders decides that Red Hat no longer matters.
I appreciate your thoughts, but strongly disagree on cause and effect. CentOS did NOT cause RHEL.
A teenager can literally install an enterprise OS on their computer in their bedroom. CentOS Stream changes none of that. In fact that teenager is now a RHEL contributor giving me (old Scott McCarty) ideas about how to make RHEL better now. That never happened with downstream CentOS.
Who on earth could CentOS Stream not fill that roll? That makes no sense to me.
We agree on “those sales” – I stated that in the article. CentOS Stream will still put price pressure on RHEL. Let’s be honest with that. I’m totally fine with that. It’s even healthy for the market place and Red Hat OMs like me. We can’t get lazy.
The OS matters more than ever, and we need more contributors, even if that sacrifices users who are mad.
Teenagers in their basement can still sign up for a completely free Redhat Developer account and download and install the real RHEL Server distribution completely free. Not rebuilt RHEL, the actual RHEL served directly from the Redhat network. I know, because I’ve done this (even though I’m not a teenager and I don’t live in a basement).
The more I’ve thought about this the more convinced I am that this change is a total non-issue. I DON’T work for Redhat. But CentOS Stream is literally just the latest RHEL except that it’s about one week ahead of RHEL. The old CentOS was at its worst up to three months behind! One week ahead or two months behind, which would you prefer? I know I would prefer the former. If you have some use case that’s not satisfied by what Redhat offers (you need to be on an exact RHEL point release long term, you need more than five years support, you can’t use free RHEL on a free Developer account) then you should really be paying Redhat.
And if the community needs someone to fill the role of the old CentOS, they’re welcome to do that, and it looks like they already are working on it. But that will happen on their own dime. Redhat doesn’t owe anyone this.
I’ve also had this same exact realization as I’ve wrapped my brain around this change. It basically feels like three commands, and no more issue. What am I missing? People had a warm and fuzzy being downstream from paying users. That’s the main epiphany I’ve had.
So much drama from the rpm’ers!
Ubuntu and debian for the win!
I have used Debian and Ubuntu. I still do use them. But they are no substitute for this particular use case. Debian has three years of security updates and two additional years of “long-term support.” Ubuntu provides 2.5 years of “hardware and maintenance updates”, 2.5 additional years of “maintenance updates”, and three additional years of “extended security maintenance.” Note that the three additional years of ESM for Ubuntu require either a monetary payment or (free) registration and subscription with usage restrictions in the free registration case (namely, personal use only), very similar to what Redhat provides with a Redhat Developer subscription which gets you access to RHEL and its 10-year support lifespan. There is nothing in the .deb world that compares to the previously offered 10-year CentOS support life. The new(ish) CentOS Stream offering is very comparable to Ubuntu LTS: it’s five years of updates using CI delivery.
In summary, the only thing that changes is that Redhat’s registration-free offering now matches Debian/Ubuntu instead of greatly exceeding it. If you’re willing to submit to registration, Redhat gives you 10 years, Ubuntu 8 years, and Debian 5 years. Also, there still exist third-party downstream rpm-based alternatives (Oracle Linux, Rocky Linux) with the full 10 years of update support. The 10 year support lifespan isn’t anything that deb-based distributions have ever come close to achieving.
I’m no teenager, but I don’t talk to enterprise sales people for anything, nor do I spend time counting licenses. If I wanted to do that I wouldn’t run Linux.
I understand the frustration with sales people and licenses, but on premises, I haven’t figured out a better way to monetize an enterprise Linux distribution to keep people employed doing what they love. If you have a better idea, or you are making money in some other way, I’m all ears.
Open source is free as in freedom, not as in beer.
I’d encourage you to look at the code contributions in things like the Linux kernel, CNCF projects, Kubernetes, etc. Red Hat is a huge contributor and we don’t have an open core mode, we give and receive. We get free code from the ecosystem, and we publish our code on git.centos.com. I think that’s far above and beyond, especially given that much of the code is not GPL but we give it all away as if it is GPL. We believe in open source, for the philosophy.
I will never feel guilty about Red Hat making money. We don’t do anything unethical. We create way more value than we capture. We don’t lay people off once every couple of years like SUSE and Canonical.
I do recognize the value Red Hat adds to Linux, and I love the stability of RHEL (as CentOS). I too wish I had an answer as to how to get you fair compensation for that. I wouldn’t want you to feel guilty about making money; I certainly understand how hard it is to do that and meet payroll.
Unfortunately I don’t need support and my employer doesn’t pay for anything they don’t have to. And tracking licenses either on-prem or in the cloud is just not on. It’s not unusual to spin up a dozen instances for a week and then never need them again, and those are running custom-built AMIs, not something you can just license instance hours for.
You’ve re-defined what a rolling release distro is to fit the RH announcement about CentOS 8.
If we are to believe that RHEL really is (now, via v8) a rolling release distro, users would expect that is has:
1. no . versioned releases
2. no pinning to release option
3. no version specific patches
4. only support for latest
5. world updates & upgrades
6. update early & update often
AFAIK, RHEL8 still does (1), (2), (3): it has major.minor versions, allows version pinning, releases version specific patches. You’d probably argue that RH strongly encourages & facilitates users to do (5) and (6), and that’s fine. But I don’t think someone can argue that RHEL has (4) without ignoring EUS / LTS and overlapping major releases. More importantly, (1)-(3) fly at the face of how users manage rolling release distros.
Arch is my daily driver, and I’ve ran Gentoo in a previous life. Both are considered de facto rolling release distros that we can compare to point release distros, like RHEL/CentOS. The comparison is stark when it comes to how users maintain systems, like upgrades. There’s a reason that the Gentoo package manager update command (emerge) includes the parameter “@world” — because rolling distro users are updating every package globally, and every package to latest.
Pinning a rolling release distro is either a) not-possible, and/or b) an anti-pattern. You might agree on (b) even for RHEL but CentOS still made it possible (and expected?) to pin versions, and users have re-stated their uses cases for doing this pinning. The mechanism for them doing this on CentOS is `yum –releasever=N.N` and evidently RHEL8 still has `subscription-manager release –set=N.N`, even if discouraged. This pinning flies at the face of rolling releases. Fundamentally, it’s just not accurate to even say “download Arch v10.1” because speaking like that implies a point release. A users can, however, “download RHEL8.3”. Going beyond fundamental definitions, rolling release distros also make pinning difficult. Gentoo attempts to help some users maintain older packages out of sync with latest but Arch goes so far as to state partial, non-global upgrades are “not supported”. Some would say Arch is the purer rolling release distro — however, just from poking around on this, I did find that Arch’s official package repo includes a LTS of the kernel. I’ve not used it, but I’m also not aiming for stability or LTS support.
Finally, the support — commercial or community support — of rolling releases is expected on latest, and latest only. No LTS, no EUS. If a user falls behind, they’re expected to upgrade; and because there are no point releases, that means all packages to latest in repo. And to beat the horse dead: “latest” for rolling distros does not mean the largest minor/dot release of a major release because those don’t exist.
I don’t think you’ll find a lot of folks defending your re-definition of RHEL as a rolling release. Supposedly Stream really is rolling but I’ve not tried it to know, and am all ears on that. For RHEL, I don’t think customers will go for a rolling release because they expect:
1. standardizing (SOE) on . versioned releases
2. pinning to release options
3. version specific patches
4. support on older releases (LTS/EUS)
Perhaps RHEL9 will get rid of (1)-(4)? And become really rolling 🙂
So, as a former, and long time, Gentoo user, I apareciste tour thorough responde 🙂
Inconrir with almost everytjing You said, but for the point of this conversation it’s too philosophical.
Technically, RHEL isn’t a rolling release because you do have to upgrade between major versions. Historically, this was a migration but 7+ is now pretty much an in place upgrade, but I digress.
It’s definitely always been an anti-pattern to lock versions, but yes we let people, and we even sell them EUS. We don’t want to, but they literally demand it. This is how the world works 🙂
Sigh, I have contemplated proposing that we should move to a full rolling release in RHEL 9, but I’m not convinced the world is ready for it. Especially, after this announcement.
Also, by your definition, stream basically isn’t a rollingnreleas either. All the major versions of software like the kernel or glibc are locked and back ported. So, it’s not a rolling release, but effectively the same from an update perspective. A yum update always works. That’s all I meant by casually throwing rolling around.
Either way, my technical brain now went down the rat hole. I changed that section around to not refer to RHEL as a rolling stream. I want to be as precise as I can. Instead, I highlighted the Application Compatibility Guide and the levels of compatibility RHEL offers combined with the SemVar system so that people can understand what this change really means.
>long time, Gentoo user
Nice! small world.
>Application Compatibility Guide
That’s a strong point for RHEL even if it stays a point release and has LTS/EUS “because [customers] literally demand it”.
To be honest, I think there’s a use case for point releases too. Not every use fits rolling. Pro/con, blah blah.
There’s still a lot of discussions to be had around point releases. I think two valid conversations to have are 1. could Stream add them? 2. Should RHEL remove them. I think both are valid conversations!
I was mad before I read this article. Now I’m furious. The language in this article makes me nauseous. I am speechless.
Pull requests welcome.
I second that sentiment, Keith. I too was angry before reading this article but I hoped I would find good reasons to change my mind and continue to trust RedHat. Instead I found a mixture of simple minded confusion and shocking apathy.
No one is angry that CentOS Stream exists. (I would prefer that RedHat call it something other than CentOS, since it’s not CentOS and has nothing to do with the purpose for which CentOS was created, but if it’s useful for RedHat and others I wish them well.) All the arguments fatherlinux makes in favor of CentOS Stream are irrelevant because CentOS Stream is not CentOS.
People are angry because RedHat is breaking a promise to the community. CentOS 8 was supposed to get security fixes for the duration of RHEL’s life cycle. That was the primary reason to use it at all. Now RedHat has pulled the rug out from under everyone and fatherlinux is claiming this is far because “boo hoo it costs money”? That’s a calculation RedHat should have made before making that promise. It’s fine for RedHat to make a business decision to cancel CentOS 9 and to decide the life cycle of any future products it offers. But they ALREADY made that decision for CentOS 8. Now they’re changing the rules after people in the community have made commitments on that basis.
Instead of showing empathy for such people, fatherlinux draws a false equivalence to RedHat’s business problems. If a product or service RedHat relied on changed its contract after RedHat committed to it you’d better believe fatherlinux would be singing a different tune. Imagine if RedHat decided those pay checks fatherlinux gets should be half the size in the future. Would he shrug, say, “everything changes” and get back to work? Unlikely!
I took the time to read this article hoping someone from RedHat would offer reasons I could understand and respect. Instead I learned that not only does RedHat not care about keeping promises, they they think the people who rely on those promises are not contributing to their community. This demonstrates breathtaking ignorance. Plenty of CentOS users file bug reports, post patches, answer questions on forums and even recommend the RedHat ecosystem to our customers. I’ve done all of those things in the past, but the next time I’m tempted to do something like that I’ll remember this article and find a better use for my time. I’ll bet I’m not alone.
We do care, or I wouldn’t have destroyed my weekend writing this, and updating it 20 times. We are listening. I’m sorry you think all of this work stems from ignorance of what community means. I can promise it’s not for lack of trying to understand.
You lost me entirely at “because I had read and used his Apache docs”.
Rich was pretty famous in the Apache community. He wrote the most insanely good docs for some of the Apache modules. When I first met Rich, I had never really work with upstream contributors before. I was basically star struck. I was just some red neck sysadmin from Akron, OH, and to me it was kind of like meeting a famous person to me. That’s all I meant. Rich is a super good guy.
Rich is very much a typical Red Hatter in that I know he will call balderdash when he smells something not right.
Internally at Red Hat, we fight like cats and dogs about things we believe in, or don’t believe in. There is always a debate going on about something. It’s the nature of open source. One time, when I was fairly new, Lee Congdon our old CIO told everyone we were moving to Google Docs, and people put up a digital petition on a local server. They were mad becasue it wasn’t open source, and it had 700 people within hours or a day. Lee ended up reversing course. A few years later we moved anyway, but their opinions mattered. I’d never worked at a place where you could literally have an anarchy-like revolt. I used to joke that if you did that at IBM, you’d get fired, but so far so good, lol. Not a single Red Hatter that I know has been fired, and we still debate everything.
That’s all I meant with that reference. Rich would not sell the world on some evil plan. He’s not that guy. Period.
I understand Red Hat / IBM, after all, everyone wants to make money, it couldn’t be different with this companies. However, you changed the rules of the game in the middle of the game and this is very dirty… very dirty. Centos 8 should go until 2029, no matter if it’s free or not, or was there a notice on his license saying that he could or could not go until 2029 ??? I believe there was not.
Another unfortunate point when trying to do all this justification – there is already something with the same purpose as the centos stream (Fedora), it could very well control the beta and alpha by version numbers, you don’t need to kill an entire distribution for that purpose.
The only point I agree with you is to wait and see how the story will end in a year, we have good options like Oracle Linux, but if I have to pay for Red Hat / IBM, so be it – which I don’t want and having to read another long text and discover that in the end there is none.
And again… DO NOT CHANGE THE RULES OF THE GAME IN THE MIDDLE OF THE GAME.
As I’ve said other places. RHEL has sped up massively with 3 years between major versions and six months between minor versions. That means the 3-6 major versions are in some phase of maintenance at any given time. Waiting until 2029 to shift focus to stream would have been death by a thousand paper cuts. Then, CentOS would have went away anyway, because we would be put out of business.
I think this comment perectly describes the problem i have with your article. For the most part, its very well thought out and i will admit had made me understand the change a lot more. However your hyperbole is just ridiculous :
“Then, CentOS would have went away anyway, because we would be put out of business.”
RHEL’s revenue in 2018 was 3.4 BILLION dollars. Centos support was a blip on your radar. Saying anything else is simply not factual from what i can see.
“I’ve never even seen a CentOS user file a RHEL issue (aka Bugzilla) or ping me on Twitter for CRI-O, Podman, Buildah, Skopeo, or anything else I work on at Red Hat. These users weren’t even on my radar.”
This is categorically untrue and unfair. In court a statement like this would get a loud OBJECTION. Plenty of us users of Centos make bug reports. I dont know how you would even begin to know if a person making a report was a RHEL user or not anyway. Most of the people who use centos ALSO use RHEL.
” I get why people are mad about having an expectation changed. That said, providing maintenance for CentOS Stream 8 until 2024 is still pretty damn good for $0.”
I don’t think any of us would describe this as “pretty damn good”. The reactions range mostly from “what a piece of shit move” to “well that’s shady, but i can understand”.
“What you might not realize is that whatever workload you are running on Downstream CentOS is creating value for your business, especially in years 6-10. Moreover, you’re likely charging for the service you run on downstream CentOS and capturing value.”
IMO, and my experience the opposite is correct. Major, important applications i need current and supported are upgraded every 3 years or so to stay on a current modern version of RHEL. Applications i need so have but DONT make me money sit on whatever the hell version of centos was around when i needed them. I do of course still need those to not be security vulnerabilities, so killing support for those early means i now have to think about redoing all those little piddly apps again.
On a more general note, i think overall, your article is fair, but i also think the tone in which it is written seems to imply that for anyone to be mad about this change they must have been a freeloading centos user who didn’t care about RHEL. It definitely sounds like it was written by someone who is personally offended that other people may have a visceral reaction to this change.
I’m not offended in the least, though I’ll admit I am inclined to get into a ranty mode, and for that I apologize. I actually completely welcome Rocky Linux. In fact, I go further. As I saw somebody post about Rocky Linux, “I hope we can be friends.” I think it’s a way more healthy relationship to have a downstream rebuild not connected to Red Hat. I also think it’s a healthy relationship to be friends, but compete. Rocky existing will put pressure CentOS to be stable, and destroy this perception of “beta software” which to me is just weird.
Will fast and stable beat slow and stable? You decide. Game on.
The CentOS change may be a significant strategic mistake.
The tone of RedHat employees I would say sees the status quo CentOS as an unreasonable burden delivering no benefit to RedHat, and thus they need to become free development and test for RedHat. Meanwhile, mere mortal users need to be paying or even if permitted to do it freely, must be tracked through registration. RHEL clones frustrate as it is freeloading on RedHat efforts, which may be true to some extent, but those freeloaders do provide value. The reaction to the disappointment in CentOS fail to consider “What is in it for the users?”
This is a repeat of 2004, where RedHat in the face of being *the* Linux distribution decided freeloaders should be testers (Fedora) and stable users should pay for it (RHEL). One major consequence of that act was Ubuntu suddenly making huge inroads out of nowhere, as the RH users had to go *somewhere*. I would argue the bleeding of mindshare was only staunched by CentOS and projects like it. To the extent some had not yet bailed on RedHat yet, that was a life raft to keep RedHat relevant in the onslaught of Ubuntu (who now have the majority share, but CentOS is big enough to command mindshare). If RedHat had perhaps done the same change, but let ‘Enterprise’ be cost and registration free, then Ubuntu may not have happened, and RedHat would be the vast majority of Linux installs today. As it stands, I’ve been in Canonical sales presentations, and they make a compelling argument about how they have much more share than RedHat, so technically they seem to be in a compelling position to be listened to, and perhaps in practice the more authoritative.
At this pace, I expect the community that RedHat wants to exploit will evaporate, moving wholly over to Ubuntu where things are a bit more straightforward (free distro is also the paid distro) with greater pressure. Sure Rocky Linux may pick up the slack, but the CentOS scenario certainly has people thinking a move wholesale away form the RedHat ecosystem is a safer bet for long term strategy.
RedHat does a lot and employs a great deal of people, but they should not dismiss how much CentOS keeps their paid offering relevant. So long as Canonical, SuSE, and Oracle are participating, there are other viable options and with less than 2% of hosting share without the free user bump, it’s hard to imagine people continuing to think of RedHat as an authority.
The sane thing would be to adopt the Oracle model, RedHat would be free for self-support, and subscriptions are just for support. RedHat has more credibility, the free user bump would proudly bear the RedHat name directly, and Canonical would no longer have sales presentations showing how embarrassingly small RedHat’s share is. Without that 19% free user chunk, I would expect RedHat to see even their 1.8% share erode over time.
I welcome the competition from SUSE, Canonical, and Oracle. FYI, you have to pay Canonical to get LTS updates after year 5. We actually moving to be more inline with them on this. Let’s look back in a year, and see how CentOS Stream does. I just don’t people realize that not much has changed. The maintenance has been cut to 5 years, but you are grossly underestimating the challenges that CentOS has presented in our development pipelines. I’d have to write a whole different blog entry to explain. Stream removes this tension and makes every Red Hatter a potential Stream contributor. That wasn’t true before. It was a tiny team, kind of fire walled off from everyone.
The thing is while there may be more contribution to what is now called ‘CentOS’, the value was not the CentOS brand so much as it was uncomplicated access to a well tested base, the product that the CentOS name meant to 99.9% of the userbase. No need to register to download, ability to mirror freely, yum repositories without doing rhn registration, no overly complicated stratification of what could and could not be installed. To say not much changes would contradict the statements issued that say CentOS stream isn’t production appropriate, that it’s testing for RHEL, and if you need stability contact a RedHat sales rep (instead of probably going elsewhere).
If you are moving to be more inline with the competition on this, that would be a fantastic thing to actually say either before or concurrent with the “no more stable CentOS” announcement. If this is what is truly coming in 1H 2021, a lot of headache could have been avoided by coming out saying what RedHat plans to do.
I don’t think anyone is attached to the CentOS *brand* and can recognize that RedHat expending resource to do RHEL *and* do more effort to rebrand every release and make it free is a silly situation for all involved. RedHat endorses simple access to installation and repositories, but under a weird different name that lacks a clear upsell to support like a free RHEL would. The simple answer would be “ok, registration and rhn is now an optional part of the experience for paying customers, but RHEL 8.4 and up shall be freely downloadable with plain old yum repositories for the would-be CentOS users, for the verbatim CentOS experience without the brand confusion and easy path to upgrade to paid support”.
The language thus far has alluded to potential free access for certain use cases, but those terms seem to imply potentially awkward limitations and does not recognize that RHN and channels, and registration is part of why CentOS is many cases is easier to manage. It would be nice to have had clarity from the onset.
These are fair critiques. I and many others at Red Hat have been griping about Subscription Management forever. That said there are actually a lot of improvements now days, I just think a lot of people aren’t aware as it takes time for information to disseminate. Check out Subscription Watch: https://access.redhat.com/documentation/en-us/subscription_central/2020-04/html/getting_started_with_subscription_watch/con-what-is-subscriptionwatch_assembly-about-subscriptionwatch-ctxt. I think SW along with low/no cost subs will solve a lot of this. Believe me, as Product Manager for RHEL, I wish there was some way to “not have” subscription management at all. When you move into the world of containers this also gets a lot better with UBI (my baby): https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
The challenge with subscription management has always been that there’s no way to protect price pressure. There are horror stories I’ve heard about Canonical, and even SUSE. Look at the layoffs. Seriously, nobody notices. SUSE recently acquired Rancher (Kubernetes thing), then layed a bunch of people off. Some of these people were friends of friends of mine. We’ve never had layoffs like that at Red Hat, because our revenue stream has always grown nice and steady. This is because we found a fair, consistent way to monetize enough of the value that we create.
That said, I fully admit the world is/has changed and subscription management is painful in cloud and containers. We are working on it, furiously, I assure you. It’s not easy.
I understand the difficulty of trying to ensure proper entitlement for paying users and to some extent how partitioning into different subscriptions helps RedHat size future investment and have a clearer picture on how support for a particular product team is doing relative to their actual install base.
I also noticed that CentOS 8 included relatively fewer of the projects that CentOS 7 did compared to RHEL, and that seemed to pan out ok, giving RHEL some stronger guarantee of revenue streams for certain target markets without a lot of credible alternatives. While not ideal for a user, a solid move for RH with little downside.
It’s easy to see how Stream promises greater value for people running CentOS project (they are no longer relegated to mostly mere rebadging) and RedHat (more timely feedback from a community assuming they could convert the entire userbase), and select users (not so aggressive as to run anything important on Fedora, but wanting more ability to actually work changes). However mostly CentOS users are, with respect to RedHat, only ever going to be “freeloaders” in a direct sense (though as mentioned, they are immensely important in an indirect way)
.
For the ‘base’ distribution, the onerous restrictions, however mitigated, don’t make sense for people that would have been content with CentOS (no support available). In a world without viable competition this move would have been without much downside. However, when you have a few commercial competitors, one of which is a mostly a rebadge of your own offering, stepping back from freeloaders entirely introduces risk of eroding your mindshare, and the freeloading clones that appear to impede RHEL market share will continue, but RedHat will get reduced value from those communities, as upsell to “real RHEL” has a higher than necessary hurdle (in Orcale’s case, the ‘obvious’ upsell direction is very much not what RedHat would want)..
I think this points to RHEL base approximately equivalent to Oracle Linux being more easily available, and keeping some of those additional subscriptions as-is, since you have the data to suggest this is a fairly safe thing to hold back without outrage, and not much competitive pressure on that front either.
Certainly keep paid support and registration for base for paying customers, but enable the CentOS-like experience as well.. Those users by and large aren’t going to convert to RHEL and will find harbor elsewhere, why not keep them in the fold and even more aligned to your strategy? They are going to freeload no matter what, better that they are attached to your ecosystem as they might get involved in non-freeloading scenarios over time, rather than saying the org should buy from Canonical, or that it would be easier to buy support for Oracle for existing Oracle installations than migrate the platform to RHEL.. I can’t imagine you would lose any paying customers as everyone already knows they can find self-support without compromising compatibility with RHEL, even if CentOS is killed off. This reduces the complexity of a lot of work to the seemingly trivial result of rebnrading.
> I’d have to write a whole different blog entry to explain.
Please do. But also note, this is their own fault. No one asked Red Hat to acquire the CentOS project.
You mentioned something about people having to pay for LTS (for Canonical) after year five. I don’t expect an operating system to be running on the same version for more than 5 years. I expect that a better version of the operating system is out by year four or maybe on year five. Canonical doesn’t force you to sign up for a subscription to download the next version of the operating system, unless you want paid support. Generally in place upgrades are easy and don’t cause many issues, Personally as a DevOps/Failover Systems Engineer I prefer to handle all of my own issues as much as I can instead of putting my problems on others, which would cost me more money than I make, so I really don’t like support contracts. for the past 5 years I had primarily focused on CentOS, but as soon as they announced stream I started making a lot of my projects in Ubuntu. This just puts the nail in the coffin for me on any sort of hope of using any red hat software. A lot of people pointed out that this community should have realized that as soon as this turned into Stream, that we all should start moving over to Ubuntu or other OSes
I’m confused by your comment. Stream has maintenance for 5 years, just like LTS Ubuntu. You are never forced to buy a contract. You can upgrade in place from 7 to, and will be able to upgrade in place from 8 to 9. They are literally almost exactly the same. The difference with Stream is, you will be influencing the next dot release. Take a look at the RHEL 8 ACG and internalize what Stream really means. This is not nearly as earth shattering as most people think it is.
The RHEL and therefore CentOS Stream ACG is ridiculously more stable than Ubuntu. If you want more control, this actually gives you that, not less.
The messaging is very different.
Value judgements about how well they do at it aside, Ubuntu does not proclaim that free users are getting the beta version of the OS, but only paid users get to stick with a stable version. They proclaim the free edition as being identical to the production appropriate version.
Really, RedHat would fare so much better in the internet reaction if they had just said “RHEL will replace CentOS and be as easy to access and update, and RHEL Stream is now available for people who want in on the development of the OS”. The CentOS brand is not a particularly valuable one to try to make mean something new, and only complicates RedHat’s ability to capitalize on the success of that segment of users. No one is going to shy away from the RedHat brand but remain firecely loyal to the CentOS brand. It wasn’t the brand, but the sensibilities that were valued.
Correct about the messaging, I am not here to contribute to a development cycle. I was using CentOS as a fairly stable, well tested by the Community, Operating System to deploy applications.. I don’t want to be a tester, and I don’t want to pay for support, I know my way around a terminal, and the community was GREAT about help already. I wasn’t strictly referring to CentOS Stream, but was kind of mentally adding the RHEL subscription model to it because, moving CentOS to an upstream, feels like a money grab, to push the OS that people considered stable, to be in the development, ie the beta part of the cycle, therefore making sure that they 1. Get bugs, 2. don’t get support for them, and then 3.Urged to ugprade to RHEL for support and bug fixes. It totally screams of a ploy to move more people to a paying RHEL subscription.
I get it, but this isn’t YouTube, this is Enterprise Software. We’re not a guy talking into a camera, we have like 5000 engineers that need a paycheck. The CentOS team is tiny, maybe 10ish people (I’m not 100% sure on size), but theynare just revising the work of a thousand people. The container subsystems team alone has 20-50 people (depending on how you slice it) that need a paycheck.
We need to accelerate RHEL growth. You need a free operating system that is perceived to be as stable as RHEL, one of the key value propositions that compete with RHEL. You don’t want to test, we need testers.
If our wills are at odds, then they are at odds. Personally, the level of testing that you are imaging and that I’m imaging are on differently planes of existence. Paying RHEL users were already doing this “testing” as you call it, so forgive me if I don’t feel that bad making the CentOS community look at bits first, instead of paying customers.
Also, if you’re not here to contribute, no worries. Run three commands and enjoy four more years of free maintenance. Nothing changes for you other than year 6-10 maintainance (you’ll have to do an I place upgrade). I think that’s more than fair.
This comment is exactly my sentiment. You keep saying that this actually brings Red Hat more in line with the model of companies like Ubuntu, but the elephant in the room is that it’s a CentOS Stream is simply RHEL Beta now, and Ubuntu is just Ubuntu. That’s a massive difference whether you’d like to admit it or not.
Yeah, it means we’ll be able to innovate even faster than Ubuntu. Thank you for your usage.
I am opposed to this change but appreciate you providing your viewpoint. I actually like the idea of Stream and think it serves a good niche.
Your article talks about the benefits to RH you get from Stream by distribution of effort through Stream. That’s fair. However, you’re essentially forcing people off their existing release into 1. A licence purchase, or 2. into becoming a tester.
What was the harm in keeping the CentOS downstream available and allowing Stream to prove its value through merit?
You’ve mentioned the stability that we will likely achieve but I notice your article completely ignores the security update process.
What will this look like for RHEL and Stream? What about embargoed fixes? Will Stream be (is it?) some hybrid of upstream features and downstream security errata?
To be fair, I didn’t mean to ignore the security aspect. I keep adding to the article as I think about things, and that’s still sort of on my to-do list. I agree that there is a downgrade in security patch SLA from RHEL, but it’s about the same as downstream CentOS was anyway. Forgive me because I’m responding off the top of my head and brainstorming this as I write.
The harm in keeping downstream was paying people’s salaries. Wild guess, call it 10 people that costs us $1.5M to $2M a year for the CentOS rebuild team. Plus infrastructure costs for computers, software maintenance, etc. I don’t know the full cost for the CentOS program but call it $3M/year.
So, just like CentOS users are complaining, we don’t have a magical $3M budget to double the investment and put that much work into Stream and keep paying for downstream.
Now multiply that $3M by 9 more years, the hypothetical time when we could have gotten rid of CentOS 8 without making people mad.
By 9 years from now, the cloud world will have changed so much we won’t even recognize it. Plus, it would have cost us a hypothetical $18M in costs based on my wild guess. It would have cost $36 for the same investment upstream and downstream. Worse, this is a rare pool of talent.
These numbers might look small for an AWS or Azure, but I assure you those are NOT tiny numbers for us.
Just like CentOS users are complaining that they can’t go get budget out of nowhere, we have the exact same business realities.
So, that makes it a zero some game. It’s our business reality, or these free users business reality. Which one are you going to choose?
Mind you, I’m a product manager so I’m used to doing back of the napkin calculations like this, but I don’t run the CentOS stuff, so my numbers are a wild guess.
To be fair, if that’s the reasoning, RH shouldn’t buy CentOS in the first place. CentOS Stream is actually Redhat Beta in reality. RH is buying the CentOS brand, and that’s about it.
Don’t get me wrong. I will use CentOS stream for Ovirt.
How does CentOS Stream fit in for communities like the HPC community (but I’m sure others from what I’ve read in forums) that rely on downstream out-of-tree kernel modules? Think popular projects like OpenZFS, Lustre, Intel Omni-Path, Mellanox OFED, IBM Spectrum Scale that target builds at RHEL/CentOS point releases (and these things break between RHEL point releases). Where are we to go now that CentOS will be Stream only and thus impossible to reliably use with these tools at all (the coming RHEL 7.4 kernel that is in Stream now isn’t supported by these projects yet and won’t be until it lands in RHEL or even weeks after that).
For RHEL to stay relevant in these communities generous free RHEL licensing will need to be provided and GUARANTEED explicitly by RHEL to stay that way (so that there is never any future confusion) for these organizations and their employees to use, no bait and switch after 2-5 years. Not to mention the thousands of folks who run homelab environments based on CentOS (that exceed the current developer subscriptions) and also rely on tools like OpenZFS and the like.
This is a good piece of feedback and I will bring it back to the RHEL team. A peer on my team, Jeffery Katcher, who is a new hire, is actually dedicated to HPC. I will bring this up to him. It may already be on his radar, but I’ll make sure.
This blog has been very informative, thanks for providing so much insight in the internal thought processes at RHEL. As a home user of the CentOS distribution I was under the impression that the stability and overall benefits would vanish.
After taking the time to read this I am confident to perform an upgrade in 6 months to CentOS Stream.
why tho? Just use something else
Why tho? For home user it makes no difference.
We run a mixed environment of RHEL and CentOS. CentOS for dev/test and HPC clusters of machines, and RHEL for everything else. We benefit from the consistency between the 2 versions to reuse tooling, configuration, and packaging. We gladly support RedHat by buying RHEL, but paying such high subscription prices for 100’s of HPC servers we don’t want any support for isn’t palatable. If RedHat had a lower subscription fee, we would have no problem with this move.
There will be more streamlined low/no cost options for these use cases.
I think Stream will be fine for many of these use cases as well. It’s just not as different as everybody thinks.
Dear Scott,
Thanks for the clarity and the will to explain the change from a RHatter perspective, very appreciated. A lot has been said already and I don’t want to extend myself (I’m planning an article on my site for that). I only have a question:
How far from reality is stating CentOS Stream is a Release Candidate for RHEL?
I ask this because, my appreciation is lots of people are merely consuming CentOS and are mad because their perceived quality is gone. A quality which, I believe, was never there in the same amounts their perception is. Would that “Release Candidate” phrasing help them understand the move? It could make things worse, and I would have preferred to read your explanation or RHEL releases instead of a “cool” Rolling Release model, but this may be another way to put it.
Waiting to hear from you.
Albert
I tend to agree, it’s not “becoming beta software” which is how people perceive it, which admittedly annoys me. There’s this perception that they were getting something perhaps even more stable than RHEL which frustrates me. CentOS was a “blind rebuild” that included any bugs RHEL had. Calling CentOS Stream a Release Candidate with good visibility is a pretty good description, I like it.
Dear Scott,
I’ve published my very opinionated view of this case on my site. I believe you may agree with some points but may strongly disagree on others. I’d love to hear from you on this particular, publicly or privately, however you want and if leisure time gives you the opportunity.
https://www.adminbyaccident.com/gnu-linux/the-centos-party-is-over-isnt-it/
Thanks for your dedication.
Best regards
As an example, many of these CentOS users have no idea and honestly don’t care about how much work the Podman team does to stabilize it before cutting the RHEL branches and releasing. Stream will get all of this work. It’s not like we just do rolling branch updates, hmthat would be insanity. Each cut of Podman is specifically chosen and QE’ed and patched often many times before we even do a build that’s public (even in the RHEL alphas/betas).
We don’t just do PRs against a rolling branch and auto build RPMs. There is a human process in between. That’s why we don’t break RHEL customers.
Yes, so lets say you are in charge of the company. Will you run release candidate distro on production? I hope not, so this is reason why many wont use CentOS stream.
Interesting read.
My primary concern is the heavy handed manner of how this was done. I think I have to assume that it could happen again if my needs don’t align with RedHat’s needs. Next time it may have a more significant impact on me. That’s a risk.
I can definitely see how there is suspicion and I am sympathetic to that. That said, this was negotiated and voted on:
https://lwn.net/Articles/839553/
But, as mentioned above, the vote means nothing since RedHat has a majority on the board.
I have the free developer subscription from Red hat and ironically the reason I used CentOS over RHEL for this project was I didn’t want to risk having the rules change half way through the project. I should have just stuck with AWS.
I’m not mad, I’m hurt. I know I have no right feeling this way, because I don’t actually pay Red Hat any money and therefore don’t really feel I have a voice. However, I am part of a larger community that supports Red Hat. I know Red Hat has been in the top five contributors both in money and in code for the last 20 years, if not the first. They even support organizations they’re not even making money from. This love I have for Red Hat is both out of respect for what they do and what they’ve done. I have spent the last four years building and teaching curriculum for a college based on CentOS. I’ve done this because I know when these kids graduate, they’re going to probably move into jobs that support Linux, and I’m hoping they use RHEL, because Canonical doesn’t do half of what Red Hat does for Open Source and Free Software. I have spent the extra time making sure they know how Fedora, RHEL and CentOS all fit under the Red Hat umbrella. I don’t fault Red Hat for needing to make money, or even changing gears. I do have issues with the disregard they’ve shown to the overall community, by just dumping this on us all. It felt kind of like a big F-U to us. I know I don’t directly support Red Hat financially, but I feel I’ve grown their supporting community. It’s the community that ends up being IT people, Network Admins, Programmers. If you piss off the Red Hat community, who is left to buy and support Red Hat products.
“Stated another way, changes to compatability level 1 or 2 software are in the category of patches, not major or minor change.s The only reason the RHEL dot release increments is because some of the higher level pieces of software in the compatibility level 3 and 4 can change.” That’s not quite right. It’s common for there to be minor rebases to any component in a RHEL minor release, provided ABI is not modified or removed.
Fair. But, we don’t rebase the kernel and glibc. Nor do we generaly rebase anything critical like the built in versions of Python, Perl, etc. In a nutshell, there is a cultural bias toward never rebasing. I was going against the culture of RHEL to create the containier-tools:rhel8 application stream. It’s still not widely understood, culturally, why I created this. It’s been 18 months, and I’m still the new kid on the block with this appstream.
I have been active in CentOS community for 12 years, last 9 years I am main admin of official CentOS group on Facebook. When I tok over there was 300 members, now there is 26.800+ members. One part of this comment was originally written for a article on Medium where RH employee said “CentOS is NOT dead”, and other is part of reply in CentOS mailing lists, so it might feel like it is not addressing you specifically. Sorry about that but this is Frankenstein’s monster.
First thing I must notice, fatherlinux, is that your lack of using CentOS and communicating with CentOS community robs you of true understanding what “CentOS Linux” was, and what it meant. I used CentOS Linux to get a stable distro. I can trust without paying for it. I and small companies I installed CentOS for were NEVER gonna buy RHEL. Ever. We do not have enough money for it. So it was always gonna bee free Linux distro. In Serbia where I live Linux community is 80-90% Debian/Ubuntu. Between 2008-2015 I was almost alone in preaching benefits of RHEL/CentOS. I was almost ridiculed. It got so bad that I finally gave up trying to change their minds.
What your western mind does not realize is that USA is 330 million people, and Europe is another 800 million. There is money there. But India is 1,2+ billion people and China has 1,5 bn people. Entire Asia has 4bn people, and legal software and paying for licenses is very low due to general low income. But they are getting technologically better and to avoid Microsoft licenses many choose Linux. India even started teaching Linux in schools, but guess what, they use Ubuntu! Every child in India is learning how to use Ubuntu. As a admin on CentOS Facebook group, majority of newbies are from Asia, mostly from poorer countries as India, Pakistan, Thailand, Mianmar etc, Those that reach CentOS community via Facebook are true newbies. They barely know to use Facebook and majority does not know support forums and mailing list even exist on the internet. Bug report systems especially. Over the years I have built several pages of resource links, explanations, (safe) howto’s, etc. I did everything to make newbies understand, appreciate and adopt CentOS and via that the RHEL way. I preached like a zealot. And zealots ARE NEEDED for RHEL, because it has many issues but only positive point was it was 99% clone of RHEL.
When we explain them that there are no audio/video codecs due to legal restrictions, that kernel is backported and adding some module is not possible if they compile vanila kernel because it brakes the kABI, that they need to install centosplus kernel because driver they needed RHEL does not want to support, that CentOS tools for development are too old because of version freeze, that they need to install several 3rd party repositories to make their server work, they then asks us “but why would we use CentOS instead of Debian, Ubuntu, SUSE…?”
And every single time the response was “Because Red Hat is great company that created great and very stable product and CentOS is almost total clone of RHEL, and when they learn how to manage CentOS they can go for RH Linux certification. And that is it, for them CentOS does not have any other competitive edge over other Linuxes beside “99% clone of RHEL”. SELinux was more of the repelling point, vast majority would just turn it off when they read first Howto on the internet, and great efforts went to pleading with them to try to learn SELinux.
So the fact that there are 1.000 guys that do developing on CentOS does not mean that majority of CentOS users are “developers”, on the contrary, vast majority of servers running CentOS, especially rented VM’s are run/controlled by guys like me whose main job is not to manged Linux servers, that is only a side job and many only barely understand what they are doing. They installed the system, configured it via some tutorial, and left it running.
I am one of community members that was very vocal about death of “CentOS Linux”. No one said that “CentOS” is dead, but “CentOS Linux”. CentOS is Open Source Project, and “CentOS Linux” was product of that project, 99% binary compatible CLONE of RHEL. CentOS project would take RHEL source, debrand it and compile it so that CentOS is “bug-for-bug” clone of RHEL. There were many instancies where some package was not compiled against exact version of dependency package, so when CentOS dev tried to compile it they would not get same binary as RHEL’s, so they would spend time locating which version of which dependency package RH devs used.
It was done for 14+ years so that “CentOS Linux” would be binary compatible with RHEL. This is important if you use some software that is only supported on RHEL, and on no other distro. Using CentOS, you were (resonably but not officially) assured that software running on CentOS works same on RHEL (and vice versa).
This binary compatibility is MAIN and for many ONLY reason why they used free-as-a-beer CentOS, some on test servers (save on 1 RHEL license for system that does not create money), and others in not-for-profit organizations and small companies (that can not afford $300+/year for RHEL license ) in actual production. Even Hosting companies offer “CentOS Linux” as option for installation, and they save a ton of money on RHEL licences, but transfer that save to cheeper “CentOS Linux” hosting. If no RHEL clone ever existed, all of the CentOS users above would not learn how to use RHEL/CentOS but would start and continue to use Debian or other Linux distributions for all those purposes.
So yes “CentOS Linux” IS DEAD, and instead of that product, RH, now owner of (once Open Source) CentOS trademark offered DIFFERENT PRODUCT, “CentOS Stream”, something that was never meant to replace “CentOS Linux”.
And here is main problem with “CentOS Steam”:
“CentOS Stream” is designed to offer a vast minority of CentOS users DEVELOPMENT platform on which they can prepare their software (that will run on RHEL) to support next RHEL minor release (like 8.1, 8.2, 8.3, 8.4, etc). Current RHEL minor version is 8.3, and “CentOS Stream” (I choose to call it “RHEL Stream”) is set of packages that WILL BE RHEL 8.4 in next (up to) 6 months. Unlike other Open Souce projects (run by community via consensus), “CentOS Stream” devs will only be RH employees who might be persuaided to add something new, but that is not likely. Only part of “CentOS Stream” that will be trully Open Source (with imput from community members not employed by Red Hat will be “SiG”‘s, Special Interest Groups that will develop their own software on top of RHEL’s next minor release, I mean “CentOS Stream”). Kernels will be biggest problem because every time RH employee decides to work on new kernel, community from ElRepo that compiles driver modules (that RH does not WANT to support/spend resources on) will have to be recompiled. And Red Hat disables A LOT of drivers, specially for older hardware!
Considering that can be any day in a span of 6 months, any time CentOS Stream user that uses 3rd party drivers (does not run “CentOS Stream” on RHEL approved hardware!) updates kernel might lose network or even storage (drivers for unsupported RAID controler or network attached storage not officially supported by RHEL). Would you risk running your server on “CentOS Stream” if it means daily fear it will stop working and need physical access to it to troubleshoot problem (maybe it was not kernel this time but actual hardware issue?).
Chris Wright’s words:
“To be exact, CentOS Stream is an upstream development platform for
ecosystem developers. It will be updated several times a day. This is
not a production operating system.”
Who ever tries to minimize this issue needs to know following: Hundreds (all of them) of 3rd party drivers from ElRepo project that work on RHEL 8.2 STOPPED WORKING on RHEL 8.3, and ElRepo needed to recompile ALL OF THEM. Luckily, with “CentOS Linux” that follows RHEL minor releases this might happen only every 6 months and any driver reported to have issue is recompiled against new kernel and will work for another 6 months. Imagine such nightmare happening every time soem RH dev decides kernel need some changes?
To conclude:
When RH employed CentOS Core team in 2014 they promised that nothing will change for “CentOS Linux”. According to Johnny Hughes, member of the CentOS board this change of direction, discontinuing of “CentOS Linux” happened my RH liaison stating that changes will be made how ever rest of the CentOS board votes (with implication concluded by me that those against will lose RH employee status). Board was initially against, but then they capitulated in front of Red Hat blackmails and decided “to vote for changes unanimously”. Red Had flexed it’s muscles, members of CentOS Board will be forever remembered as exchanging reputation and respect for income in Red Hat, and users decided such tactics deserve abandonment of Red Hat.
Some 30% of people commenting negatively say they will move to Debian/Ubuntu regardless of any positive points Red Hat employees try to make, at least 60% will stay on CentOS Linux 7 until EOL but will switch CentOS Linux 8 to Springdale, Oracle, or Rocky or Lenix in next 12 months, and big non-for-profit institutions will wait to see what will happen with “free RHEL licensees” for them. Around 70-80% of sysadmins and CentOS users commenting will never, ever, recommend RHEL to anyone.
I have to rebase my server from CentOS 6, and I am going with Springdale for now, and will start learning Debian. I will soon resign as admin in Facebook group (Many think that FB group is owned by me) and I was already asked by some FB users if I plan to create new EL group they can switch to. Only reason to delay is to try to persuade members and visitors that they do not have to rush with switching to Debian/Ubuntu, that there is still time.
Ljubomir Ljubojević when you posted your comment on my blog, I started to read it and came to the “you don’t understand” part about CentOS. Then, I got to the “Western Mind” insult and I stopped reading. I decided to read it later. I try not to comment/respond when I feel anger. If your goal was simply to make me mad, you succeeded.
I’ll say this, I worked at American Greetings and I led their move from Fedora to CentOS, so yes I do understand CentOS. I chose it. I convinced my boss at the time. I built the core builds. I rolled it out on 1000 web servers. We used it to avoid paying Red Hat for RHEL, and still got most of the stability. Surely, we got enough stability for our needs with e-cards.
Your accusation that “I don’t get it” is misplaced.
Second, I’ll address the Western Mind comment, which is offensive. You don’t know me at all. My mom and her entire side were immigrants from Eastern Europe. My mom is still stateless and has no citizenship anywhere. Worse, I grew up in an abusive household:
http://educatedconfusion.com/let-me-explain-poverty/
Finally, I’ve spent an immense amount of time exploring and trying to contribute in poor countries like Mexico, Bolivia, etc.
I understand your angle about software freedom and helping the poor. It was my primary attraction to free software.
There is no rational argument to be made that CentOS Stream cannot or will not help fill this gap, just as CentOS did. None. That argument gives zero good will toward Red Hat, and zero good will toward me personally.
There are two main gaps that CentOS Stream can fill – availability and contribution. Only CentOS Stream can fill these two gaps, CentOS could only ever fill the availability gap. Karsten Wade explained this extremely well here:
https://blog.centos.org/2020/12/balancing-the-needs-around-the-centos-platform/
Finally, if you truly want to lift people out of poverty, then I would argue that the contribution gap is way more powerful. Without contribution, an individual cannot fully express their freedom and free agency, nor fully fulfill their greatest overlap of passion and profession. More here:
https://youtu.be/-gKT55VTHBM
Claiming that CentOS fulfilled these is a very impotent argument.
I’ll post this comment to my blog as well.
I’m happy to continue the discussion, but please don’t assume that you know anything about me without doing any research.
That’s lot of spin but it doesn’t change the fact that.
1. Redhat bought CentOS.
2. Redhat changed CentOS from what it was.
Sure, you have all your reasons. Redhat could of just not buy CentOS and start it’s own RHEL Stream. But since Redhat bought it, it has every right to do what’s best for it (it being Redhat, not CentOS users). Just admit it and move on. Fortunately this is open source. Rocky Linux or something will replace it, just like LibreOffice/OpenOffice, MariaDB/MySQL.
And, you won’t hear a single person at Red Hat complaining about Rocky Linux. In fact, I welcome them as healthy competition for CentOS Stream.
Summarize:
A) RHEL
For Long Term Support (LTS) no major changes in 10 years,
Pay for RHEL as you need it
B) CentOS Stream,
Faster Update a couple of weeks or month ahead of RHEL
Note: Existence f Competition won’t let RedHat to make CentOS crappy or unstable
More Staff and contributors
If you were supporting yourselves with CentOS Linux you can do with CentOS Stream
You don’t Pay for RHEL but you contribute in case of bug, IN FACT YOU ARE SUPPORTING YOURSELVES don’t you
C) Rocky Linux
If 2 to 3 months delay is not an issue (Security!?, window of vulnerability)
If you want perception of RHEL but delays and getting far behind don’t bother you!.
Pretty much dead on.
BUT security is already an issue with CentOS stream, right? A slower than normal SLA for security. If it’s upstream, why wouldn’t security go there first? Maybe I am wrong here, but this seems the “Pay Us!” carrot here. It is this point in the CentOS Stream discussion where my company and I bail out. Being ahead in development but behind in security makes CentOS Stream a 100% no go for my company. We are a not-for profit Health System and we have a mixed environment of RHEL and CentOS, and security is the #1 concern with regards to when we patch and what OS we are using. We like Red Hat, but you are expensive in regards to the value you provide in comparison to the models of your competitors. We had begun to evaluate Ansible as a config manager earlier in the year, but bailed because of the ridiculously over-priced price model, especially after the 5k per 100 node no support model was pulled. Even if we could look past your lack of concern of the security of the Cent you are going to provide, and HIPPA says we can’t look past it, what’s to stop you from simply deciding you aren’t going to focus on a different aspect of Stream and reserve that only for your paying customers? Given the behavior Red Hat has exhibited here, and the way they handled the subs with those who had adopted the 5k Ansible model, why should we have any faith that if we moved to the paid sub, Red Hat wouldn’t just more than double the price on the subs and give us ~1 year to come up with the money or find another solution. For us, it appears time to thank Red Hat for a decade long relationship and re-evaluate which distribution we need to use to migrate to while we begin to sunset our systems currently employing Red Hat/Cent systems.
I do understand the pain here, and I hope we will have some options soon for non-profits (and research).
we have been a paying RedHat customer for maybe 15 years. we have pulled out solaris and Aix machines and replaced them with x64 servers from Intel and amd running vmware and RHEL. it is literally millions of dollars a year cheaper than proprietary unix machines were.
before anyone berates me for being a rich kid we work off a gross margin of low single figures. I just wont run production grade applications without support as unplanned downtime can cost millions per hour. we are currently deploying openshift in parallel as part of the journey to hybrid cloud native solutions.
we run our oracle dbs on oracle Linux and leverage ksplice for live patching. we spoke with cloudlinux some time back re their kernelcare live patching and it is good, but with kpatch seeming to come of age in rhel 8.3 I guess we dont really see the need anymore. I don’t see cloudlinux as true free solution as they will always need to sell kernelcare to make revenue. it is not a community project.
we do use some centos for sandboxes on aged hardware for people to kick tyres with. we also run okd.io for same reason
I too was perplexed by the centos branding on stream, but, if you own the mark it is your choice.
on downstream seems to me redhat are saying they just aren’t going to spend the money my company pays them to make patches and releases for others who don’t pay anything. seems reasonable. I would have like to keep it for my free tier for sandboxes but we will make do.
for those who say that they don’t need support but expect someone else to build package and maintain their distribution, I have to disagree and say that this is support. if its built right then you don’t have to open support cases every day.. what on earth do you think I am willing to pay for. i have admins with more than 20 years experience, they don’t need L1 to tell them what the options on a command are but I don’t want them spending their precious time building and patching
the source is still published so build away at will if you want to or create a new community and donate your time and effort to do what you think others should do for no reward .
I think redhat do a fairly good job all round. we pay them a lot but I think the business value we get from that investment is well worth the price.
cheers
brian
This article is basically stated as such. We had a lot of free loaders taking away profit from Rhel by using centos. You guys charge 349 USD a year for a self managed/unsupported production version of enterprise. I remember Rhel promising the exact same thing about how fedora would drop the sale price for enterprise as well. It seems to me the price has actually gone up.
Also you’re company is basically free loading off other people’s hardware to test and release a paid version of the software.
I’m sorry that that’s all you could take away from this article.
First, the price hasn’t went up in the 9.5 years I’ve worked at Red Hat. That’s factually wrong.
Second, We do inherit the goodness of all of the free software we consume, and we are always grateful for that. That said, we are far from freeloaders. We employ thousands of engineers. We do tons of hardware testing. Take a look at the hardware compatibility list [1]. Just putting that list together is more work than most of our competitors do. Now imagine the hundreds of engineers who work on that.
Third, I’m never going to feel guilty about Red Hat charging money for the value we create. I’m sorry you can’t understand that. I’m also proud that we have never had major layoffs [2]. Just read the Glassdoor entries [3]. Many of these layoffs are never reported, just a few weeks ago I had a friend laid off from SUSE after they acquired Rancher. Not reported in the media. Same thing happened at Canonical 6ish months ago.
If you want to preach to me about ethics, and freedom, I’m happy to have that conversation, but I can tell you from direct experience contributing, donating tons of money, and being intimately involved in the business of open source (sales, product marketing, product management) that it’s a tough business. Having a strong business model is ethical and makes a lot of people’s lives more stable, and allows us to build way more open source software.
[1]: https://catalog.redhat.com/hardware
[2]: https://www.channele2e.com/business/talent/canonical-ubuntu-layoffs-mark-shuttleworth-returns-to-ceo-position/#:~:text=Canonical%2C%20the%20backer%20of%20Ubuntu,will%20return%20to%20that%20position.
[3]: https://www.glassdoor.com/Reviews/Employee-Review-Canonical-RVW14789102.htm
Security wise, RHEL at EAL4+
Competition such as Ubuntu at EAL2
CloudLinux no EAL
If ABI compatibility of 10 years is your concern, who are you kidding you must get RHEL
CentOS stream is not beta (Fedora) , let me call it for now hypothetically as RHEL Stream, where you get the software that goes into RHEL, previously it went into RHEL first then CentOS figured it out and compiled it.
Our hypothetical RHEL stream now gets minor releases before RHEL, with 5 years life cycle , then it is not CentOS yeah it is not, we call it CentOS Stream.
“
Everything that goes into CentOS Stream is actually approved for release to paying RHEL customers. It’s just released in a … well, in a stream … rather than in a big dump every six months.
“
This is why Facebook uses it for 1 million servers they have.
https://www.zdnet.com/article/where-fedora-fits-in-the-new-red-hat-centos-stream-linux-world/
” then it is not CentOS yeah it is not, we call it CentOS Stream.”
An admission that CentOS is no longer CentOS, and there is no more CentOS. RH just buy the CentOS brand and killed it.
That’s a lot of words. Most not relevant. One question: I CentOS Stream suitable for production in an enterprise? Yes or no?
No. Neither was downstream CentOS downstream. If you did that, you were an individual who was not risk averse.
But, if you were foolish enough to think downstream was, then CentOS Stream is too. Don’t even call it CentOS Stream, call it Always Ready RHEL (Google it).
This is not just some CI/CD system, thrown over the fence crap. It’s a real, ways ready RHEL. Personally, I trust it more than downstream CentOS because it’s built by RHEL engineers.
I for one am not upset. Thank you for the honest and candid post. I’ve been a RedHat user since the beginning and almost went to work for you folks in the late 90s for they alpha platform.
over the recent years I’d shifted over to using Ubuntu instead because the development focus of what I work on which is in the iot space kind of dominates there.
But I look forward to upgrading a couple of Centos6 servers I still run to Stream this month.
My take is that overall this is a win-win.
Thank you for the positive feedback! It is much appreciated. I would be lying if I said responding to all of the negative comments wasn’t an emotional drain, but these comments make it worth it.
Thanks FatherLinux for detail and lengthy post .
The real tragedy is the fox got into the hen house. That is, Red Hat owned majority of the seats on CentOS Board.
Once the fox is in the hen house it’s very difficult to contain the damage.
If this didn’t happen we most likely wouldn’t be having this discussion today. This is a valuable lesson for future projects like CentOS.
1) Fatherlinux you say you understand CentOS by stating that you recommended in past job to switch from Fedora to CentOS at American Greetings. Was those 1000 servers you mention running production work loads ?
As you know there are hundreds (probably millions but I don’t know the exact number) of CentOS instances running real world production work loads today.
2) If you were still at American Greetings would you recommend to you boss that you convert those 1000 CentOS servers to CentOS 8 stream ?
3) If you answered yes to both questions 1 and 2 . When will Red Hat migrate their internal production infrastructure running on RHEL to CentOS 8 Stream ? If CentOS 8 stream is so great as you stated and in time it will only get better than CentOS then why not eat your own dog food ?
I wish someone on CentOS board that doesn’t have any ties to Red Hat anymore would explain to us what really happened to the project. Was CentOS project in financial trouble that it went out and seek assistance from Red Hat? Or did Red Hat approach the CentOS project. ? It would be interesting to get all those details to avoid mistakes like this in future projects.
1. Yes, I recommended, designed,and moved AmericanGreetings.com, BlueMountain.com, HollyHobbie.com, CareBears.com, and a bunch of other production sites to CentOS circa 2007. For the record, on February 14th and some other holidays, the e-card sites are in the top 10 sites on the internet, even to this day AFAIK. These were large, production web sites. We did it because Fedora moved to fast, and we didn’t want to pay for RHEL. Hands down, we wanted the stability of RHEL without paying for it.
2. No, if I worked at American Greetings today, I would recommend that they move to RHEL. Being fair, we investigated RHEL even when we moved to CentOS. At the time, I didn’t know how to negotiate very well, and we the Red Hat sales people didn’t understand our workload very well. My personal knowledge of both sides of that negotiation arm me with tools that most people don’t have (though many enterprise architects at big companies do have these skills). If I were at AG today, I’d recommend RHEl, but I’d also pressure the Red Hat sales people to discount or use an HPC SKU because web workloads are all the same. These websites are made up of large pools of servers that all do the same thing, quite similar to HPC workloads. I would likely negotiate with Red Hat for a price that is fair given how similar these workloads are. That would likely get us to a price point that makes sense for both parties. IMHO, technical sales is about 1. getting to the right technical solution (RHEL), then negotiating a price that allows both the customer and the vendor to capture some value. Pricing and packaging is a suggested starting place. My 2c. More on my product management views here: http://crunchtools.com/the-delicate-art-of-product-management/
3. I wouldn’t work at Red Hat if I didn’t think RHEL was the best Linux distro out there. I’ll never feel guilty about recommending it over CentOS, CentOS Stream, Fedora, Ubuntu, Debian, and even my favorite personal distribution, Gentoo. I think RHEL is the best tool for production workloads. Red Hat IT never used CentOS, so why would they ever use CentOS Stream? They used RHEL because it is the best and they have free access to it. That said, I’m sure Red Hat IT will use a bit of CentOS Stream to see what’s coming. Honestly,there are times that a feature outweighs the stability provided by RHEL. This is true for me as a product manager. There are often times, I just need to test out a feature quickly with a newer version of Podman.
So, all that said, I do understand what you are asking. I totally understand why people used CentOS. Because it was perceived as being the closest thing to RHEL, maybe not quite as good, but the closest to RHEL and there was no charge. Personally, I think Always Ready RHEL aka CentOS Stream, will fill that same niche. If you are OK not quite having RHEL, then CentOS Stream will be good enough, but it will also have the advantage of helping contribute to RHEL which means it’s sustainable forever.
If you really, really, really 1. don’t have the budget for RHEL and 2. don’t have the skills to negotiate 3. or have a bunch of workloads which would never be discounted. If those three things were true, I would just use a Forman server, and snapshot CentOS Stream. Personally, that would get me point in time snapshots (I purposefully didn’t say dot releases because they aren’t really real). I probably would snapshot them the day after RHEL was released and honestly, that would be good enough for me. That said, it’s kind of a pain because you have to pay attention to RHEL release dates, which to be fair is what you should probably do anyway if you aren’t paying for RHEL and being informed by a Technical Account Manager or sales person. My 2c.
If I were a managing director with a budget and you suggested the move to RHEL, there would be a Come To Jesus meeting where if you couldn’t offer a more viable solution that involved NEVER using RHEL, you would then be let go and I’d get someone else to make the correct decision.
Thanks for the feedback. Much appreciated. Once again, I ask: as CTO is your product available as a downstream rebuild?
I like the article and you make a very strong argument; personally I have no issue with the ethics of CentOS Stream, though I am confused about two things.
Firstly, why are people calling this a rolling release, when there’s a completely new version every 5 years?
And secondly, from what I’ve understood, it sounds like there is going to be the latest version for 5 years and then the moment there is a new version, the previous version won’t be supported? Surely there should be a courtesy period for users to update of like 6 months or a year or something along those lines? Couldn’t they support if for 6 years for instance, and for the last year of it’s support, just take the updates straight out of red hat, I’d assume that would impose minimal cost on red hat?
Yeah, you were not the only one confused. In fact, even I used the word “rolling release” and somebody critiqued me. The problem is, the words “rolling release” have different meanings to different people. By the most innovative definition of rolling release, you’re right CentOS Stream is NOT a rolling release. It’s a distro that has ABI/API compatibility for 5 years, and yum upgrades “just work” for those five years. Technically, that’s not a rolling release, because there won’t be major version upgrades of core software like the kernel glibc and the kernel.
RHEL 9 comes out in about a year and a half, targeted for Novermber 2022. CentOS Stream 8 will still go for another 24 months. So, there will be a 24 month overlap. Also, there is a LOT of work going on in RHEL upgrades. While none of it is ironed out, I absolutely expect to see upgrade tooling being worked on in CentOS Stream to get from version to version. That will give us 24 months. With RHEl 8 and RHEl 9, the upgrades are still “fairly” new, but getting better quickly. I expect by RHEL 10 and RHEL 11, the upgrades will be ridiculously smooth.
I just hope people don’t confound upgrades being somewhat new with CentOS upgrades not working. They are orthogonal issues. My 2c.
This article has shed light on the subject.
I do think that when you use something for free, you can’t complain about it.
If IBM moves teams from CentOS downstream to upstream of RHEL, I don’t see why any complain can be done.
I feel like this is a corporate decision to realign their product in a way that still respects the open source community.
Now if you used CentOS in providing value to your company for free and never paid any return, you might find this change hard.
Well, if you ain’t paying for support or guaranty, you are the one that has to fill that part toward your company.
Perhaps better communication would have prevented a bit of hate.
But I do feel that if you use opensource, you should give back and not “piggy back” on everyone’s work.
I for one will move on to CentOS Stream.
Thanks for all the explications and the perspective of someone from inside RH.
Oftentimes I will use CentOS to workshop a combination of systems, or automation/deployment strategies – iterating over ideas which are then able to be introduced into the enterprise on our RHEL environments. It’s not that I’m freeloading by using CentOS, but neither does the free “developer subscription” suffice.
Sometimes the overhead of licensing and subscription management is an overhead that is sometimes best put aside. CentOS lets me do that, and I can do it in my own time with my own stuff. Everyone benefits. Like people said, the more I’m edged towards alternatives, the less of an ambassador I am for Red Hat in the workplace.
As a product manager you should read the room, and realise that Red Hat doesn’t necessarily understand its customers, and that you have a significant amount of work repairing relationships that, arguably, were fairly one sided to begin with. Listen to your customers, we’re not just one homogenous group.
Definitely listening, and learning. Why wouldn’t CentOS Stream work totally fine for workshops where subscription management is a pain? Also, have you seen our announcements about subscription manager changes [1][2]?
[1]: https://www.redhat.com/en/blog/building-better-subscription-management-experience-part-1-simple-content-access
[2]: https://www.redhat.com/en/blog/building-better-subscription-management-experience-part-2-subscription-watch
Also, one more thing I forgot. If the world is moving toward containers, and UBI is free, is’t Red Hat really moving in the right direction and listening to customers? In 10 years, what emotional relationship does a Linux vendor have with a Linux user? I suspect a HUGE, like more than 80% will be through containers. UBI is freely available, doesn’t have a subscription, and runs great everywhere but is most supportable on RHEL/OpenShift. I feel like the UBI model unblocks a lot of this tension. Please tell me if you disagree?
https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image
What you haven’t provided is this:
Why should any user or volunteer developer even consider staying with CentStream, when CentOS aka RockyOS is an option.
Well, I think in open source, our intentions are often selfish to some extent, so I try and explain things in the context of what “they” get. Here are some reasons:
I’m sure there are many others, but these are just the ones off the top of my head.
This WILL result in short-term gains for RedHat sales, as some will be forced to purchase RHEL licenses. However, RedHat botched the messaging and DID NOT have plans in place to help those This WILL result in short-term gains for RedHat sales, as some will be forced to purchase RHEL licenses. However, RedHat botched the messaging and DID NOT have plans in place to help those using CentOS for cost/sensitive applications migrate to RHEL.
CentOS has SIGNIFICANT market share, far greater than RHEL. Many businesses rely on it and many supercomputers run it. Migration to anything else, including RHEL, will incur significant cost which is difficult for non-commercial entities especially in these Covid-challenged times.
Destroying CentOS destroyed goodwill. This will be VERY HARD, if not impossible, to replace and will cause long-term damage to RedHat and IBM. Some customers will spurn RHEL even if offered inexpensive licenses.
Many had planned for a long life cycle for CentOS 8. For example my products run for over 10 and I need a stable, very-long-term O/S. CentOS was our solution due to cost. These changes ARE CAUSING PROBLEMS with my customers and our teams.
We are NOT cloud-based because of hardware constraints, so we run on bare metal or in simple VMs. We feel leftout.
Issues:
1. horrible messaging
2. lack of RHEL migration options
3. not listening to CentOS community
4. significant loss of goodwill
5. loss of trust in RedHat
6. loss of options for long-term
I expect there will be many business school papers describing what went wrong with this move and how it had a significant negative impact on RedHat long-term.
I hope I am wrong, but time will tell.using CentOS for cost/sensitive applications migrate to RHEL.
CentOS has SIGNIFICANT market share, far greater than RHEL. Many businesses rely on it and many supercomputers run it. Migration to anything else, including RHEL, will incur significant cost which is difficult for non-commercial entities especially in these Covid-challenged times.
Destroying CentOS destroyed goodwill. This will be VERY HARD, if not impossible, to replace and will cause long-term damage to RedHat and IBM. Some customers will spurn RHEL even if offered inexpensive licenses.
Many had planned for a long life cycle for CentOS 8. For example my products run for over 10 and I need a stable, very-long-term O/S. CentOS was our solution due to cost. These changes ARE CAUSING PROBLEMS with my customers and our teams.
We are NOT cloud-based because of hardware constraints, so we run on bare metal or in simple VMs. We feel leftout.
Issues:
1. horrible messaging
2. lack of RHEL migration options
3. not listening to CentOS community
4. significant loss of goodwill
5. loss of trust in RedHat
6. loss of options for long-term
I expect there will be many business school papers describing what went wrong with this move and how it had a significant negative impact on RedHat long-term.
I hope I am wrong, but time will tell.
I understand your frustration, but I can assure you sales does not have a targeted uplift to revenue based on this change. This was not about forcing sales. Never was, never will be. It’s about having a long term, sustainable plan in place which aligns incentives such that CentOS has a long term future. If Red Hat wanted to drive sales, we could do all kinds of nasty things. You forget that much of the code is Apache and BSD licensed and does not need to be released. Red Hat still goes above and beyond to ensure that rebuilds are still possible. That isn’t changing.
Personally, I think most workloads are moving toward containers and Red Hat Universal Base Image is a much better free solution than CentOS ever was. IMHO, that is the area that needs more brains focusing on it (which is what my brain is payed to do), not the traditional bare metal/virtual machine CentOS user. My 2c.
I read your opinion. I’m not swayed. Stream is deeply unethical. If CentOS Stream actually launches? Then the next update of my thousands of machines I have out in the world as part of my company’s product line will be from CentOS to another Linux Enterprise platform that actually cares about true Open Source Enterprise. Selfish? No. I’m on the right side here. This is wrong and is greedy. If you deal with actual RHEL (which we tried, and it is the most poorly managed licensing system I have ever dealt with in my over 20 years of experience in IT from grunt to CTO), you know you do not want to be in that realm of business with them. This a bad deal, made by people who just do not care.
I strongly disagree. What’s wrong with expecting free users of an enterprise operating system to become part of the upstream contribution process. Why shouldn’t they be expected to contribute in some way, such as driving down the development and research costs of RHEL (just getting CentOS Stream user’s feedback, not testing that’s a myth).
As a CTO, do you give a downstream rebuild of your product away for free? I don’t think so.
I would recommend this to understand how upstream contributions reduce the development costs of downstream products: http://crunchtools.com/the-delicate-art-of-product-management/
Something with this article does not feel right. Almost every paragraph feels like:
* “This is the issue”
* “True, this was badly handled”
* “But it is important / necessary for RH”
* Some solution what the reader can do for RH
Aside from that:
About change:
“Red Hat […] creating Alpha and Beta releases […], but literally almost nobody uses them. […]. CentOS Stream will change this.”
What is the reasoning behind this? Why would anyone suddenly be interested in that?
About faster updates:
I can only speak from personal experience, but I want *less* updates. I do want to have security updates, but often I have neither time nor budget for continuous changes. I even have one machine which got its last update 4 years ago, because replacing that machine would mean throwing a lot of dependent hardware away. (It’s also the same reason I still depend on Windows XP on some (non-networked) machines.)
I couldn’t get the budget for fully replacing that thing even if it broke down tomorrow – it would need some non-optimal temporary solution.
About support time and trust:
Maybe it would have been reasonable to reduce it to the same timeframe as CentOS 8 Stream (like 2024 or so), but reducing it from 10 years to less than three years is kind of a middle finger to the users. Having its deadline now before the deadline for CentOS 7 adds quite some absurdity.
Mind you, I’ve only got about eight physical CentOS-Machines left (not counting VMs), so I’m not sure if my opinion is too relevant here – I’m almost exclusively on Debian and Fedora, even switched a few boxes from CentOS to Fedora Server about half a year ago. (I think I already anticipated that some of the changes in CentOS wouldn’t go well with my needs and views, but I didn’t think that they were that abrupt and that far-reaching)
About userbase:
In the last few years I’ve been the primary driver in my local community for rpm-based stuff – either because I’ve been the person setting the stuff up or because I’ve been asked for opinions about which OS to choose. But when looking back the last two years I’ve essentially stopped recommending it since it started to feel like RedHat is applying pressure to the community. Mostly I feel that in fedora, because I use that more than CentOS – lots of small things – CPE deciding important stuff without community inclusion, and forcing them through even though there is almost no support for that (GitLab); ELN; Modularity; lots of decisions in both CentOS and Fedora have an (implicit or explicit) addition somewhere along the lines of “we decided that way, because RedHat wanted it that way”.
I don’t have numbers, but I think the “loosing the userbase”-process has already started before this decision – what comes to mind is the epel8 repository – it feels like a lot of packages have lost maintainers for the EPEL repositories or even don’t have any builds for epel8 (or ELN) at all anymore.
About binary compatibility:
This has been also noted in another comment already. If this deviates too far between CentOS and RHEL this means splitting the non-redhat-distributed (to a large degree proprietary) software stack – currently a lot of requested proprietary software is available for [Windows, MacOSX, Ubuntu, Android, iOS], and some have the set [Windows, MacOSX, Ubuntu, Debian, Fedora, CentOS, tar.gz] – so having to split the CentOS package into CentOS and RHEL would produce the same problem as Debian/Ubuntu currently have – those packages for Ubuntu downloadable from the vendor pages or github releases pages (and there are a lot!) quite often don’t run on Debian properly anymore.
Hope a bit of this text gives a bit of insight or ideas.
Thanks for the feedback, I’ll try to respond to each section to make it more clear:
Something with this article does not feel right. Almost every paragraph feels like:
* “This is the issue”
* “True, this was badly handled”
* “But it is important / necessary for RH”
* Some solution what the reader can do for RH
I’m sorry it’s formulaic in nature. I got frustrated on a Friday night, started writing and this is what came out.
Aside from that:
About change:
“Red Hat […] creating Alpha and Beta releases […], but literally almost nobody uses them. […]. CentOS Stream will change this.”
What is the reasoning behind this? Why would anyone suddenly be interested in that?
So, I was being hyperbolic. People surely use the RHEL Alphas/Betas, but it’s a huge commitment on both sides – for customers and engineering. In particular, in our subsystem team (Podman), we release quite often, basically every 12 weeks. The Alphas and Betas are released so long before the next dot release that the version of Podman in the Betas ends up being much older than what we release in the GA dot release. It “kind of helps” but it’s not what our customers want. They’d rather wait 3-4 weeks before GA and just get early access with a few more weeks of testing new features (not necessarily lower stability which is the common perception). Our subsystem team can share the exact version of Podman that will be released in the next GA release as long as we can get all of the gating tests to work. This essentially makes CentOS Stream Always Ready RHEL (which is what we used to call this build internally).
About faster updates:
I can only speak from personal experience, but I want *less* updates. I do want to have security updates, but often I have neither time nor budget for continuous changes. I even have one machine which got its last update 4 years ago, because replacing that machine would mean throwing a lot of dependent hardware away. (It’s also the same reason I still depend on Windows XP on some (non-networked) machines.)
I couldn’t get the budget for fully replacing that thing even if it broke down tomorrow – it would need some non-optimal temporary solution.
I think a lot of people want less updates. That’s why that’s a major part of RHEL’s value proposition. CentOS Stream is slightly faster than RHEL, but WAY slower than Fedora. It’s a compromise. It’s not as slow as downstream CentOS. We’ve decided that RHEL serves this role and we’d rather make free/low cost RHEL available to meet those needs than continue to feed a brand called CentOS. See this article for some details on free and low cost solutions using RHEL. They might be quite surprising: https://www.redhat.com/en/blog/new-year-new-red-hat-enterprise-linux-programs-easier-ways-access-rhel
About support time and trust:
Maybe it would have been reasonable to reduce it to the same timeframe as CentOS 8 Stream (like 2024 or so), but reducing it from 10 years to less than three years is kind of a middle finger to the users. Having its deadline now before the deadline for CentOS 7 adds quite some absurdity.
Per above, we’d rather move these people to no/low cost RHEL.
Mind you, I’ve only got about eight physical CentOS-Machines left (not counting VMs), so I’m not sure if my opinion is too relevant here – I’m almost exclusively on Debian and Fedora, even switched a few boxes from CentOS to Fedora Server about half a year ago. (I think I already anticipated that some of the changes in CentOS wouldn’t go well with my needs and views, but I didn’t think that they were that abrupt and that far-reaching)
So, your eight boxes would be well within the 16 node limit for production for free. The new announcement should make your life easier, not harder. And, it will be real RHEL, not CentOS.
t userbase:
In the last few years I’ve been the primary driver in my local community for rpm-based stuff – either because I’ve been the person setting the stuff up or because I’ve been asked for opinions about which OS to choose. But when looking back the last two years I’ve essentially stopped recommending it since it started to feel like RedHat is applying pressure to the community. Mostly I feel that in fedora, because I use that more than CentOS – lots of small things – CPE deciding important stuff without community inclusion, and forcing them through even though there is almost no support for that (GitLab); ELN; Modularity; lots of decisions in both CentOS and Fedora have an (implicit or explicit) addition somewhere along the lines of “we decided that way, because RedHat wanted it that way”.
I don’t have numbers, but I think the “loosing the userbase”-process has already started before this decision – what comes to mind is the epel8 repository – it feels like a lot of packages have lost maintainers for the EPEL repositories or even don’t have any builds for epel8 (or ELN) at all anymore.
This is interesting feedback. For Fedora, I can’t really comment, as admittedly I’m not super close to the community goings ons. Modularity is a touchy subject which I am aware of. I believe EPEL has always had this problem, especially when a new major version drops. CentOS never had any real independence from RHEL from a feature perspective. A downstream rebuild, by it’s very nature and purpose can’t innovate. That’s exactly why we think CentOS Stream 8 is the future. It’s a very strategic move to reinvigorate RHEL development. We do believe it has become too separated from the community. It was/is a myth that downstream CentOS ever generated RHEL users. That just never happened. At least CentOS Stream users will help in the value creation process for the RHEL ecosystem.
About binary compatibility:
This has been also noted in another comment already. If this deviates too far between CentOS and RHEL this means splitting the non-redhat-distributed (to a large degree proprietary) software stack – currently a lot of requested proprietary software is available for [Windows, MacOSX, Ubuntu, Android, iOS], and some have the set [Windows, MacOSX, Ubuntu, Debian, Fedora, CentOS, tar.gz] – so having to split the CentOS package into CentOS and RHEL would produce the same problem as Debian/Ubuntu currently have – those packages for Ubuntu downloadable from the vendor pages or github releases pages (and there are a lot!) quite often don’t run on Debian properly anymore.
I’m still SO confused on people’s perceptions about this one. CentOS was NEVER binary compatible with RHEL. That was a misperception because things “just worked.” CentOS Stream is not like Ubuntu and Debian, that’s a bad comparison. CentOS Stream is more like RDO to Red Hat OpenStack or OKD to OpenShift. These are sufficiently complex pieces of software that having an upstream build from the same sources (and the next versions of back ports/fixes/features).
CentOS Stream can be described as “The latest version of RHEL with spoonfuls of code added nightly. The spoonfuls of code are back ports, CVE fixes and small Features (rarely is there a large feature in RHEL, the ABI/API compatibility promise mostly prevents crazy changes). These spoonfuls of code also must pass all of the gating tests in our CI/CD system. That means CentOS Stream is basically Always Ready RHEL.
If somebody, really wanted to slow it down further, they could easily still a Forman Server in front of the updates and promote Content Views on the days RHEL dot releases drop. This is s such a small speed bump, but it forces community members to pay attention and participate. This is good for the sustainability of RHEL. If RHEL isn’t sustainable, then neither is CentOS. There is a virtuous cycle that a lot of people are missing.
Hope a bit of this text gives a bit of insight or ideas.
It does. Again, thank you for the long feedback, it’s much appreciated.
Another great response on why CentOS Stream is more interesting than RHEL Betas. You don’t have to reinstall every time (really annoying): https://lwn.net/Articles/843350/
I have been a RHL user since the 96, ran fedoraforums.org with other community contributor. Understanding that Fedora is the “testing ground”. To be honest, RH has no business acquiring CentOS in 2014, it has too much conflict of interests and regardless what is the reason removing release cycle, it is always seen as money grab. We assisted many NPOs migrating from Windows to CentOS to lower their operation cost, again they are NPO and makes little aside from donations – it left a bitter taste in many consultant like me.
Trust is more expensive than anything else in business, RH has broken one, a big one. No more contribs from me – not even on fedora project
I’m really sorry to hear that. I really am.
So. I am the distribution engineer for an embedded appliance product. Traditionally, we have used CentOS Linux as our base for these reasons:
1. We can’t ask customers of an embedded appliance to pay for RHEL licenses for obvious reasons, so RHEL is not an option. Red Hat’s licensing is based around servers, not around embedded appliances, and isn’t appropriate for our business model.
2. Long support life cycle.. This means we don’t have to re-base our proprietary software that runs on top of the distribution very often.
3. Prompt security updates. Rather than have to maintain a team that does nothing but look at CVE’s and check to see whether our embedded distribution is succeptible and manually update packages, we simply apply any RPM updates that have come out to fix security problems as they come out, and voila, done.
4. Continuity with prior releases of Red Hat Enterprise Linux. We don’t have to ret-train engineers on how to, e.g., do advanced network configuration. The skills they learned with RHEL 5 twenty years ago still work in RHEL 7, with some caveats.
5. A commitment that Red Hat Software made in 2016, when it acquired the CentOS project, that it would continue the CentOS project under the same terms and conditions as the prior independent CentOS project.
Red Hat Software has basically broken all of that with its recent RHEL 8 / CentOS 8 decisions. Red Hat made the decision for RHEL 8 to steer customers to its own solutions rather than existing community solutions for things like, e.g. Java applet container servers, and broke the continuity with prior releases of Red Hat Enterprise Linux (note, that’s just one example). Red Hat made the decision to break their 2016 committments to the CentOS project and break the long life cycle and end of life guarantees. CentOS Stream is not really an alternative because we’re not Facebook — we don’t have the staff to monitor a continuous stream of RPM’s of variable quality from alpha to essential security patch in order to apply to our appliance distribution.
This is not a communications problem. This is Red Hat Software going back on commitments they made to the world in 2016.
I am currently looking at what is required to re-engineer our application for Ubuntu LTS. It doesn’t look like it’s going to be difficult, since we use automation to create our distribution images that is to great extent distribution-independent. Red Hat Software’s current management has proven that they are not honorable people who will live up to the commitments they made in writing, so there is no longer any reason for us to trust any promises that Red Hat Software makes to us. I am sure that right now Red Hat is saying “but you did not pay us any money anyhow so we don’t care”, but consider this: we have multiple college interns onboarding every semester. Those college interns learn whatever Linux distribution our product is based upon. Once they graduate, they go out in the world and are the next generation of decision makers. So they have a decision once they rise in corporate ranks whether to buy Red Hat Enterprise Linux or a Ubuntu Linux support contract with Canonical.
Which decision do you think they’ll make?
Mindshare counts, and Red Hat has basically lost the next generation of decision makers, who might be using a free distribution now — but that doesn’t mean they’re always going to be poor college students. Linux won the server wars because of poor college students who graduated and became decision makers — Linux was free to use, so they became Linux experts rather than Windows experts. It looks like the current management of Red Hat Software has forgotten how they got where they are.
The sad part is, all your company would have had to do is talk to us about an embedded deal. We do them all the time. The pricing is completely different, but apparently CentOS did NOT lead to you running RHEL like everyone keeps claiming. It didn’t even lead to you having a conversation with Red Hat to discover, and discuss the embedded options.
Why would we pay for RHEL 8 when we can run Ubuntu for free and it will actually require *less* engineering time on our part compared to porting our software to RHEL 8? Ubuntu includes functionality that Red Hat removed in RHEL 8, functionality that we use in RHEL 7. And fundamentally, Linux is Linux. Our automation tools that we use to create our appliance image abstract away most of the distribution differences. E.g. we tell it we need packages x, y, and z, and it installs them into the image. Whether it is being installed by apt or yum is invisible to us.
Now I hear you saying “but you’re not paying Red Hat any money so why should Red Hat care?” Well, because the student interns who come through our company are going to be learning Ubuntu now, not Red Hat. Ubuntu’s mindshare grows. Red Hat’s declines. When they graduate and rise in the corporate ranks, they will be signing support contracts with Canonical, not with Red Hat Software, for the corporate servers.
Fundamentally, Red Hat is eating its seed corn. RHEL already had major issues with the release of RHEL 8, which broke continuity with earlier releases of RHEL in ways that are significant to application vendors. We were already looking at a significant effort to port to RHEL 8 due to missing functionality available in RHEL 7 but which would require work from us to provide in RHEL 8 — missing functionality which is available in Ubuntu 20.04 LTS. At this point we have a choice: we make a major effort to transition our product to RHEL 8, or we make a major effort to transition our product to Ubuntu 20.04 LTS. In the end it comes down to dollars and cents — we will be going to Ubuntu 20.04 LTS. And because we as a corporate entity standardize on a single Linux distribution for *everything*, our corporate infrastructure will eventually be coming along for the ride too — meaning any revenues from that will be acruing to Canonical, not to Red Hat Software.
These decisions may result in short term gain for Red Hat Software as people running CentOS 8 in corporate environments transition to RHEL. But in the long term, it’s as disasterous as Microsoft’s pricing decisions for Windows Server in the late 2000’s — pricing decisions which worked for Microsoft’s short term profit but led to Linux, not Windows, becoming the standard operating system of the cloud.
First of all, I have to commend you on answering so many of these comments, despite disagreeing with a lot of them. That has been no small task, and I am certain that it has taken it’s tool on you. I honestly think that most of us get most of it. Like you said, this move is about sustaining Red Hat, and I think all of us get that… well, business is business. Some have mentioned the anemic ~2% of Red Hat share of enterprise servers versus the ~19% of the CentOS share and the ~48% of the Ubuntu share , and I think here is the point of frustration for some. Why WOULD so many places, with a production environment, choose CentOS or Ubuntu over Red Hat? It’s simple price. We have RHEL severs here, for our MOST critical servers, but quite frankly you are expensive. When compared to your competitors, what you offer simply isn’t THAT much better. Is it better? Yep. Is it better enough to warrant the thousands of dollars it would take to run RHEL on everything we would like to run it on. Hell no. Look, I love you guys, and I have been singing the Red Hat song for all to hear since I started learning Linux, but what those enterprise server numbers show is that you guys are simply overpriced. We aren’t huge, but we have over 15,000 nodes, and as a not-for-profit we simply can’t afford you. You could say “Talk to someone at Red Head” (and we have a rep), but honestly, it’s like haggling with a used car salesman. I get it. You need to make money. The rep’s job is try and get as much as they can, but when a company like mine says “we have exactly this much to spend here” then we have exactly that much to spend. There is no negotiating here, because we have nothing to negotiate with. We have a very finite and fixed budget, the wage index is what it is, and a canned response or canned bundle/plan very rarely comes close to filling our needs. We have been re-evaluating our environment almost since the moment Red Hat said that security for CentOS stream, despite being upstream from Red Hat, would not have an acceptable SLA. Since then, we have been in talks with other vendors who seem much more interested in working with us and our economic restrictions out of the gate. The initial quote from Red Hat was so far off from what would be practical for us that we haven’t even bothered with talking further about it with anyone from Red Hat, and we were up front with the money we had to spend.
I have been long in getting to this point, I know, and it was in an attempt to show the disconnect, but here is the bottom line. The ~2% of the enterprise server market has less to do with something like CentOS being free (we would much rather pay for a license that use something that requires a lot of worker hours that we don’t have to spare, but it has to be a practical cost) and more to do with your current pricing structure/model. It may have been noted, but I can’t remember for sure, but a lot of those Ubuntu servers running in production environments are not free. You can say it isn’t about the money, and I am sure it isn’t all about the money, but it’s hard to back that argument up when your own free downstream product was kicking your tail. IF CentOS was in the 48% range and Ubuntu was in the 19% range, then I would get this move a little more, but even adding in CentOS, the Red Hat “family” comes in at just 21%. I understand removing one cost to take on another, we have to do that penny for penny, but this was poorly executed by a company that seems to be financially able to absorb some cost for good will and to save reputation. We did not start our migrations from CentOS 6 servers to CentOS 8 and RHEL 6 to RHEL 8 in December as was planned, and were instead migrated to another distribution. If Red Hat had just adopted the EOS for CentOS 7, I think you would have had time to prove that CentOS Stream could be viable for some of the users to continue to use. Proof, not promises, because like it or not, your camp’s word has taken a big hit, just ask the companies that adopted your self-support Ansible Tower model sub. Those teams are having to scramble now, and they WERE paying Red Hat. My 2c, re-evaluate your pricing with more levels and more flexibility and stop haggling like used car salesman.
Sorry, I read these responses in reverse. I totally understand your point, but you have to understand that open source is a TOUGH business. There are two principles in economics which I’d like to highlight. First is Marginal Utility [2]. The second is Subjective Value Theory [3]. In a way, open source makes it difficult to tease these theories apart. Normally, Marginal Utility would inform us on the value of having 75 Tea Pots, but open source and software in general is easy to copy and use in many, many places. Subjective Value Theory informs us that different value propositions in a product will have different values to different customers in different use cases.
Now, take pricing and packaging, a key job of product managers for any product or service. Pricing and packaging has a profound effect on how people perceive a product, how much value they feel they are capturing and how much value the company delivering the product is capturing. Long story short, it’s outrageously complex.
I get your frustration, I really do. But, it’s super difficult to make these decisions. Building a successful business is difficult but billionaires joke that it’s more difficult to keep money than make money. The same is true with products. I’m over sharing a bit, but suffice to say we are trying to make the best decisions that we can about the pricing and packaging of RHEL. We are creating a lot more low/no cost subscriptions and trying to balance the free to fee sides of the scale.
I do really value this feedback and we are listening. The CentOS change isn’t about generating top line revenue and capturing value for Red Hat (aka making money). It is about protecting the bottom line costs and making sure we are investing in the right direction to support RHEL and OpenShift.
[1]: https://opensource.com/article/21/2/differentiating-products-upstream-suppliers
[2]: https://en.wikipedia.org/wiki/Marginal_utility
[3]: https://en.wikipedia.org/wiki/Subjective_theory_of_value
BUT security is already an issue with CentOS stream, right? A slower than normal SLA for security. If it’s upstream, why wouldn’t security go there first? Maybe I am wrong here, but this seems the “Pay Us!” carrot here. It is this point in the CentOS Stream discussion where my company and I bail out. Being ahead in development but behind in security makes CentOS Stream a 100% no go for my company. We are a not-for profit Health System and we have a mixed environment of RHEL and CentOS, and security is the #1 concern with regards to when we patch and what OS we are using. We like Red Hat, but you are expensive in regards to the value you provide in comparison to the models of your competitors. We had begun to evaluate Ansible as a config manager earlier in the year, but bailed because of the ridiculously over-priced price model, especially after the 5k per 100 node no support model was pulled. Even if we could look past your lack of concern of the security of the Cent you are going to provide, and HIPPA says we can’t look past it, what’s to stop you from simply deciding you aren’t going to focus on a different aspect of Stream and reserve that only for your paying customers? Given the behavior Red Hat has exhibited here, and the way they handled the subs with those who had adopted the 5k Ansible model, why should we have any faith that if we moved to the paid sub, Red Hat wouldn’t just more than double the price on the subs and give us ~1 year to come up with the money or find another solution. For us, it appears time to thank Red Hat for a decade long relationship and re-evaluate which distribution we need to use to migrate to while we begin to sunset our systems currently employing Red Hat/Cent systems.
Well, the first off, I’ve been at Red Hat for 10 years. For 10 years, the price of a standard RHEL subscription has been $799. Many services I use change their pricing and packaging. HBO upped my subscription price. Disney did it this month. LastPass changed their subscription model last month. I mean, I can name product after product that changes the pricing and packaging. This isn’t even a price change. CentOS Stream is just a packaging change. The cost is still $0.
If your company is making money, and I’m sure they are, then it’s difficult for me to understand you being upset with this change. Nobody is giving me free stuff to run my business. Security is a key feature people pay for with RHEL. I would recommend one of the many downstream rebuilds of RHEL if you really need $0 Enterprise Linux.
But i just wanna know (reading is hard) does this all mean I i wont be able to get CentOS with the latest code to learn REHL? Is the freebee cut off for us learners and explorers? Has money really taken over and the curious and the wannabees been pushed to the side for a dollar?
CentOS Stream is future RHEL, so you will always have that to learn on. And, since it comes from Red Hat, and is essentially the next dot release of RHEL, you are actually in a place to learn the latest greatest faster than RHEL and downstream rebuilds.
The only reason people used CentOS is because it was stable and supported for 10 years vs Ubuntu LTS, and it is thus the *other* Enterprise Linux distro to use.
There’s a reason Microsoft Windows is popular, and the primary reasons are 1) Central Management, 2) 10 year lifecycle, 3) Copious documentation and workarounds available on The Google. Originally, the popularity was also a result of 3) GUI Management Consoles for all admin tasks and 4) Ease of development of software, but these advantages no longer apply thanks to advancements in Linux. Guess what, neither 3) or 4) were solved by Red Hat. Neither are nearly most of the useful documentation and tutorials for configuring software on RHEL/CentOS RHEL tutorials; most of them are CentOS tutorials.
Just change this to a 10 year lifecycle and promise to keep patches of RHEL to within a specific timeframe of CentOS patches, and most the complaints go away. Otherwise, people will move to other distros anyway that track RHEL, and you will have shot yourself in the foot and ruined all the benefits you’ve delineated here you might have gained had CentOS continued to be popular. So, which is it? Do you want it to succeed or is it a burden on the company? You can’t have it both ways. Thus, these arguments are disingenuous.
We want people to use RHEL instead of CentOS. We are doing a ton to provide no cost/low cost RHEL subscriptions. The 16 production nodes provided by the recent RHEL Developer Sub changes is a great start. We’re doing similar things for HPC/Science use cases, and learning environments. This is part of a major revamp of the way we release RHEL to the world. We want CentOS Stream to be for upstream development. IMHO, if we would have/could have made these changes to RHEL earlier, CentOS wouldn’t even have existed (similar to how SUSE and Ubuntu have no downstream rebuild). My 2c.
I’ll note that CentOS existed because of the uproar over the discontinuation of Red Hat Linux in favor of the ( Red Hat Enterprise Linux / Fedora ) pair, where RHEL was commercial and Fedora moved too fast for anybody other than Linux OS developers. At the time that these decisions were made by Red Hat, Red Hat had probably 75% of the market share for Linux, with SuSE, Mandrake, and Debian being far behind and all of those were available for free at the time. And yes, I was there. The first Red Hat Linux I installed was Red Hat 3.0.3 in 1996. In fact, we bought dozens of the boxed sets to distribute with our servers at the time in order to destigmatize the fact that we were using Linux, it was like “sure, we’re using Linux, but we’re using real commercial Linux with a manual and everything, not some hackerware downloaded off the Internet.” (Yes, I’ve worked for server or appliance vendors for the past 25 years).
And you are disingenous when you state there are no community rebuilds of the other distributions. Heck, Ubuntu originated as a community rebuild of Debian originally, and there are many community rebuilds of Ubuntu, such as Linux Mint. The only major distribution that does *not* have community rebuilds is SuSE Enterprise, and that’s likely because their market share is so tiny (less than 2% of the enterprise server OS market) that nobody cares. Even there, there are community rebuilds of the upstream OpenSUSE for various target audiences.
Ultimately, as with the original decision to create RHEL in the first place, the end result is going to be reduced market share for Red Hat. Red Hat once had 75% of the Linux market. What’s that market share down to now? Even if you include CentOS, that’s down to around 25% maybe of the Linux server market now, with Ubuntu having over 50% market share now. The overall Linux market of course exploded with the cloud, where Linux became the operating system of the cloud, so Red Hat could still maintain increasing revenues and profits despite declining market shares. But now we’re at the end of the cloud explosion. And 50%+ market share for Ubuntu… well, I guess the slide to SuSE Enterprise levels of irrelevancy will take a while, but it’s inevitable. Of course, by the time it happens everybody involved in the decisions that led to that will have cashed out and retired, so (shrug).
You can easily switch your existing CentOS server to AlmaLinux, and get the remaining 8 years of support back.
Just google it and thank me later.
More power to you. I encourage you to participate in and contribute to the Alma Linux community.
You’re way overthinking it. This is just flatline profiteering, crushing faith in the centos brand so IBM can eventually justify killing it altogether and pushing people to the pay-only solution. Not to mention that gross “first one’s free” rhel model.
But hey, who cares that there’s no free distro that’s binary compatible with RHEL. I’m flatly calling bullshit on the whole “centos stream is stable.” It won’t be any more stable than fedora, which I use daily. I like it, but every kernel pull comes with the potential for explosive failure and manual microcode updates. That’s not a stable server strategy by any stretch of the imagination.
I’m not waiting for the “Fedora subscription” to happen, I’m bailing out along with all the RedHatters who are currently shopping their resumes. Krishnan’s gutter punking of RedHat will be complete inside of two years, at which point it’ll just be another fond memory that big blue hooks up to the milking machines and forgets to feed until it dies. I wouldn’t be surprised if the next year doesn’t bring a rhel fork to a new license a la elastic co.
If you want to see the future of redhat open source, look no further than here:
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=4acd47644ef1e1c8f8f5bc40b7cf1c5b9bcbbc4e
I can assure you IBM had nothing to do with this decision. I’m just trying to share the thinking that goes on when you are a Product Manager in situations like this. I’m trying to be transparent in this post so that people can understand the reasoning. There’s no conspiracy.
I discovered AlmaLinux OS.
Loaded it up last night, ironically the day they release their 8.4!
Looks like a great replacement to CentOS with the bonus of 8.4 upgrade!
https://almalinux.org/
Cheers,
EWM
Exactly. Be happy, be excited that down streams like Alma and Rocky are easy to build. People forget that not everything in a Linux distro is GPL. In fact, much of the user space is Apache and BSD licensed. Red Hat goes above and beyond to make all of the source code available in one place, and easy to access. This makes it easier for down streams to rebuild. Now those down streams can suggest changes by contributing to CentOS Stream. it’s actually a win for everyone.
The author seems to have a complete lack of understanding of what happens when corporate enterprises “take over” open source projects for their own profit motives.
AKA openoffice vs libraoffice.
Where is openoffice now? Used by hardly anyone.
I’m the author here. I’ve contributed to and even been a core contributor to open source projects. I’m not understanding where you are coming from. I’m sorry that you are upset, but leaving 10+ comments is not productive to the conversation.
The problem is not in switching to the CentOS stream.
– The problem is a broken word (support time).
– The problem is the loss of trust and credibility.
So the loss of this rare commodity.
Its loss is too high a price that no article of any length can fix.
I’m very sorry that this happened, and it’s even sadder that you don’t understand how bad it is and how much damage it caused.
I would suggest Alma or Rocky Linux. They are both down stream rebuilds. Red Hat makes all of the source code available and easy to access in one place (even the Apache, BSD and other licenses which don’t require redistribution). This makes it easier for Alma and Rocky. Also, now Rocky and Alma developers/contributors can participating in changing the down streams, by contributing to CentOS Stream. This is a win for everyone. You can consume for free, and contribute upstream. Before, you could only consume for free, but it was quite difficult to suggest changes in RHEL without a subscription.
Dear @fatherlinux,
I wasn’t “Mad About The CentOS Stream Change”… I really looked at it as normal repositioning.
We were having “Fedora » RHEL » CentOS” and now you’re just moving it to “Fedora » CentOS Stream » RHEL”.
To all the people shaming RH I said… wait and see what happens!
Until today, February 4th when I received the Ansible newsletter.
Buried in there was “Withdrawal of CentOS 8 test containers” ( https://github.com/ansible-collections/news-for-maintainers/issues/3 ).
It’s not a move from CentOS 8 to CentOS 8 Stream… no… it’s “withdrawal of CentOS 8″… and probably no support for CentOS 9 Stream… CentOS 10 Stream… and so on!
If “CentOS Stream” is supposed to be more stable than Fedora but little less stable than RHEL… it would be rational to ditch Fedora and move to “CentOS Stream”. But they ditched CentOS and kept Fedora.
A lot of FUD on the move to CentOS Stream… the move above… reignites it!
Thoughts?
Disclaimer:
– I’ve been a Fedora user on Desktop for more than 15 years
– I’ve been a CentOS fan since forever
– I’m an Ansible fan… I’ve even had presentations at local conferences
eRadical,
Thanks for reaching out. I definitely understand your point of view on this. It “sounds like” it was just a resource thing. Since they were packaging/testing on RHEL upstream, they didn’t feel like they needed to have an entire branch for CS9. That said, I would encourage you to make your opinion know with the Ansible upstream team, especially if you have needs specifically for CS9!
Really nice article. I am approaching “RPM world” in this moment of change, and a question keeps coming in my head. If Centos Stream 9 (based on Fedora 34) is a rolling release, what will happen when Red Hat 10 will come (based on Fedora XYZ)? Will Centos 9 keep its road going or it will stop “rolling” and Centos 10 will come? Is this change from 9 to 10 the 5 years span you are talking about?
Yes, CentOS 8 for example will run until May 2024 [1]. This is not a coincidence, this is when RHEL 8 will be at 8.10 it’s final release which will have major features. At this point, RHEL goes into a much slower development phase, and the focus of CentOS Stream 8 will move onto CentOS 9. The same thing will happen to CentOS 9 about 5 years after RHEL 9 is launched, and 2 years after RHEL 10 is out. The plan is to have all of these overlap this way going forward. That means, you will have a two year window to move from CentOS Stream 8 to CentOS Stream 9, and the same going from 9-10. Does that help? Admittedly, we haven’t done a good job of explaining this grand vision, and I think I need to add it to my nex “State of Enterprise Linux” [3] blog at the end of this. You’re giving me ideas for new material!!! 🙂
[1]: https://endoflife.date/centos
[2]: https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
[3]: https://crunchtools.com/the-state-of-enterprise-linux-in-2022/
This is really complex! It’s not simple FOSS as in the old days. Thank you for explaining how Red Hat benefits and the community does as well. I see nothing wrong with a win/win situation.
¡Saludos desde México! 0/
First, I don’t use Centos in an Enterprise capacity, nor do I recommend it to anyone else, because it is always behind on anything security related.
That being said, I might have found your blog interesting except for a detail.
“Who Moved my Cheese” is a piece of sardonic patronizing tripe that is used by middle managers in an effort to pursuade people to accept a layof in the most patronizing and demeaning way, with absurd stereo types. Because you quoted that load of garbage, I immediatly hold your opinion suspect, and likely full of hidden bad intentions.
Tell me I’m wrong. .
You’re wrong. I had to read Who Moved My Cheese when I was like 25, and I hated it. I am perennially annoyed by any kind of corporate training hype. Who Moved My Cheese was definitely corporate training hype. So was Shackleton, Discover Your Strengths and boatloads of other training like that. That said, I had a moment of lapsed judgement. I used it as a shortcut trope, partially because it was late on a Friday night when I wrote this piece, and partially because I was being lazy.
My intentions are good. I was just annoyed. And, it came out in my choice of trope. But, I do appreciate your, what I think is shared, ire for these tropes.
How do you even purchase RHEL? I tried and found the experience awful. All the sales people wanted to do was sell us Ansible and a few other products. Not once did anyone want to discuss what license/subscription level/model or however the hell they sell RHEL. Once I made it clear we weren’t interested in Ansible and whatever else they were selling, that all I needed was to run a handful of RHEL servers the conversation all but ended. The whole thing was drawn out and pointless. I could have migrated all my servers to another distro in the time that was wasted.
I’m really sorry to hear that. That experience shouldn’t have happened. If I can help in any way, let me know. I can try to reach out to our sales team and figure out what happened. At a minimum, they should see this comment from you so that they can improve their interactions.
L FREAKING OH EL. Go Shill somewhere else man. Company company company. Talking about value propositions as though there isn’t a massive amount of unpaid code coming in on the front end. Giving back to the community is what RH should do to justify profiting off of the work of others. Your moral arguments literally show you have a poor grasp of the work going on outside and in front of your own ecosystem. Not that this comment will get published if I say this, but maybe you will see it in your auto-mod logs. Honestly… eat a dick.
I’m sorry you’re this angry about the CentOS change. I work with litrally thousands of open source developers, and hundreds of them are my personal friends. I know how tough their jobs are, and I appreciate how hard they work. I also appreciate that many of them get pay checks to buy houses, feed their families, etc. Red Hat goes above and beyond with RHEL, releasing code that is protected by the GPL, as well as BSD, MIT, Apache, etc, many of which have no requirement to release. I defintiely contemplating not publishing this response because you insulted me at the end, but I realized that it’s annoynymous anyway, so it doesn’t really matter.
Please correct me if I’m wrong – but CentOS doesn’t publish any vulnerability feeds right? So many tools have a hard time identifying security vulnerabilities in Centos 9 stream. This is available in RHEL 9. Is that going to be fixed or is this a feature used to compel adoption of RHEL?
That’s essentially correct, there is no database of CVEs which apply to CentOS Stream which is provided by Red Hat. There are third party scanners and databases, and they compile their own data from scratch. Some of those are free, some are pay-for. The only vulnerability feed provided by Red Hat is with a RHEL subscription.