Background

Impetus

Recently, I saw a discussion was started, asking about the importance of the operating system on the LinkedIn OpenStack group:

Ubuntu can overtake Red Hat in private clouds because the OS doesn’t really matter

I found the conversation wildly interesting because several people expressed reasons for why they think the operating system does or doesn’t matter. The comment below expresses an opinion that it does matter, but that having “old” versions of a kernel negatively affect the potential to deploy OpenStack:

RHEL6’s kernel is too ancient for OpenStack. and RHEL7 is in beta for a very long time …

I find that, conversations and comments like these are very common. The danger is, so many people feel like they intuitively understand the way Linux distributions are developed and released. In this article I want to clarify how Red Hat and others leverage and contribute to the Open Source Software Supply Chain. My intent is to clarify why the operating system matters and explain what the version number of packages, including the kernel, have come to mean.

Vocabulary

It is critical to have a basic understanding of two important words in software development: Rebasing and Backporting. In this article, I will use the following Wikipedia definitions:

Rebasing

Rebasing is the act of moving changesets to a different branch when using a revision control system, or, in some systems, by synchronizing a branch with the originating branch by merging all new changes in the latter to the former.

Backporting

Backporting is the action of taking parts from a newer version of a software system or software component and porting it to an older version of the same software.

 

Open Source Infrastructure Software

I will use the Linux kernel as the main example in this article, but the concepts discussed here, apply to many pieces of infrastructure software. Examples include:

• User space tools
• Libraries
• Linux Distributions
• OpenStack
• oVirt
• Gluster

Kernel Use Cases

The Linux kernel is a piece of infrastructure software that is used in a lot of different ways throughout the Open Source Software Supply Chain.

The chart below, developed with data from the Linux Foundation’s 2012 Report, shows the top 20 contributors to the Linux Kernel.

One can imagine many different drivers for consuming and contributing to the Linux kernel. Examples include:

• As part of a research project for a graduate student: A student may experiment with the TCP stack discovering a way to lower latency with certain types of packets. If the project succeeds a patch may be submitted. If the work fails, nothing will ever go back into the Linux kernel. In this scenario, it may be desirable to work with the latest release of the kernel.
• As part of a hardware development project, such as the next generation of microchips: Researchers at Intel, Texas Instruments, Ericsson and other want to add changes to support new features in their hardware. This will influence which versions of the kernel they use to develop their hardware. Hardware vendors often work under the assumption that their changes will be back ported to older version of the Linux kernel.
• As a sub-component in an open source operating system, such as Red Hat Enterprise Linux, Fedora, or Android: In this scenario, stability may be balanced with features. As such, re-basing less often from kernel.org and back porting changes may be quite attractive.
• As part of an embedded system, such as an electric charging station: In an electric charging station, the developers may not be extremely concerned with the version of the kernel. For example, much of their work may rely on Java, so they just need a stable place to run a JVM.
• As part of a smart phone: Typically, companies like Samsung or HTC will rely on whatever kernel is built into a distribution of Linux such as Android or Tizen.
• As part of a connected video game console: For example, Sony, with their PlayStation may have very specific requirements around which versions of the Linux kernel they will use for the life of the video game console. They may or may not want to re-base from kernel.org.
• As part of a virtual appliance: If for example, the appliance is based on Java, or Python, the version of the kernel may be of very little interest to the developers.
• As part of a private operating system, such as internally at Amazon or Google: These companies have their own schedules for rebasing from kernel.org and hence releasing a new “version” of their OS.
• As a foundation for another project, such as KVM, or OpenStack: For projects like these, many versions of the kernel and associated user space tools will be important. One can envision that OpenStack and KVM will run for many years in production and as such will run on various versions of the underlying kernel. To solve this challenge, companies like Red Hat create a support matrix of operating system versions and OpenStack versions that will be supported together.

 

Kernel Versions

Notice, that at the time of this writing, the 3.15 kernel was the current mainline kernel. Historically, the mainline kernel is what is consumed and supported when Linux distributions such as Debian, Ubuntu, Fedora, do their releases. Each distribution has a different philosophy on versions. Some distributions support only one kernel version during their support lifecycle, while others periodically update to the latest stable kernel version.

Red Hat Enterprise Linux has a different philosophy than other Linux distributions. During major releases, Red Hat rebases from the upstream linux kernel. During minor releases, Red Hat back ports changes.

Major versions of Red Hat Enterprise Linux are released about once every 3-5 years. Rebasing such a large code base is difficult work and Red Hat does significant testing and QA. For all software, including the Linux kernel, approximately 5596 bugs were tracked, fixed and resolved between Fedora 12/13 and the release of RHEL 6.0 on November 8th, 2010.

Once complete, Red Hat supports the certified version of the kernel for 13 years, which means ABI/API compatibility, patches, phone support, hardware certification, etc.

To preserve ABI/API compatibility, Red Hat carefully back ports features, bug fixes, and security fixes. Red Hat continues to do this work for all currently supported versions of RHEL. At the time of this writing the primary work was on RHEL 5 (2.6.18 kernel), and RHEL 6 (2.6.32 kernel). Don’t let the version number of the RHEL 6 kernel trick you, it does not immediately indicate what “features” are in the kernel. Even though the kernel is version 2.6.32, it has features back ported from the current mainline kernel.

This backporting methodology is further refined by three phases of development. During minor releases (5.1, 5.2 or 6.1, 6.2, 6.3), patches are released for the Linux kernel and other software according to the following summarized description:

• Production Phase 1: Security Fixes, Bug Fixes, Features (Hardware/Software Enablement)
• Production Phase 2: Security Fixes, Bug Fixes, Features (Minor Hardware Enablement)
• Production Phase 3: Security Fixes, Bug Fixes
• Extended Life Phase: Limited Security Fixes, Limited Bug Fixes

Backporting of kernel changes during the lifecyle of the major versions RHEL has another interesting byproduct; the set of open bugs, and their resolutions is well understood. For example, all of the RHEL 6 kernel source code changes are tracked publicly at bugzilla.redhat.com. There is no rebasing, so every change to the 2.6.32 source tree is tracked and understood before being applied!

Furthermore, Red Hat works together with the Linux kernel developer community. The community is completely aware of Red Hat’s backporting methodology and supports the longterm kernel. The goal of the longterm kernel, is a set of stable, well known and supportable features.

Brief History

Backporting was not always accepted as best practice and there was originally some debate. In 2004, the CTO of SUSE publicly stated that he did not think it was a good idea to backport changes from the 2.6 kernel to the 2.4 kernel. At the time, everyone was worried about fragmentation (sound familiar?). On the other hand, Linus Torvalds’ response was:

I think it makes sense from a company standpoint to basically ‘cherry-pick’ stuff from the development version that they feel is important to their customers. And in that sense I think the back-porting is actually a very good thing

Coincidentally, the first longterm kernel was started by, then SUSE employee, Greg Kroah-Hartman with version 2.6.16. The 2.6.16 kernel became standard for SUSE 10.X releases, while the 2.6.18 became standard for Red Hat Enterprise Linux 5.X releases.

Seeing that this worked well, a cabal of developers got together at a few different Linux conferences and determined that based on their future distro release cycles, we could all aim for standardizing on the 2.6.32 kernel, saving us all time and energy in the long run. We turned around and planted the proper seeds within the different organizations and low-and-behold, project managers figured that this was their idea and sold it to the rest of the groups and made it happen. Right now all of the major “enterprise” and “stable” distro releases are based on the 2.6.32 kernel, making this trial a huge success

Greg still maintains the longterm kernels and other organizations are finding needs for longterm kernels, such as the 2.6.34 and 2.6.35 kernels. Furthermore, the LTSI industry wide project which:

creates and maintains a long-term industry tree, which is expected to be stable in quality for the typical lifetime of a consumer electronics product, typically 2-3 years

Red Hat is not the only large organization backporting changes to longterm kernels. The LTSI project creates a staging tree and a stable tree, once per year, and originally started with the the 3.0 longterm kernel. Creating a stable branch allows the group to add features and bug fixes, respectively. These LTSI trees have their own maintainers, project managers, and systems administrators.

Conclusion

As of this writing, some of the code in the 3.14 kernel is 20 years old, while other parts are hours or days old. The same is true with the 2.6.32 longterm kernel!

So, the next time, you see someone posting information about how old one distribution is over another, please point them to this article to help clarify