Background

The goal of this article is use the OpenShift Platform as a Service (PaaS) as a learning platform for Django. Most of the technical articles out there about running Django on OpenShift assume the user already understands how to administer Django environments and projects. This article is written from the perspective of someone who has done some python programming and wants to learn some Django without doing a bunch of setup work.

Since each OpenShift Gear “…is a container with a set of resources that allows users to run their applications”, a user can ssh in to test, troubleshoot, debug and learn. This turns out to be quite convenient for learning Django.

First, we must deploy and OpenShift application. The deployment is completely automated with the Django Quickstart. Once completed, the web interface will return all of the connection information necessary for Django, Git, and SSH. Estimate 5 minutes.

https://www.openshift.com/quickstarts/django -> Deploy Now

Once the application and framework are setup, it’s time to log in, do some basic configuration, and setup a developer workflow. This will allow us to focus on learning Django.

Use ssh to connect to the OpenShift Gear and setup a few things. Luckily, most of the work is done for you by OpenShift. Use the connection information provided for your Gear when the OpenShift application was created in the last step.

Each OpenShift Gear has a small data directory where your application’s database is stored. Conveniently, this can also be used for scratch work. Clone your application’s git repo to your data directory. This is not recommended for a production application, but will allow you to start learning Django from a Mac, Windows, or Linux Desktop without worrying if the correct version of Django is installed locally.

Configure a few environmental variables to make your workflow easier

Here are a couple of tests to verify that the Django environment is working correctly:

Login from scratch and create a new Django Application to get comfortable with the workflow. These are the basic instructions to follow every time you connect to the learning environment. We are doing a couple of things here. First, we are creating the application with the Django admin utility, which creates a new directory. Then we are adding the directory to our git repository and committing the change. Finally, we are pushing the change, at which point, OpenShift will take over and perform all of the necessary steps to make your application live.

The following six part Django Tutorial is great, but there are a couple of things to be aware of when working in an OpenShift environment. First, having your git repository checked out in the OpenShift Gear is only a good idea for a learning environment. Second, the portion of the tutorial called The development server will not work properly because the Django Quickstart configures and adds the OpenShift components necessary to start the Django application for you.

Finally, turning debugging on can make running through the tutorial a lot easier

Modify debug statement

Commit and push the changes

At this point, you can run through the tutorial, modifying the data model, and interacting with components in the shell, have fun!

Background

Recently, I came across an article entitled: 5 Reasons Not to Use CentOS. While I actually disagree with all five points from a technical debate standpoint, I think this article is really the result of a few pain points that some developers express when talking about enterprise editions of Linux.

Working as a technology evangelist at Red Hat, I attend a multitude of conferences and community events. When speaking with developers, a common concern is that the versions of packages in Red Hat Enterprise Linux (RHEL) and all downstream rebuilds are too old for how fast they need to move to provide value to their business. I believe that in some cases, this is a completely valid concern.

There is the age old struggle between innovation and stability. There are several factors involved in the stability of an Enterprise Linux distribution including release cycle, support lifecycle, library compatibility (API) and binary compatibility (ABI). Obviously, there are trade-offs between stability and having the latest features, some of which I will explore in this article.

There is much more to stability than how many bugs are tracked and resolved over a particular period in time. Both RHEL5 and RHEL6 will be supported for 10 years respectively. The total lifecyle of support for the platform is important because, the longer it’s supported, the more value developers and users can derive from their application software. Their work can continue to provide business value without being derailed by migration and testing on a newer version of the platform.

To provide an example which both developers and users alike can connect with, take Gnome 2. There have been quite a number of users frustrated with Gnome 3; many just don’t want to use it. This has led to several forks including Cinnamon and MATE. These forks fragment the community, but worse are expensive from a resource perspective. With RHEL6, which is based on Gnome 2, the user is supported until 2020. This means those, that do not want to switch to Gnome 3, have plenty of time. In the case of Gnome 2 vs. Gnome 3, the user is free to find another acceptable desktop or perhaps wait for the next major version to mature, but imagine your business had to support several versions of a major piece of accounting software; how expensive would that be?

What is the Release Cycle?

To provide a little background, RHEL has a target release of every three years, but as with any large software project, release dates often slip. RHEL5 was released GA March 15, 2007 while RHEL6 was released November 10, 2010. This was just shy of four years between RHEL5 and RHEL6.

First and foremost, I have no special insight or authority when it comes to the RHEL product release cycle, but during this time there was tremendous change in the Red Hat Linux world. KVM became the hypervisor of choice and it is obvious that plenty of technical resources were dedicated to KVM and Red Hat Enterprise Virtualization. Making these tremendous changes while at the same time providing stability, takes time.

During the time it took to release RHEL6, the version of the packages in RHEL5 began to show their age. As time went on, the desire for newer packages in RHEL5 seemed to become more pronounced. The early adopters had moved on to newer versions of languages, libraries, and frameworks. The early understanders (which I consider myself) even questioned how long the next release was taking.

When RHEL6 was released, the contrapositive appeared true. The desire for newer packages subsided and developers went back to work on creating applications, knowing that the work they created would provide business value for up to ten years. So, how to balance innovation (features) with stability?

Well, I suspect the resolution to the challenge of older packages is a combination of things. First, I think the problem manifests it’s self most evidently in the versions of languages, frameworks, and libraries in loose order of priority, so I will focus my examples there.

I believe that tighter adherence to the 3 year release cycle combined with Software Collections and the new Developer Subscription could help to alleviate the challenge of older versions of Ruby, Python, PHP, Perl, Java, C and C++.

In addition to tighter adherence to the targeted release cycle, as Ruby, Python, PHP, and even MySQL mature, release of major versions of these software packages may slow down. This may be offset by the challenge of newer languages and frameworks such as Node.js, but this is again where Software Collections may play a major role.

Finally, there is a range of use cases especially across vertical markets. In some markets such as Technology and Media, things will always change quickly, perhaps too quickly for an enterprise distribution of Linux. In my experience, this is the exception and not the rule. In verticals such as Banking, Energy, Manufacturing, Government, Transportation, etc, platform stability combined with some developer discipline on package versions can combine to create applications which provide great business value over a long ROI cycle. This allows developers to focus on creating new application instead of maintaining existing applications.

Better explanation

Background

I manage the crunchtools lab and the infrastructure for this blog similar to a development data center. I have a rigorous weekly checklist, which includes optionally applying operating system patches as they are available. I do not perform the updates every week because of time constraints, but when I do, I patch all of the systems. Most of my infrastructure is built on Red Hat Enterprise Linux, but I run this blog on Linode which doesn’t have an image for Red Hat Enterprise Linux. They have the ability to create a custom image, but I have continued to use the CentOS build, partially to better understand the differences from a hands on perspective.

Several weeks ago, I patched both the CentOS operating system, on which this blog runs, and the other Red Hat Enterprise Linux systems in the crunchtools environment. The latest available patches were applied which caused an Apache web server outage, but only on the CentOS system. Clients could not connect so, I sifted through the system logs, but didn’t see any AVC denials or messages that indicated the cause of the problem. After some experimentation, I realized that SELinux was blocking access to the web server. The SELinux booleans looked fine, so to return to service as quickly as possible, I temporarily disabled SELinux.

A couple of weeks later, I applied some new patches that were available. To some relief, after applying these new patches and testing SELinux, Apache was able to accept connections again.

• Operating System: CentOS 6.4
• Installation Date: Tue 12 Jul 2011 11:24:06 AM EDT
• Provider: Linode
• Hardware: XEN Hypervisor
• Repositories: CentOS-6 – Base, CentOS-6 – Extras, CentOS-6 – Updates
• Optional Repositories: Extra Packages for Enterprise Linux 6 – x86_64

On February 27th updates were applied, which caused the outage. Though perhaps not the cause, the SELinux policy was updated.

At this point, I spent some time digging through the logs and did not see anything to indicate what was blocking the web server. I restarted the web server and eventually decided to reboot the system. When the system came back up, the web server still couldn’t be accessed. At this point, even tough there was no indication that SELinux was blocking access to the web server, I disabled it. As soon as SELinux was disabled, the web server was able to accept connections again.

A couple of weeks later, new patches were applied, which included a new SELinux policy. Once the new policy was installed and I enabled SELinux, the web server was able to accept connections while in enforcing mode

During this strange outage, I decided to analyze some of the core variables which correlated with this outage.

CentOS and RHEL have significant differences in the way they receive and consume updates. From the log snippets below, it can clearly be seen. The number of patches and the dates on which updates are received are different. Notice, CentOS receives 251 patches when it is updated to CentOS 6.4, while RHEL receives 674 patches.

The number of patches is a key difference. When RHEL 6.4 was released, all of the patches were released together. This is because they were all tested together and sent through a quality assurance (QA) cycle together. When CentOS 6.4 was released, a subset of the RHEL patches were released. This means that, on any given patch cycle, the permutation of patches applied to a CentOS operating system is different than RHEL.

A CentOS user does not inherit all of the testing and QA for a given RHEL release or patch set. Since the CentOS repository provides a different permutation of patches, the CentOS team may have to wait until users report a problem to start fixing issues, such as the web server outage described here. This kind of regression bug could have been caught with some kind of automated testing.

Given that the kind of regression described above could have been caught and fixed with automated testing, I decided to analyze the testing process used for CentOS. As I analyzed the pieces of the CentOS test suite which cover Apache and SELinux, I discovered that one of the contributors, Athane Madjoudj, contributes to not only CentOS, but also the Fedora test framework. For this discussion, I will focus on the CentOS versions of the test framework. The test components described from the Fedora framework are only mentioned to provide guidance for possible future integration of SELinux coverage into other tests. Integrated SELinux coverage could have prevented the outage described here.

A quick analysis of the tests around Apache and SELinux show some shortcomings in the test coverage, but the infrastructure necessary to test for behavior described in this post mortem is fairly complicated to set up and maintain.

Since CentOS is a community supported operating system, the CentOS test suite focuses heavily on testing installation. CentOS does not have a large number of high level application testing or post installation testing such as verifying configuration options or kernel variable tuning.

In particular, the CentOS test suite contains httpd_php.sh which is a good place to start this analysis. This script does some basic testing, but has some limitations. Notice that the web server is only tested from localhost. This is less than optimal because sometimes user space utility bugs, kernel network, kernel firewall, or other configuration problems can cause a web server to be available to localhost, but prevent access from a remote machine.

Similarly, the SELinux test checks for AVC entries in the log and whether Enforcing mode is enabled. This is only a very high level test which does not provide much coverage.

In contrast, a couple of Fedora tests do have integrated testing of SELinux. For example, the Tuned test script does do some SELinux testing.

• HTTP/PHP
• SELinux

• List of Test Cases
• Tuned

As a final step, I thought it would be prudent to check the Bug Trackers for CentOS and Red Hat Enterprise Linux for reported problems with SELinux. I searched for http, 155, and 195; I could not find a bug which correlated with SELinux behavior described in this post mortem.

For a point of reference, I went on to investigate the number of open SELinux bugs. ^[1] I found that there were 301 open SELinux bugs open for Red Hat Enterprise Linux and 50 open for CentOS.

The difference in bugs could be interpreted in any number of ways. It may be indicative of more RHEL users testing and finding SELinux bugs^[2], it might indicate that RHEL has more SELinux bugs, or it might indicate that CentOS patches SELinux bugs faster than RHEL, but that is doubtful.

• Red Hat Enterprise Linux: 301 SELinux bugs
• CentOS: 50 SELinux bugs

Red Hat Enterprise Linux Bugzilla

CentOS Bug Tracker

In this particular case, CentOS experienced an outage and I could not quickly determine why. After a quick analysis syslog messages and the audit log without finding anything, I attempted a reboot. Apache would still not respond to connections. At this point, I took a guess and just disabled SELinux, which succeeded in allowing Apache to accept connection. At this point, I saved /var/log/yum.log and /var/log/messages so that I could document the outage here.

Several weeks later, I applied available patches bringing the CentOS system to the latest available patches, and SELinux and Apache began working together again. During this time the RHEL servers in the crunchtools environment did not experience the same outage. This is in no way meant to imply that the inverse couldn’t have happened. I believe it is a is possible that a bug may cause an outage on RHEL but not affect CentOS, because RHEL receives updates first and on any given day, the two bit streams are not the same.

While RHEL and other enterprise Linux rebuilds share upstream source code, the binary builds are not completely the same. Temporally, each distribution contains different packages and versions. That is to say, on any given day, hour and minute, a rebuild cannot be the same as RHEL. Although, RHEL and rebuilds such as CentOS share Major and Minor versions, for example 6.4, each contains different permutations of patch sets. This makes them different in critical ways; rebuild distributions do not inherit all of the benefits from the testing and quality assurance (QA) provided by Red Hat. This is also the reason that rebuilds not inherit the same hardware and software certifications from third party vendors such as HP or SAP.

Architects and engineers will sometimes investigate a mixed environment of a rebuild distribution in development, while using RHEL in production. Typically, this architecture is investigated to save on subscription costs in the development environment. This architecture contains several important caveats. Since rebuilds are downstream distributions from RHEL, they can contain bugs that do not exist in RHEL. Using a downstream distribution to build an upstream test environment creates some interesting challenges.

If a bug is found in the test environment, it could be specific to the rebuild. If it is specific to the rebuild, Red Hat support cannot be called to work on a fix. At this point, the operations team will have to work with the community to develop a patch, or even develop a patch themselves. Even if the bug can be reproduced on a RHEL system, the development environment can’t be patched until Red Hat develops and publishes a patch, and the downstream community rebuilds and distributes it. This could take months and there is no guarantee that the patch will ever be released. Over years, this workflow can create significant technical debt for engineers. Technical debt leads to increased cost.

Increased technical debt between testing and production environments can work both ways. This could lead to untested problems in production which were never found or tested in the development environment. It is better to have a homogeneous environment built from either all CentOS or all RHEL. Having a mix, sets the environment up to accrue, at least some, technical debt, which off sets the costs of running RHEL in development.

Specific to the crunchtools environment, the challenge with managing RHEL and a rebuild like CentOS has been determining if they are at the same revision at any given point in time. The way the RHEL repositories are exposed via channels vs. the mechanism of base, extras, and updates in CentOS make it difficult to have a core build and associated update repositories for management of both over time. It becomes tempting to apply RHEL updates from my Satellite server to a CentOS box, but that would clearly defeat the purpose for which most users deploy a rebuild.

As this post mortem attempts to demonstrates, there are differences between RHEL and any rebuild. Installing and supporting multiple distributions between development and production environments creates technical debt. Specific to the crunchtools environment, CentOS and RHEL have different build, test, and release mechanisms. Each distribution supports and manages individual Bug Trackers and completely different support mechanisms. Architects and business analysts should consider these variables when calculating ROI.

1. Accessed March 26th, 2013 10:47PM EST
2. http://news.cnet.com/8301-13505_3-10312978-16.html

Background

In my ever evolving lab, it came time to integrate Red Hat Enterprise Linux (RHEV) with Identity, Policy, Audit (IPA).There were a few caveats and searching Google didn’t help, so hopefully this article can save you some time.

Integrating the two was fairly straitforward. The biggest challenge was finding a quick and easy way to provide reverse DNS. I use an external DNS provider which does not do reverse DNS for internal IP addresses. This article will dig into how to setup a full blown proof of concept without needing an external DNS server to provide reverse DNS.

For those that might need a brief primer on DNS records, the following article is excellent.

The following items are assumed to be available and working.

• Clean Red Hat Enterprise Linux 6.3+ Server
• Working forward and reverse DNS (both can be satisfied with dnsmasq, described below)
• Fully installed and working Red Hat Enterprise Virtualization (RHEV) environment

• Install IPA on Clean Red Hat Enterprise Linux 6.3+ Server
• Option: Install/Configure Dnsmasq on RHEV Server
• Configure RHEV Domain

Option: if an external DNS server is available, the following SRV entries should be added. For those unfamiliar with SRV records, this will allow services such as RHEV or the IPA client to discover the correct Kerberos/LDAP/NTP servers by searching DNS.

Option: If host based firewall rules are required, the following entries should be added to: /etc/sysconfig/iptables

Install packages

Configure

Restart ssh

Reverse DNS lookup of the IPA server is a strict requirement. rDNS can be provided by Dnsmasq, which is available from the base RHEL channel.

Dnsmasq is a lightweight, easy to configure DNS forwarder and DHCP server.

If there is no access to a reverse DNS server, dnsmasq can be installed and configured on the RHEV-M host.

Add the following line to /etc/dnsmasq.conf

Configure the upstream DNS server in /etc/dnsmasq-resolve.conf

Configure the standard DNS resolver to look to local host. This will chain all lookups through the local dnsmasq daemon.

Option: Forward DNS SRV records can also be satisfied with dnsmasq

Configure the new domain

Finally, here are some techniques used to troubleshoot integrating RHEV and IP.

The RHEV-M logs do not display much information with regard to a reverse DNS issue.

The logs for the Kerberos domain will show nothing because the RHEV-M host isn’t resolving the correct controller.

Check the SRV records

Watch DNS and Kerberos activity

Abstract

Esta presentacion fue presentado en CampusLink 2.0. Fue la primera vez que yo he hablado profesionalmente en Español. Disfruté esta experiencia demasiado y espero tener la oportunidad hacerlo de nueva en años futuros.

El Caso de Red Hat: Un Negocio Exitoso Basado en Software Libre

Background

From the organizers:

CampusLink 2.0 es un foro para el contacto directo entre estudiantes, maestros, profesionistas y expertos; inmersos en el mundo de las tecnologías de información.

Translation: CampusLink 2.0 is a forum for direct contact between students, teachers, professionals, and experts; Immersion in the world of information technology.

This is one of the most wonderful events I have ever attended. It was great to spend time with all of the students at the University of Chihuahua.

Arquitecto de soluciones en Red Hat – Fedora. Amplia experiencia en operaciones, análisis de ingeniería en entornos de red. Ha trabajado para compañías multinacionales, lo que le ha brinda un ámplio perfil de competencias en administración y operación de tecnologías. Experto en monitoreo de fallas, análisis e ingeniería de datos.

There were three all day tracks at this event

Background

From the organizers:

The tenth annual Ohio Linux Fest will be held on September 28-30, 2012 at the Greater Columbus Convention Center in downtown Columbus, Ohio. Hosting authoritative speakers and a large expo, the Ohio LinuxFest welcomes all Free and Open Source Software professionals, enthusiasts, and everyone interested in learning more about Free and Open Source Software.

This year, I spent my birthday down at Ohio Linux Fest at the Columbus Convention Center.

I had a great time speaking with peers from Rack Space and HP.

Room Opens 6:30pm, Dinner 7-8pm, presentation 8-9pm

Abstract

This presentation walks through some of the software development processes used to develop Shiva, a mass ssh client. The development of Shiva was used as a teaching aid for a class taught at Ohio Linux Fest 2012. Shiva was used to demonstrate concepts such as good command line interface design, testing, configuration design and advanced features of Bash, such as background functions. This presentation and associated class were developed to help systems adminsitrators think more like developers.

bash_devops

Background

This weekend I decided to upgrade my home network with a Cisco WRVS4400N wireless router. Like a typical router it can provide standard wireless services WPA2, DHCP, etc, but the this model also provides support for four distinct VLANs and four distinct SSIDs. This has allowed me to create separate networks for work, play, and a renter who resides in an upstairs apartment.

The goal of this network setup was to have three distinct SSIDs connected to three different VLANs. This would allow me to separate traffic for work, play, and a renter. Here are the main objectives:

1. Create three VLANs mapped to three different SSIDs
2. Allow Cisco router to provide DHCP service on VLAN 1 (virbr0)
3. Allow RHEL6 Server to provide DHCP service on VLAN 2 (virbr1)
4. Allow Cisco router to provide DHCP service on VLAN 3. VLAN 3 is a non-standard VLAN only at DC4 which provides Internet access to the rental unit.

The goal was to have the RHEL6 provide the exact same IP addresses that are provided on VLAN 2 because it is bridged to virbr2 which is attached to all virtual machines created on the RHEL6 server and RHEV Cluster that is housed at DC4. This allows all new virtual machines and physical machines to communicate with all of the crunchtools.com network

There is a caveat with regard to DHCP. According to the this post, the Cisco WRVS4400N cannot pass the RHEL 6 provided DHCP to the wireless network because of some missmatch with the wireless and switched network modules. This means that when a laptop is connected to VLAN 2, it must be assigned a manual IP address.

Background

Recently, I discovered how to use the openssl provided CA script to create a certificate authority and self signed certificates. Traditionally, I had ran all fo the commands manually. When using the CA script it is critical to understand the underlying security concepts.

Openssl has infrstructure to create a long lived Certificate Authority

Certificates signed are tracked. Index file is database for certs in ”’newcerts”’

Scripts which come with the openssl package on RHEL can be used to build a certificate authority, complete signing requests, signings, verification

Setup openssl.conf file

Defaults

Customizations

Create the certificate authority. Remember to put in a Common Name, or the Certificate Authority will not be built correctly and will be unusable.

Create the signing request, then sign it with your CA. If you generate a new certificate with the CA script, it will not be signed at all. This method keeps track of all certificate in the CA directory. This tracking mechanism allows a certificate to be revoked should it become compromised during it’s lifetime.

Then sign it

Verifiy it

There are times when an administrator would like to generate a key that is not encrypted. For example this will allow the key to start in apache, postfix, dovecot, or vsftpd. This can be done by modifying CA.

Change from:

This tells openssl not to encrypt the generated private key. This option stands for no DES.

These commands can be much easier to memorize and use than raw openss with all of it’s options. This method also tracks certs and enables a sane method for tracking active and revoked certs.