The Systems Administrator’s Lab

Recently, I listened to an O'Reilly webcast called "The Myths of Innovation" where Scott Berkun discussed the concept of a lab. He showed a picture of Edison's lab which showed wooden tables, lamps, and beakers. Systems administrators are also inventors.  We are required to script, program, and configure exotic servers and equipment. To discover new solutions, we need a lab. This is especially true with cloud computing and virtual infrastructure where machines are created and destroyed in a very transient manner.  You need a lab to track all of the successful and failed experiments.

KVM Virtualization 101

Background For those of you who are familiar with virtualization, KVM is a hypervisor which is built into the Linux Kernel. This allows a Linux desktop or server to simulate multiple pieces of hardware, which can then have different operating systems installed. KVM uses the QEMU virtual machine format. QEMU may be familiar to those

Bootstrapping and Rooting Documentation: Part 2

Background In Bootstrapping and Rooting Documentation: Part 1, I laid out a blueprint for using documentation as the bootstrap for entry into an operations environment. In this article I will dig into the three main concepts mentioned in Part 1. In Part 3, I will demonstrate our use case for a data center of about

Bootstrapping and Rooting Documentation: Part 1

Bootstrapping and rooting are two concepts often associated with computing, but not the documentation of computing environments. Sometimes concepts such as these are called design patterns and I would like to try and introduce a new pattern called Documentation Bootstrapping. I think this is a critical pattern for the creation of any successful and useful documentation.

Unix/Linux Signals 101

Background Unix/Linux allows a user to have control over a program that they are running by sending what are called signals. These signals are then normally handled by the program in a way that is compliant with Unix/Linux standards. Two of the most important signals that are commonly sent to a program are called SIGTERM

Unix/Linux Filesystem Permissions 101

Background Standard Unix filesystem permissions are less complex than Windows file system permissions and Linux ACLs. Though, this lacks flexibility which is sometimes needed, In many cases it can be leveraged as an advantage. Often the complexity of ACLs can allow administrators to create file system permissions which are cumbersome to audit and document. The

KVM Cluster with DRBD/GFS

Background Recently, I started a project at (www.eyemg.com) to migrate from VMWare to KVM. Our standard server deployment is based on RHEL5 running on HP DL380 hardware. Given our hardware/software deployment, it made sense to align ourselves with Red Hat’s offering of KVM. We are able to achieve feature parity with VMware server while adding

Snort Alert Log: Simple Analysis and Daily Reporting with Arnold and Petit

Background This script was developed last year to give a quick and dirty analysis of the Snort alert log. In typical fashion, it’s is far from perfect, but approximately right is better than absolutely wrong. Obviously, the intersects could be combined in new and creative ways, this is just one that works for us. Also,

Centralized Logging System, Analysis, and Troubleshooting

Background Building a feature complete centralized logging system that provided the ability to trouble shoot problems and pro-actively find new issues before they became service outages was a top priority when I first started at (www.eyemg.com). I call it feature complete because it has successfully done both for us without spending too much time of

DevOps Toolchain: Problems with Automated Deployment, Data & Workflow

Background Automated deployment is obviously not new, but until this point, there was not much push in the open source world. Recently, the idea of DevOps or Ops 2.0 is gaining ground. We are starting to think of our deployment and provisioning methods more like software engineers. We are developing tools to help us provision