Markdown Content - The Systems Administrator’s Lab

---
# The Systems Administrator's Lab

**URL:** https://crunchtools.com/the-systems-administrators-lab/
Date: 2010-10-15
Author: fatherlinux
Post Type: post
Summary: Recently, I listened to an O'Reilly webcast called "The Myths of Innovation" where Scott Berkun discussed the concept of a lab. He showed a picture of Edison's lab which showed wooden tables, lamps, and beakers. Systems administrators are also inventors.  We are required to script, program, and configure exotic servers and equipment. To discover new solutions, we need a lab. This is especially true with cloud computing and virtual infrastructure where machines are created and destroyed in a very transient manner.  You need a lab to track all of the successful and failed experiments.Continue Reading "The Systems Administrator’s Lab" →
Categories: Articles
Tags: DevOps, Systems Administration
---

## Background

Recently, I listened to an O'Reilly webcast called "[The Myths of Innovation](http://www.youtube.com/watch?v=We5CfGzhxM8&utm_content=em-orm-Webcast+PR+-+The+Myths+of+Innovation:+Remixed+and+Remastered+Recording+Available&utm_campaign=Webcasts+PR&utm_source=iPost&utm_medium=email&imm_mid=064c95&cmp=em-orm-Webcast+PR+-+The+Myths+of+Innovation:+Remixed+and+Remastered+Recording+Available)" where [Scott Berkun](http://www.oreillynet.com/pub/au/2099) discussed the concept of a lab. He showed a picture of Edison's lab which showed wooden tables, lamps, and beakers. Systems administrators are also inventors.  We are required to script, program, and configure exotic servers and equipment. To discover new solutions, we need a lab. This is especially true with cloud computing and virtual infrastructure where machines are created and destroyed in a very transient manner.  You need a lab to track all of the successful and failed experiments.

Many people get caught up on which tool to use in the lab. It is more important to have and use a tool than which one you pick. If you are having trouble selecting a tool, then most likely it's just brands of beakers, just select one and move on. Don't spend too much time on the selection process, you can always change it later, which will take the same amount of time, but you will be more productive in the interim.

Anybody that has lab tools that they love, please leave them in the comments.

## Basics

 	- **Operations Development Box**: This is often forgotten, but the most important. You need a dedicated systems administrator's only development box. Often, each team of programmers has a dev box, but quite often whole teams of systems administrators do not have one server truly dedicated to doing systems administration research and development. A [gold server](http://www.infrastructures.org/bootstrap/gold.shtml) doesn't count, you need a box that is all yours. A VM host is even better, so you can experiment with new operating systems. It is preferable that this box is a newer server, in the data center and backed up, just like a programmers development box, but all for you. If you ask, I promise your boss will buy one. Don't skimp, your worth it, it will be one of the best investments in your team. The Operations Development Box has the advantage of access from all of the team members, including command history, and enables a new form of technical conversation.

 	- **A Ticket System**: Everyone says this. [Thomas Limoncelli](http://everythingsysadmin.com/) says this in the introduction to his [book](http://www.amazon.com/Practice-System-Network-Administration-Second/dp/0321492668/ref=sr_1_1?ie=UTF8&qid=1287062013&sr=8-1). I prefer [Request Tracker](http://bestpractical.com/rt/), but anything will do. Doing side work, I have even used Bugzilla. Just get it in place first and track all of your work there. It gives you a sense of "completeness" for each task and helps you move along. When you get done building/installing the ticket system, create a ticket for it and mark it completed.

 	- **A Wiki**: A wiki is the next most important part of the lab. You need a place to track your work. If you have never used a Wiki, it will take a while to get comfortable with this style of always accessible, version controlled documentation, but just experiment. At first put everything you can think of in it. Get a rough draft, go back and edit. When I am documentation a process, it usually takes two or three iterations to get it really nail it.

 	- **Fault Monitoring**: This is critical, so that you know when you are breaking something, even if it is an experimental machine. Have a fault monitoring system in place and use it to monitor the operations box, ticket system, and wiki. I use [Nagios](http://www.nagios.org/) because it has been around forever and most likely will be. I don't need the latest greatest because I do that in my lab.

 	- **Data Acquisition**:Always capture historic data and capture as much as you can. I use [Cacti](http://www.cacti.net/) to capture ping, port, MySQL insert/update/delete stats, apache thread stats, etc, etc, etc Capture anything. I use a central [Syslogng](http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/overview) box for all of our logs. It is critical to have this stuff in place for analyzing what happened after the fact.

 	- **Version Control**: You must learn the magic of this to develop long living scripts, plugins, configuration files, etc. This helps so much when collaborating with other systems administrators in your team, but it is critical when releasing tools to the internet and collaborating with others. I generally use [SVN](http://crunchtools.com/software/chev-check-vulnerabilities-script/) internally (legacy and it works), but have been know to use [Mercurial](http://mercurial.selenic.com/), and [GIt](http://git-scm.com/) as necessary

 

## Lab Guidelines

Use a simple syntax for everything, you don't want it to get in your way. Don't be weighed down by what you should do, just develop it as you can. It will make your life less stressful. Don't spend thirty hours evaluating each piece of the lab. That is what your lab will help you do more efficiently.

Skill, at using these all together will grow over time, making you more efficient at building out your lab. Data acquisition and fault monitoring will always grow, but try and get a basic syntax for the ticket system and wiki down early.

### Ticket System

In the ticket system I use **three** simple headings, **Info** (optional for each ticket), **Completed**, and **Action Items**. Leading by example with a simple syntax can get everyone using the ticket system into the same format which makes training and usage of the system easier for everyone.

As I complete items, I always copy the entire list to the bottom of the ticket so that it is easier to get a sense of where things are at without searching the whole ticket, which would be no better than a giant convoluted email. Also, always keep your action items on the bottom so that they are easier to see.
```
Info
* item 1
* item 2

Completed
* item 1
* item 2

Action Items
* item 1
* item 2
```

### Wiki

In my wikis for standard knowledge base items, I use a simple template with five headings
```
= Background =
= Architecture =
= Routine Operations =
= Special Operations =
= Installation Notes =
```

### Thinking Ahead

Once you have the main pieces in place grow it organically.

Our operations box, ticket system, wiki, monitoring and data acquisition have grown to all have redundancy. Our MySQL DB for the ticket system and wiki are even replicated with a Master/Master and failover DNS.

 

## Experiment

Go forth and build something young Edison (I prefer Tesla), but remember you must be able to do experiments to really find creative solutions. We are scientists and to get experimentation done right you need tools in place.

Here are some of the projects I have developed in my lab.

 	Log Analysis Program

 	- Backup Scripts (very complex and interact with Bacula)

 	- Scriptlog: Data acquisition sensor for bash scripts (similar to log4sh, but with nagios plugin)

 	- [Chev](http://crunchtools.com/software/chev-check-vulnerabilities-script/): Vulnerability rss feed analyzer

 	- Red Hat Build tool: Kind of like cobbler but from DVD/CD (eventually open source)

 	- Red Hat Standard Installer: Configuration script which gets us to a baseline with install/uninstall for approximately 30 modules from ntp, to mysql and apache. (eventually open source too)

 	- Cacti plugins

 	- Nagios plugins

 	- Squirrel Mail password change plugin

 	- Flat file postfix system with IMAP, POP3, Squirel Mail, Vacation, Password change, and version control on all configs

 	- Many, many, many scripts (more than a hundred and more then 20K lines of code)

 	- Evaluation of New Software (for myr lab too)

---

## Categories

- Articles

---

## Navigation

- [Home](https://crunchtools.com/)
- [Articles](https://crunchtools.com/category/articles/)
- [Events](https://crunchtools.com/category/events/)
- [News](https://crunchtools.com/category/news/)
- [Presentations](https://crunchtools.com/category/presentations/)
- [Software](https://crunchtools.com/software/)
- [Beaver Backup](https://crunchtools.com/software/beaver-backup/)
- [Check BGP Neighbors](https://crunchtools.com/software/check-bgp-neighbors-nagios/)
- [Chev](https://crunchtools.com/software/chev-check-vulnerabilities-script/)
- [Graph BGP Neighbors](https://crunchtools.com/software/grpah-bgp-neighbors/)
- [Graph MySQL Stats](https://crunchtools.com/software/graph-mysql-stats/)
- [Graph Sockets Pipes Files](https://crunchtools.com/software/graph-sockets-pipes-files/)
- [MCP Servers](https://crunchtools.com/software/mcp-servers/)
- [Petit](https://crunchtools.com/software/petit/)
- [Racecar](https://crunchtools.com/software/racecar/)
- [Shiva](https://crunchtools.com/software/shiva/)
- [About](https://crunchtools.com/about/)
- [Home](https://crunchtools.com)

## Tags

- DevOps
- Systems Administration