RHEL, Glusterfs, and the FUSE Client

This presentation overviews RHEL, Glusterfs, and the FUSE client. It was created for the Akron Linux User Group in May. There was a quick presentation followed by a demonstration using four virtual machines running RHEL 6.2. Three of the virtual machines were used as a Glusterfs storage pool using the open source RPMs. The fourth virtual machine was used to mount the storage pool using the FUSE client. I demonstrated the resilience of Glusterfs by pausing one of the Glusterfs storage pool nodes while the client was still connected. Since a distributed/replicated Glusterfs volume was used, the client paused for a short amount of time, then showed a listing of a directory with 1000 files in it. This was all done while the mount command on the client node showed the paused Glusterfs node being mounted, magical…

The Logs Are an Approximation of Reality

The logs are an approximation of reality and they cannot be taken as canonical or gospel. This is true in several senses. Logs can give insight to the standard investigative questions of who, what, when, where, and why, but almost always requires other information to truly answer all of these questions.

Today, Postfix reiterated this lesson for me. I had a problem where our gateway mail server couldn’t deliver mail to a peer. The receiving mail server kept bouncing the email address with a 550 even though the mailbox being delivered to was real and active. Gmail, Yahoo, and MSN would all accept email from our gateway, but this one provider would not accept email. Of course, it wasn’t a simple problem. We had a web server running Apache/PHP delivering to the local Sendmail server which forwarded to a Post fix gateway server, which then tried to deliver to an Exim server which received for the destination email address.

I am not going to dig into all of the details, but of course, the first thing I did was go to the logs. The problem is, the logs were wrong! In the following examples, the users and domains in the logs have been changed to be anonymous, but the logs are real.

Systems Administrator’s Lab: OpenSSH MaxStartups

Background When performing automation using OpenSSH/Cron you will inevitably run into concurrency problems. Recently, we had a problem where one machine was receiving 21 ssh connection within one second. This is because the standard cron daemon only has a granularity of one minute. In this article, I am going to quickly elaborate on how we

Log Analysis with Python

Abstract This presentation was created for the PyOhio 2010 conference. It gives an overview of Artificial Ignorance, command line graphing, and word counts with petit. It also gives a quick examples of weekly and monthly reporting when dealing with a syslog server which records switch, router, firewall, and server logs.   Presentation  

Snort Alert Log: Simple Analysis and Daily Reporting with Arnold and Petit

Background This script was developed last year to give a quick and dirty analysis of the Snort alert log. In typical fashion, it’s is far from perfect, but approximately right is better than absolutely wrong. Obviously, the intersects could be combined in new and creative ways, this is just one that works for us. Also,

Centralized Logging System, Analysis, and Troubleshooting

Background Building a feature complete centralized logging system that provided the ability to trouble shoot problems and pro-actively find new issues before they became service outages was a top priority when I first started at (www.eyemg.com). I call it feature complete because it has successfully done both for us without spending too much time of