Background

Recently, I started a project at (www.eyemg.com) to migrate from VMWare to KVM. Our standard server deployment is based on RHEL5 running on HP DL380 hardware. Given our hardware/software deployment, it made sense to align ourselves with Red Hat’s offering of KVM. We are able to achieve feature parity with VMware server while adding live migration. At the time of this writing, live migration was not available on VMware Server and required an upgrade to ESX server with vMotion. KVM on RHEL5 provides these features at a price point that is much lower than VMware ESX Server.

Our hardware/software stack is well tested and generally very stable. This makes it an excellent base upon which to build a shared nothing HA cluster system. The goal of this system is to have complete hardware and software redundancy for the underlying virtual machine host servers.

To use live migration, both nodes of an HA cluster must have access to a live copy of the same virtual machine data files. This is traditionally done with a SAN, but we use GFS2 over DRBD to gain the advantages of having no shared hardware/software at a lower cost. We implement RAID1 over Ethernet using a Primary/Primary configuration with DRBD protocol C.

I wrote this article because there is a vacuum of documentation on the Internet with solid implementation details around KVM in a production environment. The down side of this project is that it requires extensive knowledge and expertise in the use of several pieces of software. Also, this software requires some manual installation and complex configuration. Depending on your level of experience, this may be a daunting task.

My company (www.eyemg.com) offers KVM/DRBD/GFS2 as a hosted solution. VMware’s offering has the advantage of assisting with installation of many of these features, with the disadvantage of lacking a shared nothing architecture.

---

Architecture

The hardware/software architecture is described in the drawing below

Hardware Inventory

The hardware is very standard and very robust. HP servers allow monitoring of power supply/CPU temperature, power supply failure and hard drive failures. They also come equipped with a remote access card called an Integrated Lights Out (ILO). This ILO allows power control of the server, which is useful for fencing during clustering.

• (2) DL380 G6
• (16) 146GB SAS Drives 6GBps
• (2) Integrated Lights Out Cards (Built In to DL380)
• 12GB RAM, Upgradable
• 900GB Usable Space, Not easily upgradable because all of the drive slots are filled

Software Inventory

Using GFS2 on DRBD we are able to provision pairs of servers which are capable of housing 30 to 40 virtual guests while providing complete physical redundancy. This solution is by no means simple and requires extensive knowledge of Linux, Redhat Cluster, and DRBD.

• DRBD: Distributed Replicated Block Device^[1]
• GFS2: Redhat global file system. Allows 1 to 16 nodes access the same file system.^[2]
• CMAN: Redhat cluster manager

Network Inventory

Two special interfaces are used to achieve functional parity with VMware Server/ESX. This allows the clustered pairs of KVM host servers access to different VLANS while communicating over bonded crossover cables for DRBD synchronization.

• Bridge: Interface used to connect to multiple VLANS. There is a virtual bridge for each VLAN.
• Bond: Three 1GB ethernet cards and three crossover cables work as one to provide a fast reliable backend network for DRBD synchronization

Installation

Operating System Storage

First thing is to configure each machine with one large RAID5 volume. This can be done from the smart start CD^[3] or from within the BIOS at boot time by hitting F9. Once the operating system is installed the configuration can be checked with the following command.

Operating System

Currently, the version of the operating system must be X86_64 and RHEL 5.4 or above. I will not detail RHEL5 installation as there is extensive documentation. We us a kickstart installation which is burned to CD using a tool written in house which will eventually be open sourced. Remember to leave enough space for your operating system, we use 36GB on our installations. The rest of the volume will be used for virtual machine disks.

Data Storage

Now that the operating system is installed, the next step, is to create a partition for the virtual machine disk files. To get better performance from our virtual machines, this tutorial shows how to do boundary alignment per the recommendation of EMC^[4] and VMware^[5]. To achieve the most performance from this stack, the boundary alignment should really be done for guest OSes too. Currently, this cannot be achieved from kickstart.

First partition, then reboot, and align the new partiion.

Create a partition that looks like the following

Switch to advanced mode:

Then hit ‘p’, you should see output that looks like the following. Use the start column in this advanced output to calculate your boundary alignment.

Command (m for help): x

Realign the beginning

Use the following formula to get the correct starting block:

For example, if your partition currently begins on block 19682411:

Round up

Then multiply it by 128

You should end up with partitioning that looks like the following

Bonded Interfaces

This is used to bond three 1 Gbps interfaces together to be used with DRBD

Then configure each of eth1, eth2, eth3 to use the bond0 interface

Bridge Interfaces (VLAN)

Install bridge utilites

Create a bridge interface. Take the IP address from eth0 and move it to the new bridge interface

Remove the servers main ip address from eth0 and change the eth0 interface to bind to a bridge

Configure iptables to allow traffic accross the bridge

Restart services

DRBD

DRBD installation is straitforward and well documented here. Once installation is complete, use the following configuration file as a template. The most important parts to notice are the 256MB rate limit on sync and the interface that it uses. This will prevent DRBD from using all of the bandwidth on the bonded interface.

Start the DRBD service. The default installation comes with init script (see DRBD documentation for details)

Once DRBD is started, it will begin it’s initial sync. You should see something like the following.

Then run a status to make sure both nodes are working correctly.

Make both nodes primary by running the following command on each

If you have trouble getting DRBD working, use the following to troubleshoot. Be patient, sometimes it takes a while to get the hang of it.

Troubleshooting Guide

GFS2

To use GFS, a Redhat Cluster must be in place and active. The manual installation of Redhat Cluster is beyond the scope of this tutorial given the time and length I have permitted, but the Redhat documentation should get you started if you are patient and persistent.

Eventually, I will replace this link with a guide for an open source tool that I will be releasing called fortitude. Fortitude installs Redhat Cluster, provides net-snmp configuration, Nagios check scripts, and easy to use init scripts which really streamline the Redhat Cluster build/deployment process, but for now, please follow the Redhat documentation. Fortitude is written and works in our environment but needs some love and documentation before it can be release open source. Until then, I leave you with the Redhat Documentation.

• Redhat Cluster Overview
• Redhat Cluster Configuration Guide

First install GFS tools

Then create GFS filesystem

There are few tuning parameters that are critical for GFS2 to perform well. First, the file system must be configured to disable writing the access time on files and directories. modify /etc/fstab to look similar to the following.

Finally, add the following to the cluster.conf XML file anywhere in the section. By default GFS2 limits the number of plocks per second to 100. This configuration, removes the limits for dlm and gfs_controld.

Update the cluster.conf version number and push the changes to the cluster. After pushing the file, each node can be restarted individually without bringing down the entire cluster. The changes will be picked up by each node individually (tested with Ping Pong).

KSM

A newer feature that can be used with KVM is called KSM (Kernel Shared Memory Pages)^[6]. KSM allows KVM hosts to consolidate memory pages which are identical between guest virtual machines. On our HA pair running 15 RHEL5 virtual machines, we recouped 2GB of physical memory.

Below is a graph demonstrating memory usage when we implemented KSM. Notice that the memory usage dropped off by about 2 GB, which the Linux Kernel then began to slowly use for caching and buffers.

Tuning KSM is beyond the scope of this tutorial but the following script, written by Red Hat, can be used to assist in controlling how aggressively the Kernel will try and merge memory pages.^[7]

Add to default start up

Final Notes

In our environment, we have decided to manually start cluster services and mount the DRBD partition, but in a different environment it may be useful to have a startup script that will automatically mount/unmount the drbd volume. The unmount is especially useful if an administrator forgets to unmount before doing a reboot. The following script can be used as an example. Remember it should unmount very early in the reboot process, before cluster services stop or there will be problems.

Testing & Performance

• Ping Pong: Small C program used to check plocks/sec performance in GFS2
• Bonnie++: All disk I/O was generated with bonnie++. Bonnie++ also provides results on benchmarking
• iostat: Disk I/O was also measured with iostat during the bonnie++ testing to confirm results
• sar: Network traffic was monitored with sar to verify the network I/O was coherent with the disk I/O data

Analysis

When generating these performance numbers there is a desire to run the tests for months and months to get everything just perfect, but at some point, you must say that enough is enough. I ran these tests over about 3 days and eventually came to the conclusion that the performance was good enough given the price point. Since the main I/O performed by a virtual machine host is on large virtual disk files, I paid special attention to sequential read/write times in bonnie++ and these numbers seemed acceptable to me.

Disk I/O

GFS2

To verify that all of the GFS2 setting are correct use the following small test program called Ping Pong. Before the above changes, the maximum number of locks displayed by Ping Pong was about 95/sec. After the above changes, performance should be similar to the following Ping Pong output.

Bonnie++

The following command was used to benchmark disk I/O.

Results from Bonnie++ showed that we could achieve sequential reads of almost 300MB/s and writes of almost 90MB/s. This is with GFS over DRBD connected thorugh 3 bonded 1Gb ethernet interfaces.

iostat

The following command was used to measure disk I/O. A 10 second interval was used to smooth out peaks/valleys.

IOstat confirms bursts of over 200MB/s for 10 second periods

ifstat

The following command was used to measure network I/O. A 10 second interval was used to smooth out peaks/valleys. Sar provides a wealth of information, but we are using it for it’s network I/O capabilities.

IFstat verifies bursts in network I/O confirming that all three ethernet devices are, indeed, being used.

1. http://www.drbd.org/home/what-is-drbd/ [↩]
2. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5.4/html/Logical_Volume_Manager_Administration/index.html [↩]
3. http://h18013.www1.hp.com/products/servers/management/smartstart/index.html [↩]
4. http://media.netapp.com/documents/tr-3747.pdf [↩]
5. http://www.vmware.com/pdf/esx3_partition_align.pdf [↩]
6. http://www.ibm.com/developerworks/linux/library/l-kernel-shared-memory/index.html [↩]
7. http://www.redhat.com/archives/fedora-virt/2009-September/msg00023.html [↩]