Background

As part of my blog series on Running Docker in Production, I have been writing about Docker security. Coincidentally, the CIS Docker 1.6 Benchmark v1.0.0 was released at the end of May and I thought it would be fun to tackle this new security benchmark.

The CIS Docker Benchmark is meant to be a practical guide for securing Docker in production. Below I will analyze the benchmark, share my thoughts on some of the recommendations, and show you how to meet some these recommendations with RHEL and RHEL Atomic.

 

Analysis

The CIS Docker Benchmark contains six sections and a total of 84 recommendations. The recommendations are also broken into two levels, with the higher level being more secure but possibly impacting convenience. Currently, only SELinux, AppArmor, and End Point Protection (encryption, firewall, virus, etc) are listed as level two, but it is highly recommend to leave SELinux/sVirt enabled in production. The CIS Benchmark recommendations break down as follows:

• Host Configuration: 18 Recommendations
• Docker daemon configuration: 10 Recommendations
• Docker daemon configuration files: 26 Recommendations
• Container Images and Build File: 4 Recommendations
• Container Runtime: 19 Recommendations
• Docker Security Operations: 7 Recommendations

 

Meeting the Requirements

Going through every single recommendation is beyond the scope of this document, but the goal is to go through some highlights of meeting this benchmark with RHEL 7 and RHEL 7 Atomic.

RHEL 7

RHEL 7 has advantages and disadvantages over RHEL Atomic. Currently, in RHEL 7, it is easier to run Auditd, but has the disadvantage of using standard RPM packages which do not provide atomic updates and rollbacks.

Below are a couple of interesting sections to investigate with RHEL 7

Create a separate partition for containers (Section 1.1)

The first interesting thing that comes to mind, is moving to a separate partition. In the RHEL7/RHEL7 Atomic world, it is generally recommended to use a separate LVM Thin Pool because using the standard loop back device incurs a small performance penalty. Currently, this test will always fail because it does a simple test in the check in the file system (which doesn’t exist). I have filed an issue in GitHub to think about other ways to check for this.

Here is an example of how to move Docker storage to a separate LVM Thin Pool on RHEL7:

systemctl stop docker
rm -rf /var/lib/docker/
pvcreate /dev/vdb
vgextend /dev/vdb rhel
lvcreate -L 15G --type thin-pool --thinpool docker-pool rhel
systemctl start docker
 

Auditd rules (Sections 1.8 – 1.18)

The following auditd rules are recommended by the CIS Benchmark. Some of them do not apply to RHEL 7 and others are only applicable if you a registry server is being run. For convenience, and to simplify configuration management, all of the rules have been included here. There is really no problem in using all of these rules on all systems, that way if a registry server was ever later configured, the auditd rules would already be in place.

vi /etc/audit/rules.d/audit.rules
-w /usr/bin/docker -k docker
-w /var/lib/docker -k docker
-w /etc/docker -k docker
-w /usr/lib/systemd/system/docker-registry.service -k docker
-w /usr/lib/systemd/system/docker.service -k docker
-w /var/run/docker.sock -k docker
-w /etc/sysconfig/docker -k docker
-w /etc/sysconfig/docker-network -k docker
-w /etc/sysconfig/docker-registry -k docker
-w /etc/sysconfig/docker-storage -k docker
-w /etc/default/docker -k docker
 

Verify SELinux security options (Section 5.2)

This recommendation really merges two different concepts sVirt and separate Types. By default, sVirt is enabled on in RHEL7, which is not tested in this Benchmark. The Benchmark is actually testing for the use of custom types as described by Dan Walsh here. Also, this article explores more possible uses of custom types. It is interesting that the CIS Benchmark is testing for this even though no custom types exist yet…

Run the Check Container

Pull the Docker Benchmark code and run it to check out your environment. I built custom Dockerfiles for use with RHEL and CentOS which have a few tweaks to make them accurately check the audit rules and some other tests.

docker run -it --privileged --net host --pid host \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/lib/systemd:/usr/lib/systemd \
-v /etc:/etc \
--label docker-bench-security \
fatherlinux/docker-bench-security
 

Output:
...
[PASS] 1.2 - Use an updated Linux Kernel
[PASS] 1.5 - Remove all non-essential services from the host - Network
./helper_lib.sh: line 19: [: : integer expression expected
./helper_lib.sh: line 20: [: : integer expression expected
1.6.0
[PASS] 1.6 - Keep Docker up to date
[INFO] 1.7 - Only allow trusted users to control Docker daemon
[INFO] * dockerroot:x:996:
[PASS] 1.8 - Audit docker daemon
[PASS] 1.9 - Audit Docker files and directories - /var/lib/docker
[PASS] 1.10 - Audit Docker files and directories - /etc/docker
[PASS] 1.11 - Audit Docker files and directories - docker-registry.service
[PASS] 1.12 - Audit Docker files and directories - docker.service
[PASS] 1.13 - Audit Docker files and directories - /var/run/docker.sock
[PASS] 1.14 - Audit Docker files and directories - /etc/sysconfig/docker
[PASS] 1.15 - Audit Docker files and directories - /etc/sysconfig/docker-network
[PASS] 1.16 - Audit Docker files and directories - /etc/sysconfig/docker-registry
[PASS] 1.17 - Audit Docker files and directories - /etc/sysconfig/docker-storage
[PASS] 1.18 - Audit Docker files and directories - /etc/default/docker
 

Also, you can pull the code locally and run it if you run into trouble

git clone https://github.com/diogomonica/docker-bench-security.git
cd docker-bench-security/
bash docker-bench-security.sh
 

RHEL 7 Atomic

Create a separate partition for containers (Section 1.1)

By default, in RHEL 7 Atomic, Docker storage is placed on a separate LVM Thin Pool. This is not tested in the CIS Benchmark, but nonetheless, containers do not live on the root filesystem. For further information on how to expand the Docker pool, check out the storage guide.

Also, notice that the auditd recommendations specify a rule that would audit file activity in /var/lib/docker. This is an interesting way to monitor, but I would really like to see this built into the Docker daemon since RHEL/Fedora/CentOS use LVM Thin Pools which means there is no file system to monitor.

 

Auditd rules (Sections 1.8 – 1.18)

Notice that sections 1.8 through sections 1.18 fail because there is no auditd support in RHEL 7 Atomic at this time

 

Verify SELinux security options (Section 5.2)

By default SELinux and sVirt are enabled in RHEL 7 Atomic, though this benchmark does not test for this. Per the section in RHEL 7, custom types are coming soon, which will provide even greater protection between containers.

 

Run the Check Container

Pull the Docker Benchmark code and run it to check out your environment

docker run -it --privileged --net host --pid host \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /usr/lib/systemd:/usr/lib/systemd \
-v /etc:/etc \
--label docker-bench-security \
fatherlinux/docker-bench-security
 

[INFO] 1 - Host Configuration
[WARN] 1.1 - Create a separate partition for containers
[PASS] 1.2 - Use an updated Linux Kernel
[WARN] 1.5 - Remove all non-essential services from the host - Network
[WARN] * Host listening on: 9 ports
[WARN] 1.6 - Keep Docker up to date
[INFO] 1.7 - Only allow trusted users to control Docker daemon
[WARN] 1.8 - Audit docker daemon
[WARN] 1.9 - Audit Docker files and directories - /var/lib/docker
[WARN] 1.10 - Audit Docker files and directories - /etc/docker
[WARN] 1.11 - Audit Docker files and directories - docker-registry.service
[WARN] 1.12 - Audit Docker files and directories - docker.service
[WARN] 1.13 - Audit Docker files and directories - /var/run/docker.sock
[WARN] 1.14 - Audit Docker files and directories - /etc/sysconfig/docker
[WARN] 1.15 - Audit Docker files and directories - /etc/sysconfig/docker-network
[WARN] 1.16 - Audit Docker files and directories - /etc/sysconfig/docker-registry
[WARN] 1.17 - Audit Docker files and directories - /etc/sysconfig/docker-storage
[WARN] 1.18 - Audit Docker files and directories - /etc/default/docker
 

Conclusions

I think that the CIS Docker Benchmark captures a lot of good data, but I think we have a long way to go until we understand what production workloads with Docker/Linux Containers will look like in the long run. Many of these recommendations can be met with any operating system, but I think RHEL really makes it easier to go above and beyond with SELinux and there is more to come.

At this point, I think the picture is becoming clearer and clearer that mode 2, and certainer mode 1 workloads can be ran in production with Docker/Linux Containers. There are a lot of benefits, but security is top of mind for many….