Getting Ninja with Runc

Background

Have you ever tried to get runc to work? Did you have that WTF moment where you were like, this is weird, and annoying, and why do I even need to know this? When, I use docker, everything just works. Well, I am here to help. Why might you want to get ninja with Runc? Well, like any good hacker, because it helps you understand how all of the pieces fit together in this very fast moving space. This will open your mind to how this ecosystem is forming and why it is awesome.

Let’s dig right in – I am  going to assume you have runc installed. On a Red Hat based system, it’s pretty easy to get (same for most other distributions) – just install the runc package. While your at it, install the yajl package (or your favorite tool) as well to manipulate json more easily.

runc -v
runc version 1.0.0-rc2
spec: 1.0.0-rc2-dev
From the runc page on GitHub, you will find some terse documentation stating that you need to create an OCI bundle, but let’s explain what this is. At the end of the day, the OCI bundle is two main things:

1. A root filesystem. This looks very similar to what you would see on any Linux system. Basically, this is really nothing more than rsyncing the root filesystem out of a running machine, removing some things like /dev, /proc, /sys, and some other things.
2. A config.json file. This is a file that you pass to runc, or any other OCI runtime compliant tool, that tells it what options to pass the Linux kernel when it fires up the container.

 

The Gap

You will hear the config.json file talked about a lot, but nobody ever really explains what it is or where it comes from. I’m here to help. Well, it can come from a lot of different places, but there are a few that are easy if you are just hacking around like me:

1. You can generate a spec file manually. Nix this, too hard.
2. You can write a program to generate a file. Nix this, even harder.
3. You can generate a very basic config.json with the runc tool. Kinda cool, not that interesting.
4. You can steal an advanced config.json file from a running docker daemon. Very cool because it has a lot more options buried in it which explains a LOT about how things work (this is where I hurt your brain).

 

Let’s Hack

First, let’s take a look at an OCI image. We are pulling down a fresh copy of a container image. Then, we are exporting it to the filesystem to inspect it.

mkdir rhel7
docker pull rhel7
docker save rhel7 -o rhel7/rhel7.tar
tar xvf rhel7/rhel7.tar -C rhel7/
Now, let’s take a look at the configuration file that comes with the image. Notice that there are some variables that look strikingly familiar if you have used docker inspect before. Variables like Volumes, Entrypoint, Labels, Vendor, etc. Regretfully, this configuration file doesn’t provide what we need to fire up runc. More on this later.

cat rhel7/[a-z0-9]*.json
{"architecture":"amd64","comment":"Created by Image Factory","config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["container=docker","PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":["/bin/bash"],"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":{"Architecture":"x86_64","Authoritative_Registry":"registry.access.redhat.com","BZComponent":"rhel-server-docker","Build_Host":"rcm-img01.build.eng.bos.redhat.com","Name":"rhel7/rhel","Release":"84","Vendor":"Red Hat, Inc.","Version":"7.2"}},"container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"created":"2016-07-27T16:19:50Z","docker_version":"1.7.0","history":[{"created":"2016-07-27T16:19:50Z","comment":"Created by Image Factory"}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:301cd0a2eb7acd8b7194b9fcf47bac26bfa38ccbbbbc57299eaa8056241aeaf4"]}}[{"Config":"4a6b6e1a17d70b7f67787aaee800c1fdb4b145dd3f7ae48959f5d41286eadb0b.json","RepoTags":["rhel7:latest"],"Layers":["2d3ec909ab90228062a6aa2df641dae7dedc5a63d057b5ccabefb5b7c2779605/layer.tar"]}]
Now, let’s fire up a docker container to steal what we need. Plus, we probably already understand what is going on with a basic docker run command:

mkdir ./rootfs
docker run rhel7 bash
In another terminal, export the root file system and steal a sophisticated spec file to play with. Luckily, since docker 1.11, the docker daemon generates a config.json file for us and places it in a convenient directory:

docker export rhel7-rootfs | tar -C ./rootfs -xvf -
cat /run/docker/libcontainerd/`docker ps -a --no-trunc | grep rhel7-rootfs | awk '{print $1}'`/config.json | json_reformat > ./config.json
Now, for it to be useable by runc, we have to edit the config.json to remove some stuff that is specific to the container that was running:

vi config.json
Change the rootfs section. Remove the big long directory and change it to the local directory so that runc will know where to find the rootfs.
"root": {
"path": "./rootfs"
},
Remove the following entries from the mounts section. These are specific to the docker daemon. This will allow runc to run the container:

{
"destination": "/etc/hostname",
"type": "bind",
"source": "/var/lib/docker/containers/7c6f58d8c8376d29c94e453f998c0f5b224ea009a72bac24201c4e56a1babe4a/hostname",
"options": [
"rbind",
"rprivate"
]
},
{
"destination": "/etc/hosts",
"type": "bind",
"source": "/var/lib/docker/containers/7c6f58d8c8376d29c94e453f998c0f5b224ea009a72bac24201c4e56a1babe4a/hosts",
"options": [
"rbind",
"rprivate"
]
},
{
"destination": "/dev/shm",
"type": "bind",
"source": "/var/lib/docker/containers/7c6f58d8c8376d29c94e453f998c0f5b224ea009a72bac24201c4e56a1babe4a/shm",
"options": [
"rbind",
"rprivate"
]
},
{
"destination": "/run/secrets",
"type": "bind",
"source": "/var/lib/docker/containers/7c6f58d8c8376d29c94e453f998c0f5b224ea009a72bac24201c4e56a1babe4a/secrets",
"options": [
"rbind"
]
}

Note: don’t forget to get rid of the comma at the end. The mounts section should end like this:
...
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
}
],
"hooks": {
...
OK, now let’s use runc to start our newly minted container with all of it’s fancy options. OH, BTW, Red Hat systems are not setup with an SELinux policy that handles fancy runc containers, so Dan Walsh, please don’t kill me, but for this test, disable it temporarily:

setenforce 0
[root@rhel7 ~]# runc run rhel7-runc
[root@7c6f58d8c837 /]#
Voilá, we have used runc to fire up a fancy container with a lot of cool options specified in the config.json file.

Analysis

Let’s do some final analysis to tie this all together and make sure you get why we are hacking around with this. First, notice that there are three main types of options specified in the config.json file.

1. Variables that come from the image: These come from the container image itself. These are specified when the base image or image layer are built. They are dependent on the build tool that you are using. For example, variables like os and architecture variable is set to amd64 come from the image.
2. Variables that come from the container engine: These variables are generated at runtime. They can be hard coded into the container engine or specified by configuration options. This is dependent on the container engine’s implementation. For example, all of the rules in the seccomp section come from the docker engine which in turn come from profile rules which the docker daemon has implemented. While other engines may implement similar constructs, these can be different per engine.
3. Variables that come from the user: These are the options that the user specifies on the command line. The runc tool, doesn’t accept these, but the docker sli does. The docker cli converts these, and constructs the config.json that we stole in the examples above. Notice the bash variable embedded in the args section. This came from us.

 

Basically, all three of these sources get smooshed together to create a the config.json. This is cool because it’s flexible and provides for a way to input variables from images, engines, and users. Other tools like runv, Railcar also expect this pre-formatted config.json and a root filesystem. Whether they run the process in a container or a VM, the requirements are the same. As always, if you have questions, leave them below and I will answer them. Until next time.