Porting Linux to Nabla Containers
Tags: docker, linux, unikernel, anykernel
Published: 2020-04-18 08:20:05

This is an introduction of Linux Kernel Library ported to Nabla Containers.

[runnc](https://github.com/nabla-containers/runnc.git) is an OCI runtime that runs process-level isolated unikernels. It is built on the top of [Solo5](https://github.com/Solo5/solo5), a sandbox for unikernels, and several unikernels (MirageOS, IncludeOS, Rumprun) run on it. The original runnc uses Rumprun, a NetBSD based unikernel. However, as Docker is started from Linux, it is needed to have system call level compatibility with Linux. Therefore, I ported Linux Kernel Library (LKL) and musl libc to Solo5 and put together with runnc.

## frankenlibc on Solo5

frankenlibc is a set of tools to run Rump unikernels in various environments. It has a fork that ported LKL and some libraries. I used this frankenlibc fork and added Solo5 platform support.

* https://github.com/retrage/frankenlibc/tree/solo5

### Building frankenlibc

Clone the repository and checkout `solo5` branch.

```console
$ git clone https://github.com/retrage/frankenlibc.git
$ cd frankenlibc
$ git checkout solo5
```

Clone full Solo5 repository to avoid build failure and update submodules.

```console
$ git clone https://github.com/Solo5/solo5.git
$ git submodule update --init
```

Apply some patches.

```console
$ for file in `find patches/solo5/ -maxdepth 1 -type f` ; do patch -p1 < $file ; done
```

Finally, run the build script.

```console
$ ./build.sh -k linux notests solo5
```

You can find libraries and toolchain wrappers in `rump` directory after building successfully.

### Testing

Even if `notests` specified, `build.sh` builds simple tests to `rumpobj/tests`.

Create a `tap100` tap device.

```console
$ sudo ip tuntap add tap100 mode tap
$ sudo ip addr add 10.0.0.1/24 dev tap100
$ sudo ip link set dev tap100 up
```

Create `disk.img` disk image. As LKL/frankenlibc creates directories on initialization, some operations fail if read-only ISO image is used. To avoid this issue, we use the Ext4 file system image.

```console
$ dd if=/dev/zero of=disk.img bs=1024 count=20480
$ mkfs.ext4 -F disk.img
```

Note that Solo5 requires an application manifest on build time, which is embedded in a unikernel binary. In current frankenlibc Solo5 support, the manifest is common across binaries and specifies `rootfs` block device and `tap` network device. We have to provide these devices even not used in the applications.

* https://github.com/retrage/frankenlibc/blob/solo5/platform/solo5/manifest.json

Run `hello` test.

```console
$ RUMP_VERBOSE=1 ./rump/bin/rexec rumpobj/tests/hello rootfs:disk.img tap:tap100
```

In the Linux platform, `rexec` provides a sandbox environment for unikernels using seccomp like Solo5's tenders. In the Solo5 platform, it is just a shell script wrapper for `spt` tender.

## LKL Nabla Containers

Now, it's time to integrate with Nabla Containers. Since the original runnc imports older version of Solo5, I updated it and adapted the runnc code base.

* https://github.com/retrage/runnc/tree/lkl-musl

### Updating Supplied Arguments

Below is the original code that creates arguments for Solo5 tender.

```go
    var args []string
    if mac != "" {
        args = []string{r.NablaRunBin,
            "--x-exec-heap",
            "--mem=" + strconv.FormatInt(r.Memory, 10),
            "--net-mac=" + mac,
            "--net=" + r.Tap,
            "--disk=" + disk,
            r.UniKernelBin,
            unikernelArgs}
    } else {
        args = []string{r.NablaRunBin,
            "--x-exec-heap",
            "--mem=" + strconv.FormatInt(r.Memory, 10),
            "--net=" + r.Tap,
            "--disk=" + disk,
            r.UniKernelBin,
            unikernelArgs}
    }
```

In the latest Solo5 (frankenlibc Solo5 platform uses), `--net-mac` option is removed and we can specify multiple block devices and network devices with `--block:` and `--net:` options. Ideally, it should support multiple devices. However, as described before, it can specify `rootfs` and `tap` only. So, the port ends up with the support of these devices like this.

```go
    var args []string
    args = []string{r.NablaRunBin,
        "--mem=" + strconv.FormatInt(r.Memory, 10),
        "--net:tap=" + r.Tap,
        "--block:rootfs=" + disk,
        r.UniKernelBin}
```

### Creating Disk Image

I added `CreateExt4()` function and `llmodules/fs/ext4_storage.go` to create Ext4 rootfs.

```go
// CreateExt4 creates ext4 raw disk image from the dir argument
func CreateExt4(dir string, target *string) (string, error) {
    var fname string

    if target == nil {
        f, err := ioutil.TempFile("/tmp", "nabla")
        if err != nil {
            return "", err
        }

        fname = f.Name()
        if err := f.Close(); err != nil {
            return "", err
        }
    } else {
        var err error
        fname, err = filepath.Abs(*target)
        if err != nil {
            return "", errors.Wrap(err, "Unable to resolve abs target path")
        }
    }

    absDir, err := filepath.Abs(dir)
    if err != nil {
        return "", errors.Wrap(err, "Unable to resolve abs dir path")
    }

    cmd := exec.Command("virt-make-fs", "-F", "raw", "-t", "ext4",
        absDir, fname)
    err = cmd.Run()
    if err != nil {
        return "", errors.Wrap(err, "Unable to run virt-make-fs command")
    }

    return fname, nil
}
```

`virt-make-fs`, a part of [libguestfs](http://libguestfs.org/) has similar interface with `genisoimage`.

It would be better to switch `NewISOFsHandler()` and `NewExt4FsHandler()` on run time.

### Building and Installing runnc

Same as original.

```console
$ git clone https://github.com/retrage/runnc.git
$ mkdir -p $GOPATH/github.com/retrage
$ ln -sf $PWD/runnc $GOPATH/github.com/retrage/runnc
$ cd runnc
$ git apply patches/0001-solo5-elf-segment-align-workaround.patch
$ make build
$ make install
```

### Testing with Docker Images

I provided a set of Makefiles build LKL Nabla Container base Docker images. It builds Solo5 and frankenlibc, and Docker images.

* https://github.com/retrage/lkl-nabla-base-build

I also pushed pre-built Docker images to Docker Hub.

* [retrage/lkl-nabla-hello-base](https://hub.docker.com/repository/docker/retrage/lkl-nabla-hello-base)
* [retrage/lkl-nabla-python3-base](https://hub.docker.com/repository/docker/retrage/lkl-nabla-python3-base)

You can use images like this.

```console
$ sudo docker run --rm --runtime=runnc retrage/lkl-nabla-python3-base:
latest -c "print(\'hello\')"
[sudo] password for akira:
nabla-run arg [/opt/runnc/bin/nabla-run --mem=512 --net:tap=tap28157ba5950e --bl
ock:rootfs=/var/run/docker/runtime-runnc/moby/28157ba5950e3e84824bd843fd1dafb06eccc7de2020a0619d6a5b463e5f2c2b/rootfs.img /var/lib/docker/overlay2/3d36c19950e53eefded8e1933f3d7e51990fc4c7b065be6c00776eeab8fb3136/merged/python3.nabla __RUMP_FDINFO_NET_tap=4 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=28157ba5950e PYTHONHASHSEED=1 PYTHONHOME=/usr/local HOME=/ -- -c print(\'hello\')]
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Bindings version v0.6.4-6-g756accf-dirty
Solo5: Memory map: 512 MB addressable:
Solo5:   reserved @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x889fff)
Solo5:     rodata @ (0x88a000 - 0xb4cfff)
Solo5:       data @ (0xb4d000 - 0xe7dfff)
Solo5:       heap >= 0xe7e000 < stack < 0x20000000
sleeping 50000 usec
hello
Solo5: solo5_exit(0) called
```

## Conclusion

In this post, I introduced a brief of LKL Nabla Containers. It is still in an early stage and has room for improvement, but already runs practical applications like Python. I would like to measure the performance and evaluate the pros/cons.

Below is the TODO list:

* Replace workaround for Solo5
* Flexible `manifest.json` handling on build time
* ~~Pass `lkl.json` through run time arguments~~
* Do not pass `__RUMP_FDINFO_NET_tap=4` environment variable on run time

## Update: May 1st, 2020

After wrote this post, I found that LKL must use network information created by the container runtime. Otherwise, the network does not work properly. I added the 3rd feature described in the above TODO list to frankenlibc and runnc.

* [add external lkl json config support](https://github.com/retrage/frankenlibc/commit/fb4fde66c73c8bec58d754249db77edb66537955)
* [Create lkl config json](https://github.com/retrage/runnc/commit/e73c1203e8a1b19d4813917d893aec6181432e01)

The OCI runtime builds and passes JSON config for LKL at startup. LKL parses it along with environment variables and arguments.

Now, popular network applications Nginx and redis work on LKL Nabla Containers. They are available as base Docker Images.

* [retrage/lkl-nabla-nginx-base](https://hub.docker.com/repository/docker/retrage/lkl-nabla-nginx-base)
* [retrage/lkl-nabla-redis-base](https://hub.docker.com/repository/docker/retrage/lkl-nabla-redis-base)