Docker Images Without Docker — A Practical Guide
Tags: devops, docker, podman, kaniko
Published: 2020-12-02 14:36:16

The most well-known security flaw in Docker is that it requires root access to build your Docker images with the Docker daemon.

We have all read at least once that you should be careful using root access. This tutorial will take a look at the downsides of using Docker and Docker alternatives to combat those, namely:
<ul>
 	<li><a href="https://github.com/GoogleContainerTools/kaniko">Kaniko</a></li>
 	<li><a href="https://buildah.io/">Buildah</a> &amp; <a href="https://podman.io/">Podman</a></li>
</ul>
<h1>What’s the Problem?</h1>
To understand why the Docker Daemon is running with root access and how this can become a problem, we first have to understand the Docker architecture (at least on a high level). Container images are specified with the Dockerfile. The Dockerfile details how to <a href="https://docs.docker.com/engine/reference/commandline/build/">build</a> an image based on your application and resources. Using Docker, we can use the build command to build our container image. Once you have the image of your Dockerfile, you can <a href="https://docs.docker.com/engine/reference/commandline/run/">run</a> it. Upon running the image, a container is created.

Below is a simplified diagram of the Docker architecture, taken directly from the <a href="https://docs.docker.com/get-started/overview/">official documentation</a>.

<a href="https://codefresh.io/wp-content/uploads/2020/12/docker-architecture-overview.png"><img class="wp-image-18211 size-medium" src="https://codefresh.io/wp-content/uploads/2020/12/docker-architecture-overview-300x235.png" alt="Overview: Docker Architecture" width="300" height="235" /></a>

The problem with this is that you cannot use Docker directly on your workstation. Docker is composed of a variety of different tools. In most cases, you will only interact with the Docker CLI. However, running an application with Docker means that you have to run the <a href="https://docs.docker.com/engine/install/linux-postinstall/">Docker Daemon with root privileges</a>. It actually binds to a Unix socket instead of a TCP port. By default, users can only access the Unix socket using <i>sudo</i> command, which is owned by the user <i>root</i>.

The Docker Daemon is responsible for the state of your containers and images, and facilitates any interaction with “the outside world.” The Docker CLI is merely used to translate commands into API calls that are sent to the Docker Daemon. This allows you to use a local or remote Docker Daemon.

Running the Docker Daemon locally, you risk that any process that breaks out of the Docker Container will have the same rights as the host operating system. This is why you should only trust your own images that you have written and understand.

The next section will discuss alternatives to Docker. All of the alternatives discussed below can build OCI compatible container images. OCI stands for <a href="https://opencontainers.org/">Open Container Initiative</a>, which is a project by the Linux Foundation with the goal to define open standards for operating system-level virtualization; more specifically, Linux containers. This includes the runtime specification and the image specification of containers. <a href="https://twitter.com/fatherlinux">Scott McCarty</a> <a href="https://www.youtube.com/watch?time_continue=20&amp;v=rlj0UZlvGp0">discusses in detail </a>how different container standards fit together; here are some of the benefits of using container standards:
<ul>
 	<li>Higher flexibility between tools. If you can just replace the Docker Daemon in your Kubernetes cluster without noticing a difference, does it matter which tool you are using?</li>
 	<li>Gain a better understanding of containers -- what actually happens when you run a container?</li>
 	<li>Higher reliability on your infrastructure. The work that you have put into your existing infrastructure does not become obsolete when you decide to switch to different tools.</li>
</ul>
<h1>Alternative: Buildah &amp; Podman</h1>
Buildah provides a CLI tool that allows users to build OCI or traditional Docker images. Buildah can be used to create and run images from a Dockerfile and without. In our case, we are going to use <a href="https://github.com/containers/buildah">Buildah</a> to build the image and <a href="https://github.com/containers/podman">Podman</a> to run the image.

<b>Relationship between Buildah &amp; Podman</b>

Buildah and Podman are complementary tools to build and run containers. The main differentiation between the two tools is that Buildah focuses on building OCI container images while Podman specializes in the management of the entire container lifecycle. The commands used in Buildah are quite similar to Docker commands. Buildah replicates the same commands that are found within a Dockerfile to allow users to build images without a Dockerfile.

In comparison, Podman is focused on the entire container lifecycle of OCI images but also supports other container standards. Summarising, it helps you to maintain and modify OCI images, such as pulling and tagging, and allows you to run, and maintain containers created from those images.

The main difference to note between both tools is the concept of a container. Buildah is highly similar to Docker in the way containers are built and managed. Comparing this to Podman, in Podman containers are meant to be long-lived. This difference makes both tools complementary; Buildah can be used to create containers and Podman to manage those.

Furthermore, you cannot commit a Buildah container from within Podman nor vice versa. This is reflected in <a href="https://podman.io/blogs/2018/10/31/podman-buildah-relationship.html">the way that commands of both tools are used</a>. For instance, the commit command works differently in Buildah and Podman containers.
<table>
<tbody>
<tr>
<td></td>
<td><b>Buildah</b></td>
<td><b>Podman</b></td>
</tr>
<tr>
<td><b>Focus</b></td>
<td>Container Creation</td>
<td>Manage Containers; Entire Container Lifecycle</td>
</tr>
<tr>
<td><b>Container Management</b></td>
<td>Similar to Docker</td>
<td>Containers are meant to be long-lived</td>
</tr>
<tr>
<td><b>Commands</b></td>
<td>Mirroring Docker Commands in functionality</td>
<td>Some commands might be similar but have different functionality</td>
</tr>
<tr>
<td><b>Image Standards</b></td>
<td>Support OCI images</td>
<td>Support OCI images</td>
</tr>
</tbody>
</table>
<b>Getting started with Buildah and Podman</b>

Buildah has a variety of different installation options (that you can easily get lost in). We run on Ubuntu 20.04 and followed <a href="https://github.com/containers/buildah/blob/master/install.md">these installation instructions</a>:
<pre>. /etc/os-release
</pre>
<pre>sudo sh -c "echo 'deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /' &gt; /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list"
</pre>
<pre>wget -nv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/xUbuntu_${VERSION_ID}/Release.key -O Release.key
</pre>
<pre>sudo apt-key add - &lt; Release.key
</pre>
<pre>sudo apt-get update -qq
</pre>
<pre>sudo apt-get -qq -y install buildah
</pre>
Make sure that you replace the Version ID with your Ubuntu version (in case you are on Ubuntu) -- in our case, this was 20.04.

To check that the installation was successful, check for Buildah images:
<pre class="">buildah images
</pre>
Next, we will  <a href="https://podman.io/getting-started/installation">install Podman</a>:
<pre>. /etc/os-release
</pre>
<pre>echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_${VERSION_ID}/ /" | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:testing.list
</pre>
<pre>curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/testing/xUbuntu_${VERSION_ID}/Release.key | sudo apt-key add -
</pre>
<pre>sudo apt-get update -qq
</pre>
<pre>sudo apt-get -qq -y install podman
</pre>
Once done we can run the following to verify that our installation was successful:
<pre class="">podman --help 
</pre>
Let’s go ahead and build an image with Buildah. For this, we are using the <a href="https://github.com/anais-codefresh/docker-alternatives">following repository</a>. The repository already has a Dockerfile that we can use. However, you can also create or use your own.
<pre># clone the repository
git clone git@github.com:anais-codefresh/docker-alternatives
</pre>
<pre># cd into the cloned repository 
cd docker-alternatives/buildahandpodman
</pre>
Next, we are going to use Buildah to build the image based on our Dockerfile.
<pre>buildah bud -t buildah-example
</pre>
You should now see the different steps of the Dockerfile executed.

<a href="https://codefresh.io/wp-content/uploads/2020/12/dockerfile-buildah-output.png"><img class="size-medium wp-image-18212" src="https://codefresh.io/wp-content/uploads/2020/12/dockerfile-buildah-output-300x226.png" alt="dockerfile-buildah-output" width="300" height="226" /></a>

We can see the images built with Buildah by running:
<pre>podman images
</pre>
And run the container using Podman:
<pre>podman run -dt -p 8080:8080/tcp 
</pre>
We can now view our application on localhost:8080. All Podman commands are listed in <a href="https://podman.readthedocs.io/en/latest/Commands.html">their documentation</a>.

<a href="https://codefresh.io/wp-content/uploads/2020/12/running-go-application-with-buildah-and-podman.png"><img class="size-medium wp-image-18213" src="https://codefresh.io/wp-content/uploads/2020/12/running-go-application-with-buildah-and-podman-300x81.png" alt="running-go-application-with-buildah-and-podman" width="300" height="81" /></a> 
<h1>Alternative: Kaniko</h1>
<a href="https://github.com/GoogleContainerTools/kaniko">Kaniko</a> was started and is maintained by Google. In short, it is an open source tool that allows users to build images even without granting it root access. These images are built inside a container or a Kubernetes cluster. Unlike the Docker daemon, Kaniko executes all commands inside the user space.

There are several different ways to deploy and run Kaniko:
<ul>
 	<li><a href="https://github.com/GoogleContainerTools/kaniko#running-kaniko-in-a-kubernetes-cluster">In a Kubernetes cluster</a></li>
 	<li><a href="https://github.com/GoogleContainerTools/kaniko#running-kaniko-in-gvisor">In gVisor</a></li>
 	<li><a href="https://github.com/GoogleContainerTools/kaniko#running-kaniko-in-google-cloud-build">In Google Cloud Build</a></li>
 	<li><a href="https://github.com/GoogleContainerTools/kaniko#running-kaniko-in-docker">In Docker</a></li>
</ul>
To run a container, Kaniko needs three arguments:
<ul>
 	<li>A Dockerfile</li>
 	<li>A Build Context: The directory containing a Dockerfile which Kaniko can use to build your image -- e.g. COPY in the Dockerfile should refer to a file in the build context</li>
 	<li>The name of the registry to which the final image should be pushed</li>
</ul>
<b>Getting started with Kaniko</b>

We will use Kaniko inside a local Kubernetes Cluster, using MicroK8s. To get started with Kaniko and to follow the next steps, we assume that you have the following set-up:

Either a local Kubernetes Cluster or you are able to access a remote cluster on your machine.

We are going to test Kaniko with an existing Dockerfile. The application we are going to use is a basic <a href="https://github.com/anais-codefresh/docker-alternatives">Go application</a>. You can either clone the application or use your own app and Dockerfile.
<pre># Clone the repository
git clone git@github.com:anais-codefresh/docker-alternatives
</pre>
<pre># cd into the kaniko repository
cd docker-alternatives/kaniko
</pre>
Create a file called <a href="https://github.com/anais-codefresh/docker-alternatives/blob/master/kaniko/volume.yaml">volume.yaml</a> that will specify our <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">PersistentVolume</a>. This is used to run Kaniko inside of Kubernetes locally.
<pre>touch volume.yaml
</pre>
Go ahead and paste the following into your file:
<pre>apiVersion: v1
kind: PersistentVolume
metadata:
 name: dockerfile
 labels:
   type: local
spec:
 capacity:
   storage: 10Gi
 accessModes:
   - ReadWriteOnce
 storageClassName: local-storage
 hostPath:
   path:  # replace with local directory, such as "/home//kaniko"
</pre>
You will have to replace the path to your directory (the one where the files are at). The easiest way to do this is to run:
<pre>pwd
</pre>
And copy and paste the file path.

Next, we need a <a href="https://github.com/anais-codefresh/docker-alternatives/blob/master/kaniko/volume-claim.yaml">second file</a> that volume is mounted to:
<pre>touch volume-claim.yaml
</pre>
Paste the following content:
<pre class="">kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: dockerfile-claim
spec:
 accessModes:
   - ReadWriteOnce
 resources:
   requests:
     storage: 8Gi
 storageClassName: local-storage
</pre>
Lastly, we need to give Kubernetes the specifications on the pod it is supposed to run.
Add the [pod.yaml](https://github.com/anais-codefresh/docker-alternatives/blob/master/kaniko/pod.yaml) file:
<pre>touch pod.yaml
</pre>
And paste the following content:
<pre>apiVersion: v1
kind: Pod
metadata:
 name: kaniko
spec:
 containers:
 - name: kaniko
   image: gcr.io/kaniko-project/executor:latest
   args: ["--dockerfile=Dockerfile",
           "--context=dir://workspace",
           "--destination=/"] # replace with your dockerhub account
   volumeMounts:
     - name: kaniko-secret
       mountPath: /kaniko/.docker
     - name: dockerfile-storage
       mountPath: /workspace
 restartPolicy: Never
 volumes:
   - name: kaniko-secret
     secret:
       secretName: regcred
       items:
         - key: .dockerconfigjson
           path: config.json
   - name: dockerfile-storage
     persistentVolumeClaim:
       claimName: dockerfile-claim
</pre>
Modifications that you have to make to the pod.yaml file:
<ul>
 	<li>Please provide your Docker Hub account name and the repository to which you want to push.
<pre>{% raw %}"--destination=user-name/"{% endraw %}</pre>
</li>
 	<li>Make sure that you include the right location to your Dockerfile. In our case, the Dockerfile is in the same repository as the yaml files.
<pre>“--dockerfile=Dockerfile”</pre>
</li>
</ul>
Now that we have all of this prepared, we need to create the Kubernetes secret to allow Kaniko to push the Docker image to the Docker Hub; note that you can also use another registry.
<pre>{% raw %}kubectl create secret docker-registry regcred --docker-server=your-registry-server --docker-username=your-name --docker-password=your-pword --docker-email=your-email
{% endraw %}</pre>
<ul>
 	<li>&lt;your-registry-server&gt;  is: (<a href="https://index.docker.io/v1/">https://index.docker.io/v1/</a> for DockerHub)</li>
 	<li>&lt;your-name&gt; is your Docker username.</li>
 	<li>&lt;your-pword&gt; is your Docker password.</li>
 	<li>&lt;your-email&gt; is your Docker email.</li>
</ul>
In our case, we had to put the values in quotes e.g. “anaisurlichs” for your username.

Next, we are going to create our persistent volume. Note that it is important that you create the secret first before you create the volumes and the pod.
<pre># create persistent volume
$ kubectl create -f volume.yaml
persistentvolume/dockerfile created
</pre>
<pre># create persistent volume claim
$ kubectl create -f volume-claim.yaml
persistentvolumeclaim/dockerfile-claim created
</pre>
<pre># check whether the volume mounted correctly
$ kubectl get pv dockerfile
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS    REASON   AGE
dockerfile   10Gi       RWO            Retain           Bound    default/dockerfile-claim   local-storage            1m
</pre>
Lastly, we are creating the pods:
<pre>kubectl create -f pod.yaml
</pre>
<a href="https://codefresh.io/wp-content/uploads/2020/12/using-kaniko-console-output.png"><img class="size-medium wp-image-18214" src="https://codefresh.io/wp-content/uploads/2020/12/using-kaniko-console-output-300x48.png" alt="using-kaniko-console-output" width="300" height="48" /></a>

You can now view the pod:
<pre>kubectl get pods</pre>
It took some time for us to figure out how to get started with Kaniko. In our case, we used Kaniko in Kubernetes. Alternatively, you could also use the Docker Daemon to run your container images. However, in this case, there would not be much of a difference between using Docker directly. If you would still like to do this, you do not require any of the files listed above. Instead, you would just run the following command with similar parameters to what we have specified in pod.yaml.

First, we need a config.json file in our current directory that holds our Docker login:
<pre>{
   "auths": {
       "https://index.docker.io/v1/": {
           "auth": "xxxxxxxxxxxxxxx"
       }
   }
}
</pre>
On Ubuntu, you can encode your credentials with:
<pre>echo -n “username:password” | base64
</pre>
Use the encoded credentials within your config.json file. Next, we can build and push our Docker image using:
<pre>docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --destination=yourimagename
</pre>
Note that this command might take a while to finish. Below is the expected output:

<a href="https://codefresh.io/wp-content/uploads/2020/12/kaniko-console-output.png"><img class="size-medium wp-image-18215" src="https://codefresh.io/wp-content/uploads/2020/12/kaniko-console-output-300x168.png" alt="kaniko-console-output" width="300" height="168" /></a>
<h1>The Difference between Kaniko and Buildah</h1>
Buildah allows you to build images without a Dockerfile. The flexibility of building images without Dockerfiles allows for the integration of other scripting languages into the build process. Since Buildah does not rely on a Daemon but instead on a comprehensive API written in Golang, commands can be translated into other tools.

In comparison, Kaniko can be run with and without root access to build a container image and push it to a registry. While Kaniko is supported by Google, Podman has been developed by RedHat. Both projects are actively maintained.

Additional differences that we have noticed while using both tools:
<ul>
 	<li>Depending on your operating system, it might be quite difficult to install any of the mentioned tools.</li>
 	<li>In case you are already using Kubernetes, Kaniko allows you to get started right away. However, you may have to invest some time into setting up the yaml files.</li>
 	<li>Podman and Buildah are quite intuitive to use and use similar commands to those that are defined by Docker.</li>
 	<li>Additionally, Buildah has better tutorials and further documentation available, which might become handy in the long term.</li>
 	<li>You can run both Kaniko and Buildah from within a container. However, it is <a href="https://developers.redhat.com/blog/2019/08/14/best-practices-for-running-buildah-in-a-container/">not as straightforward using Buildah</a>.</li>
 	<li>We built the same image using Kaniko and then using Buildah to compare the image size; the image build by Kaniko using Docker is about half the size of the one built by Buildah.</li>
</ul>
Image built with Buildah and Podman:

<a href="https://codefresh.io/wp-content/uploads/2020/12/podman-image-size.png"><img class="size-medium wp-image-18216" src="https://codefresh.io/wp-content/uploads/2020/12/podman-image-size-300x23.png" alt="podman-buildah-image-size" width="300" height="23" /></a>

Image built with Kaniko:

<a href="https://codefresh.io/wp-content/uploads/2020/12/kaniko-image-size.png"><img class="size-medium wp-image-18218" src="https://codefresh.io/wp-content/uploads/2020/12/kaniko-image-size-300x59.png" alt="kaniko-image-size" width="300" height="59" /></a>

Overall, it depends on your use case and the tools that you are already using to build and manage your containers. Generally, it would not make much sense to use Kaniko with Docker. However, if you are already heavily using Kubernetes, then it would be an easy integration and a great alternative. In contrast, the combination of Buildah and Podman is a better alternative if you want to directly replace Docker.
<h1>Summary</h1>
In this tutorial, we compared two different alternatives to using Docker directly. Neither has been as easy to get started with as Docker. In summary, if you are already using Kubernetes, Kaniko would be a good alternative; if not, we would prefer Buildah and Podman. As you can tell, which tool would provide the best alternative depends heavily on your use case.

- - -

Are you using Docker or either of the alternatives detailed in this tutorial? Let us know about your experience in the comments!