Troubleshooting permission issues when building Docker containers
Tags: docker, node, debugging, tutorial
Published: 2024-01-01 23:20:32

Docker containers run by default as root. In general, it is not a recommended practice as it poses a serious security risk. This risk can be mitigated by configuring a non-root user to run the container. One of the ways to achieve this is to use the `USER` instruction in the `Dockerfile`. While running a container as a non-root user is the right thing to do, it can often be problematic as insufficient permissions can lead to hard to diagnose errors. This post uses an example [node application](https://github.com/moozzyk/docker-permissions) to discus a few permission-related issues that can pop up when building a non-root container along with some strategies that can help troubleshoot this kind of issues.

## Example
Let’s start with a very simple `Dockerfile` for a node application:

```Docker
FROM node:16
WORKDIR /usr/app
COPY . .
RUN npm install
CMD [ "node", "/usr/app/index.js" ]
```

The problem with this Docker file is that any Docker container created based on this file will run as root:

![Example node application running as root](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9mrs37ru8tn929udyqh7.png)

To fix that we can modify the Docker file to create a new user (let’s call it `app-user`) and move the application to a sub-directory in the user home directory like this:

```Docker
FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN chown -R app-user:app-user $HOME
USER app-user
WORKDIR $HOME/app
COPY . .
RUN npm install
CMD [ "node", "index.js" ] 
```

Unfortunately, introducing these changes makes it impossible to build a docker image – `npm install` now errors out due to insufficient permissions:

```bash
Step 8/9 : RUN npm install
 ---> Running in a0800340b850
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /home/app-user/app/node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/home/app-user/app/node_modules'
npm ERR!  [Error: EACCES: permission denied, mkdir '/home/app-user/app/node_modules'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/home/app-user/app/node_modules'
npm ERR! }
...
```

Inspecting the `app` directory shows that the owner of this directory is `root` and other users don’t have the write permission:

```bash
app-user@d0b48aa18141:~$ ls -l ~
total 4
drwxr-xr-x 1 root root 4096 Jan 15 05:48 app
```

The error is related to using the `WORKDIR` instruction to set the working directory to `$HOME/app`. It’s not a problem by itself – it’s actually recommended to use `WORKDIR` to set the working directory. The problem is that because the directory didn’t exist, `WORKDIR` created one, but it made `root` the owner. The issue can be easily fixed by explicitly creating a working directory with the right permissions before the `WORKDIR` instruction runs to prevent `WORKDIR` from creating the directory. The new `Dockerfile` that contains this fix looks as follows:

```Docker
FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN mkdir -p $HOME/app
RUN chown -R app-user:app-user $HOME
USER app-user
WORKDIR $HOME/app
COPY . .
RUN npm install
CMD [ "node", "index.js" ]
```
Unfortunately, this doesn’t seem to be enough. Building the image still fails due to a different permission issue:

```bash
Step 10/11 : RUN npm install
 ---> Running in 860132289a60
npm ERR! code EACCES
npm ERR! syscall open
npm ERR! path /home/app-user/app/package-lock.json
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, open '/home/app-user/app/package-lock.json'
npm ERR!  [Error: EACCES: permission denied, open '/home/app-user/app/package-lock.json'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'open',
npm ERR!   path: '/home/app-user/app/package-lock.json'
npm ERR! }
...
```

The error message indicates that this time the problem is that `npm install` cannot access the `package-lock.json` file. Listing the files shows again that all copied files are owned by `root` and other users don’t have the write permission:

```bash
ls -l
total 12
-rw-r--r-- 1 root root  71 Jan 15 02:03 index.js
-rw-r--r-- 1 root root 849 Jan 15 01:36 package-lock.json
-rw-r--r-- 1 root root 266 Jan 15 05:21 package.json
```
Apparently, the [COPY](https://docs.docker.com/engine/reference/builder/#copy) instruction by default uses `root` privileges, so the files will be owned by `root` even if the `COPY` instruction appears the `USER` instruction. An easy fix is to change the `Dockerfile` to copy the files before configuring file ownership (alternatively, it is possible specify a different owner for the copied files with the `--chown` switch):

```Docker
FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN mkdir -p $HOME/app
COPY . .
RUN chown -R app-user:app-user $HOME
USER app-user
WORKDIR $HOME/app
RUN npm install
CMD [ "node", "index.js" ]
```
Annoyingly, this still doesn’t work – we get yet another permission error:

```bash
Step 9/10 : RUN npm install
 ---> Running in d4ebcec114cb
npm ERR! code EACCES
npm ERR! syscall mkdir
npm ERR! path /node_modules
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, mkdir '/node_modules'
npm ERR!  [Error: EACCES: permission denied, mkdir '/node_modules'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/node_modules'
npm ERR! }
...
```
This time the error indicates that `npm install` tried creating the `node_modules` directory directly in the root directory. This is unexpected as the `WORKDIR` instruction was supposed to set the default directory to the app directory inside the newly created user home directory. The problem is that the last fix was not completely correct. Before, `COPY` was executed after `WORKDIR` so it copied the files to the expected location. The fix moved the `COPY` instruction so that it is now executed before the `WORKDIR` instruction. This resulted in copying the application files to the container’s root directory, which is incorrect. Preserving the relative order of these two instructions should fix the error:
```Docker
FROM node:16
ENV HOME=/home/app-user
RUN useradd -m -d $HOME -s /bin/bash app-user
RUN mkdir -p $HOME/app
WORKDIR $HOME/app
COPY . .
RUN chown -R app-user:app-user $HOME
USER app-user
RUN npm install
CMD [ "node", "index.js" ]
```
Indeed, building an image with this `Dockerfile` finally yields:
```bash
Successfully built b36ac6c948d3
```
The application also runs as expected:

![Example node application running as non-root](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7xvjkhson43q728v2cud.png)

## Debugging strategies

Reading about someone’s errors is one thing, figuring the errors out oneself is another. Below are a few debugging strategies I used to understand the errors described in the first part of the post. Even though I mention them in the context of permission errors they can be applied in a much broader set of scenarios.

### Carefully read error messages

All error messages we looked at were very similar, yet each signaled a different problem. While the errors didn’t point directly to the root cause, the small hints were very helpful in understanding where to look to investigate the problem.

### Check Docker documentation

Sometimes our assumptions about how the given instruction runs may not be correct. [Docker documentation](https://docs.docker.com/engine/reference/builder/) is the best place to verify these assumptions and understand if the wrong assumptions could be the culprit (e.g. the incorrect assumption that the `COPY` will make the current user the owner of the copied files).

### Add additional debug info to `Dockerfile`

Sometimes it is helpful to print additional debug information when building a docker image. Some commands I used were:

* `RUN ls -al`
* `RUN pwd`
* `RUN whoami`

They allowed me understand the state the container was in at a given time. One caveat is that by default docker caches intermediate steps when building containers which may result in not printing the debug information when re-building a container if no changes were made as the step was cached.

### Run the failing command manually and/or inspect the container

This is the ultimate debugging strategy – manually reproduce the error and inspect the container state. One way to make it work is to comment out all the steps starting from the failing one and then build the image. Once the image is build start a container like this (replace `IMAGE` with the image id):

```bash
docker run -d IMAGE tail -f /dev/null
```
This will start the container whose state is just as it was before the failing step was executed. The command will also keep the container running which makes it possible for you to launch bash within the container (replace `CONTAINER` with the container id returned by the previous command):

```bash
docker exec -it CONTAINER /bin/bash
```

Once inside the container you can run the command that was failing (e.g. `npm install`). Since the container is in the same state it was when it failed to build you should be able to reproduce the failure. You can also easily check for the factors that caused the failure.

## Conclusion

his post showed how to create a docker container that is not running as root and discussed a few permission issues encountered in the process. It also described a few debugging strategies that can help troubleshoot a wide range of issues – including issues related to permissions. The code for this post is available on github in my [docker-permissions](https://github.com/moozzyk/docker-permissions) repo.