How to copy a file from a Docker container to the host 📋
Tags: docker, linux, bash, programming
Published: 2021-08-19 22:38:55

---
title: How to copy a file from a Docker container to the host 📋
published: true
date: 2021-08-05 23:15:00 UTC
tags: docker, linux, bash, programming
canonical_url: https://www.benjaminrancourt.ca/how-to-copy-a-file-from-a-docker-container-to-the-host
cover_image: https://www.benjaminrancourt.ca/images/how-to-copy-a-file-from-a-docker-container-to-the-host.jpg
---
Recently, we needed to **download a file** from a **Docker container**. Since the file was not inside a [bind mount](https://docs.docker.com/storage/bind-mounts/), we couldn't directly access the remote server to get the file on the file system.

So we quickly found the [docker cp](https://docs.docker.com/engine/reference/commandline/cp/) utility to copy the file on our host and voila, we had the file on our computer. ☺️

```bash
docker cp CONTAINER_NAME:/opt/service/FILENAME_TO_OBTAIN .
```
<figcaption>A simple example of the <code>docker cp</code> command, where the name of the container is known.</figcaption>

Unfortunately, we didn't know the **exact name** of the container, as containers created by `docker compose` are, by default, suffixed with a randomly generated hexadecimal string, such as `.eq6l28g5mwsenh0gvag7zcg0j`.  😶‍🌫️

And because we deployed our stack in [swarm mode](https://docs.docker.com/engine/swarm/), we couldn't use the [container\_name option](https://docs.docker.com/compose/compose-file/compose-file-v3/#container_name) as it is ignored in this mode...

So, to obtain the full name of the container, we used the [docker ps](https://docs.docker.com/engine/reference/commandline/ps/) command, which lists the containers, along with its `--format` option to get only its full name.

```bash
docker ps --format='{{.Names}}' | grep "CONTAINER_NAME_PREFIX"
```
<figcaption>An example of the <code>docker ps</code> command, where we know the prefix of the container name.</figcaption>

If we combine these commands, we can have only one command to run to download our file.

```bash
docker cp "$(docker ps --format='{{.Names}}' | grep "CONTAINER_NAME_PREFIX")":"/opt/service/FILENAME_TO_OBTAIN" .
```
<figcaption>The resulting command, where we can copy a file from a container to our host.</figcaption>

It worked fine on our local computer, but the real file we were interested in was on a remote server... And this is where it all gets complicated for us. 🤮

## Constraints

- We wanted an **automated solution** where we could just provide some information and we would get the file. 🤖
- The container was hosted on a **random server** from a pool of Docker machines. 🌈
- We had to be **privileged users** to run certain Docker commands. 👑

After a lot of guesswork and mistakes, we finally managed to get a working solution under these constraints, but it took longer than expected. Let's dive into some of the issues we encountered. 😌

## Issues

### How to run sudo commands on a remote server

In my opinion, this step is the most complicated part of our final script. Please be patient with me as I explain some of the things we have done below. 🤢

```bash
# Define and export some variables that will be used in this part
export SERVER=SERVER_HOSTNAME
export FILENAME=FILENAME_TO_OBTAIN
export CONTAINER_PREFIX=ranb2002_system
export PASSWORD=PASSWORD_TO_BE_ROOT

# Create a template where the environment variables will be replaced
TEMPLATE=$(
    envsubst <<'EOF'
    # Authenticate as sudo and pass the password without indentation
    sudo -S whoami
$PASSWORD

    # Delete the previous file if it exists on the server
    rm -f "${FILENAME}"

    # List all Docker containers on this server
    CONTAINERS=$(sudo -S docker ps --format='{{.Names}}')
    
    # Check if we have a container for our application on this server
    CONTAINER_NAME=$(printf '%s\n' "${CONTAINERS[@]}" | grep "${CONTAINER_PREFIX}")

    # Otherwise, continue to the next server
    if [-z "${CONTAINER_NAME[0]}" ]; then
      echo "The container with name ${CONTAINER_PREFIX} was not found on this server ($(hostname))... :("
      exit
    fi
    
    echo "The container with name ${CONTAINER_NAME[0]} was found on this server ($(hostname))! :)"

    # Copy the file from the container to the current host
    sudo -S docker cp "${CONTAINER_NAME[0]}":"/opt/service/${FILENAME}" .
    
EOF
  )
  
# Connect to the server and run all commands from the template
ssh -T "${USER}@${SERVER}" <<< "$TEMPLATE"

```
<figcaption>The script runs a large number of commands on a remote server, where some of them require root privileges.</figcaption>
- To run multiple commands via SSH, we use a [here document](https://en.wikipedia.org/wiki/Here_document) structure that lists all the commands we want to run on the remote host.
- Since the variables were not replaced inside this structure, we encompassed it with the [envsubst](https://www.gnu.org/software/gettext/manual/html_node/envsubst-Invocation.html) command.
- To make the shell variables accessible to `envsubst`, we had to `export` them.
- Since we wanted to capture the container name in a variable, we ended up splitting the commands because our one-line command didn't work well with `sudo`.
- Since my user was not able to run Docker commands on this server, we had to gain **root privileges**. We have tried many (many) attempts and we managed to get a working solution by outputting our password to the first `sudo -S` command. The following commands don't need to have the password again. 🥺
- For some reason still unknown to me, the value of the `CONTAINER_NAME` variable is an array... So, to access it, we had no choice but to use `CONTAINER_NAME[0]`... 🤷

> As you may have noticed, we didn't need to pass our password when running the `ssh` command, because we are using [SSH public keys](https://git-scm.com/book/en/v2/Git-on-the-Server-Generating-Your-SSH-Public-Key). If you want to install SSH keys on your servers, check out my article on [how to automate ssh-copy-id](https://www.benjaminrancourt.ca/automating-ssh-copy-id)! 😉

### How to transfer the file from the remote server to our host

This part was easy to find, using [sshpass](https://linux.die.net/man/1/sshpass) and [scp](https://en.wikipedia.org/wiki/Secure_copy_protocol) commands.

```bash
# Copy the file from the remote server to our host
sshpass -p "${PASSWORD}" scp "${USER}@${SERVER}":"/home/${USER}/${FILENAME}" .

```
<figcaption>Using <code>sshpass</code> to suppress the password prompt when connecting to the remote server.</figcaption>

Since we had previously installed an SSH public key on the server and we are using the same username on our host and the server, we can **rewrite** the previous command as follows.

```bash
scp "${SERVER}":"~/${FILENAME}" .
```
<figcaption>A small command when you can make some assumptions.</figcaption>

Simpler, right? 😉

### How to find the server where the application container runs

To browse over a cluster of servers, we have provided the list as an array, along with a simple switch case. It's always useful when we need it to have a switch case example **in Bash** , so I'm putting it here for future reference. 😋

```bash
case "${ENVIRONMENT}" in
  DEV)
    SERVERS=(
      "DEV_SERVER_1_HOSTNAME"
      "DEV_SERVER_2_HOSTNAME"
    )
  ;;

  TEST)
    SERVERS=(
      "TEST_SERVER_1_HOSTNAME"
      "TEST_SERVER_2_HOSTNAME"
    )
  ;;

  *)
    SERVERS=(
      "PRODUCTION_SERVER_1_HOSTNAME"
      "PRODUCTION_SERVER_2_HOSTNAME"
    )
  ;;
esac

for SERVER in "${SERVERS[@]}"; do
  # All previous commands
done

```
<figcaption>A switch case that lists the servers we needed with a for-loop.</figcaption>

Unfortunately, since we are not able to know on which server we need to transfer the file from, we have to run all the previous commands **on each server**... 😵‍💫

So, we get a lot of `scp: /home/ranb2002/[...]: No such file or directory` error because only one server has the wanted file... 😒

### How to ensure the arguments are provided

Since our script could **fail** if some arguments were not supplied or were incorrect, we added some conditions to verify these cases.

> **Note** : if the password is incorrect or if the user is not a privileged user, the script will still fail because we haven't taken some time to figure out how to prevent this.

```bash
#!/bin/bash
echo "Usage:"
echo " ./scripts/copy_config.sh 'PASSWORD' ENVIRONMENT"
echo " where ENVIRONMENT is DEV, TEST, DEMO or PROD."
echo ""

# Make sure we have your password
if [-z "${1}"]; then
  echo "You must provide your password!"
  exit
fi

# Export the password to an environment variable
export PASSWORD="${1}"

# Make sure we have the environment
if [-z "${2}"]; then
  echo "You must provide the environment!"
  exit
fi

export ENVIRONMENT="${2}"

# Make sure the environment provided is supported
ENVIRONMENTS=("DEV" "TEST" "DEMO" "PROD")
if [[! " ${ENVIRONMENTS[*]} " == *" ${ENVIRONMENT} "* ]]; then
  #
  echo "The environment provided is incorrect!"
  exit
fi

```
<figcaption>Some checks we made at the start of our script.</figcaption>

> **Bonus** : if for some reason you need to transform an uppercase value to its lowercase equivalent, here is the Bash command you can use: `LOWER_ENVIRONMENT=$(echo "${ENVIRONMENT}" | awk '{print tolower($0)}')`. 😜

## **Final solution**

If we put together all the pieces together, our final script looks like below. 🧩

```bash
#!/bin/bash
echo "Usage:"
echo " ./scripts/copy_config.sh 'PASSWORD' ENVIRONMENT"
echo " where ENVIRONMENT is DEV, TEST, DEMO or PROD."
echo ""

# Make sure we have your password
if [-z "${1}"]; then
  echo "You must provide your password!"
  exit
fi

# Export the password to an environment variable
export PASSWORD="${1}"

# Make sure we have the environment
if [-z "${2}"]; then
  echo "You must provide the environment!"
  exit
fi

export ENVIRONMENT="${2}"

# Make sure the environment provided is supported
ENVIRONMENTS=("DEV" "TEST" "DEMO" "PROD")
if [[! " ${ENVIRONMENTS[*]} " == *" ${ENVIRONMENT} "* ]]; then
  #
  echo "The environment provided is incorrect!"
  exit
fi

LOWER_ENVIRONMENT=$(echo "${ENVIRONMENT}" | awk '{print tolower($0)}')
export DOCKER_STACK="stack-${LOWER_ENVIRONMENT}_service"

export FILENAME="FILENAME_TO_DOWNLOAD"

case "${ENVIRONMENT}" in
  DEV)
    SERVERS=(
      "DEV_SERVER_1_HOSTNAME"
      "DEV_SERVER_2_HOSTNAME"
    )
  ;;

  TEST)
    SERVERS=(
      "TEST_SERVER_1_HOSTNAME"
      "TEST_SERVER_2_HOSTNAME"
    )
  ;;

  *)
    SERVERS=(
      "PRODUCTION_SERVER_1_HOSTNAME"
      "PRODUCTION_SERVER_2_HOSTNAME"
    )
  ;;
esac

for SERVER in "${SERVERS[@]}"; do
  TEMPLATE=$(
    envsubst <<'EOF'
    # Authenticate as sudo and pass the password with no indentation
    sudo -S whoami
$PASSWORD

    # Delete the previous file if it exists
    rm -f "${FILENAME}"

    # List all Docker containers on this server
    CONTAINERS=$(sudo -S docker ps --format='{{.Names}}')

    # Check if we have a container for our application on this server
    CONTAINER_NAME=$(printf '%s\n' "${CONTAINERS[@]}" | grep "${DOCKER_STACK}")

    # Otherwise, continue to the next server
    if [-z "${CONTAINER_NAME[0]}" ]
    then
      echo "The container with name ${DOCKER_STACK} was not found on this server ($(hostname))... :("
      exit
    fi

    echo "The container with name ${CONTAINER_NAME[0]} was found on this server ($(hostname))! :)"

    # Copy the file from the container to the current host
    sudo -S docker cp "${CONTAINER_NAME[0]}":"/opt/service/${FILENAME}" .
EOF
  )

  # Connect to the server and run all commands from the template
  ssh -T "${SERVER}" <<< "$TEMPLATE"

  # Copy the file from the server to our computer
  # shellcheck disable=SC2140
  scp "${SERVER}":"~/${FILENAME}" .
done

```
<figcaption>A complicated script just to download a file from a Docker container, right?</figcaption>

Hope this post is useful to you! Stay safe! 🍻