Jupyter notebooks for Spark with customised Docker containers
Tags: docker, spark, jupyter, python
Published: 2022-01-07 09:22:47


When we work with Spark we usually want to first prototype to see if everything works as expected, before we start up big machines. 
I spent an afternoon googling and starting and stopping the docker container to finally configure some lines of code.
So I want to share my basic local setup here, so maybe it will help someone to save some time.

When looking for a docker image with spark and jupyter we find the [pyspark-notebook](https://hub.docker.com/r/jupyter/pyspark-notebook).

In my case I need to access AWS, so I need some additional libaries for the docker image.
To add them, I created a new `Dockerfile` based on the pyspark-notebook.
The additional libraries needed are `boto3` for AWS and `python-dotenv` to access environment variables.
I decided to install boto3 with apt-get as this will be installed on the operating system level. Make sure to add `-y` if the operating system is asking something during the install process, we will answer with `yes`. 
The dotenv is added via a requirements.txt so it will installed via pip, the python package manager.

Normally for the notebooks you need to have a token, but when we develop locally, we want to access the jupyter-notebook quickly and stay on the same site, without having to lookout for the new token everytime we change something.
So we need an custom configuration for that: 

```
{
    "NotebookApp": {
        "allow_root": true,
        "token": ""
    }
}
```

In the Dockerfile we copy everything we need into to `/home/jovyan/` directory. After some more googling I found out that this user _jovyan_ stands for _jupyter like environment_. Just in case you where also wondering.

The final Dockerfile looks like this:

```
FROM jupyter/pyspark-notebook
USER root

# add needed packages
RUN apt-get update && apt-get install python3-boto3 -y

# Install Python requirements
COPY requirements.txt /home/jovyan/
RUN pip install -r /home/jovyan/requirements.txt

COPY jupyter_lab_config.json /home/jovyan/
```

In the `docker-compose.yaml` we
- need to map the ports, 
- map the volumes to save the notebook locally, otherwise everything would be lost, once we shut down the container and point to the env file. 
- tell Docker where the `.env` file is located
- tell Docker to build the Dockerfile in the same folder, instead of using an image.

The final `docker-compose.yaml` looks like this:

```
version: "3.7"

services:
  # jupyterlab with pyspark
  pyspark:
    #image: jupyter/pyspark-notebook
    build: .
    env_file: 
      - .env
    environment:
      JUPYTER_ENABLE_LAB: "yes"
    ports:
      - "8888:8888"
    volumes:
      - ./data:/home/jovyan/work

# docker run --rm -p 10000:8888 -e JUPYTER_ENABLE_LAB=yes -v "$PWD":/home/jovyan/work jupyter/pyspark-notebook
```


To start the container use `docker-compose up`, if you changed something in the config use `docker-compose up --force-recreate --build` to make sure the changes are build.

Have fun.

You can find the code also [here](https://github.com/BarbaraJoebstl/data-engineering-nd/tree/master/data-lakes/project-data-lakes/compose).