Docker for setting the developer Environnement
Tags: docker, python, datascience, devops
Published: 2021-05-08 15:46:09

### Setting up a Dev Environment for the datascience

When working on a python project, a common way that people will manage their dependencies is running `pip freeze > requirements.txt` or `pip install -r requirements.txt` and coupling that with `virtualenv` to manage project level dependencies (that is required for maintaining the project environment).

Often, when reproducing the project to somebody else's we have to give the **.exe** version of the file and that is not good for testing, analysis, it is not enough to run `pip install -r requirements.txt`in the repository.  

However, sometimes there are configurations that are specific system level dependencies that are not captured. As we go along with developing the code, we will install system-level dependencies as required by the package.  

To be able to replicate all the system level dependencies, you can see how Docker could easily be used and can be shared through the docker hub just like the containers are sent and then the developer environment can be run.

Now we move to the next stage, which is figuring out how to layer instructions, and put together a Dockerfile to set up a docker container with your exact specifications to replicate your results. 

We have observed the following issues: 

1. Somebody builds a tool in a different flavor of python with several package changes (each package has a dependent package which can give different version `may be a older version for the new framework installed`) 
2. Struggling to manage `virtualenvs` for different packages, broken `virtualenvs` or else forgetting it also is a big chance here.

The use case we will explore here, is setting up a python machine with all the tools required for a project, and then setting up a jupyter notebook server within to access all those resources: 

```dockerfile

#this is just a example

FROM python:3

RUN apt-get update && apt-get install -y python3-pip

COPY requirements.txt .

RUN pip install -r requirements.txt

# Install jupyter
RUN pip3 install jupyter

# Create a new system user
RUN useradd -ms /bin/bash demo

# Change to this new user
USER demo

# Set the container working directory to the user home folder
WORKDIR /home/demo

# Start the jupyter notebook
ENTRYPOINT ["jupyter", "notebook", "--ip=0.0.0.0"]

```

You `requirements.txt` should have the list of the packages to install them thus we can have list of the modules required to be installed through pip command will be given through the `.txt` file
```
tensorflow==2.4.1
seaborn
scikit-learn 0.24.2
spacy==3.0.6
tfx==0.29.0
agate==1.6.1
asn1crypto==0.24.0
autopep8==1.3.5
Babel==2.9.1
backcall==0.2.0
bleach==3.3.0
census==0.8.17

```
This is regularly my go to list.

_After building all these_

### In the terminal run:

`docker build -t dev_ds_env .`

__Note__ `full stop` for running the environment is a must.
This should set up a jupyter notebook that you will be able to access with all the tools in the `requirements.txt` file installed. `COPy` commands will work as a magic here.

`docker run -p 8888:8888 dev_ds_env`

Now access it from your machine, try `localhost:8888`. It will ask you to copy and paste the token you were given.


Thank you all