Repository: mlcommons/inference
Issue: DLRM: save downloaded and generated files outside of Docker containers
Root cause: Architectural Constraint

The [CPU](https://github.com/mlperf/inference/blob/master/v0.5/recommendation/Dockerfile.cpu) and [GPU](https://github.com/mlperf/inference/blob/master/v0.5/recommendation/Dockerfile.gpu) Docker images are pretty bare bones. One needs to launch a container, and then run the corresponding `kickstart.sh` to clone repos, download a model, generate fake data, etc. If the container gets killed, one needs to start from scratch. Also, two different containers do not share the data. I believe this can be improved.

Suppose I want to experiment under `WORKSPACE=/datasets/mlperf-inference-v0.7` by repeating commands from either `kickstart.sh` script.

### Clone the training and inference repos
```bash
$ cd $WORKSPACE
$ git clone --progress --recurse-submodules https://github.com/mlperf/training.git
$ git clone --progress --recurse-submodules https://github.com/mlperf/inference.git
$ export DLRM_DIR=$WORKSPACE/training/recommendation/dlrm
```

### Download the ~10GB model
```bash
$ cd $WORKSPACE/inference/v0.5/recommendation
$ mkdir ./model
$ curl https://dlrm.s3-us-west-1.amazonaws.com/models/tb0875_10M.pt --output model/dlrm_terabyte.pytorch
$ export MODEL_DIR=$WORKSPACE/inference/v0.5/recommendation/model
```

Optionally, verify the MD5 hash:
```bash
$ md5sum $MODEL_DIR/dlrm_terabyte.pytorch
b7cacffcf75f767faa9cb2af397723aa  dlrm_terabyte.pytorch
```

### Generate fake data
```bash
$ cd $WORKSPACE/inference/v0.5/recommendation
$ python -m pip install numpy --user
$ cd tools/
$ ./make_fake_criteo.sh terabyte0875
$ mv fake_criteo/ ../
$ export DATA_DIR=$WORKSPACE/inference/v0.5/recommendation/fake_criteo
```
It's worth noting several things:
- `make_fake_criteo.sh` calls `quickgen.py` which is also in the `tools/` directory; hence, the need to descend there first;
- `quickgen.py` needs NumPy;
- the generated data is in `fake_criteo/`, not in [`fake_data/`](https://github.com/mlperf/inference/blob/db0b7eb368f009adf703d073765aa212d1f3a026/v0.5/recommendation/docker_cpu/kickstart.sh#L19) (maybe only for `terabyte0875`?)

Now, I can build the CPU and GPU Docker images:
```
$ cd $WORKSPACE/inference/v0.5/recommendation
$ docker build -t dlrm-cpu docker_cpu/.
$ docker build -t dlrm-gpu docker_gpu/.
```
and launch either of them mapping the inference and training repos as follows:
```bash
$ docker run -it \
-v $WORKSPACE/inference:/root/inference \
-v $WORKSPACE/training:/root/training \
dlrm-cpu

root@8042646de2b5:~# ls -la
total 44
drwx------  1 root root 4096 May 21 00:06 .
drwxr-xr-x  1 root root 4096 May 21 00:06 ..
-rw-r--r--  1 root root 3166 May 20 18:42 .bashrc
drwxr-xr-x  1 root root 4096 May 20 18:45 .cache
drwxrwsr-x  2 root root 4096 May 20 18:45 .conda
-rw-r--r--  1 root root   34 May 20 18:42 .condarc
drwxr-xr-x  2 root root 4096 May 20 18:45 .empty
-rw-r--r--  1 root root  148 Aug 17  2015 .profile
drwxrwxr-x 11 1000 1001 4096 Apr 30 09:23 inference
-rwxrwxr-x  1 root root  978 May 20 09:14 kickstart.sh
drwxrwxr-x 17 1000 1001 4096 May 20 23:31 training
```
Finally, do a bunch of straightforward `export`s as above:
```bash
root@8042646de2b5:~# export WORKSPACE=/root
root@8042646de2b5:~# export DLRM_DIR=$WORKSPACE/training/recommendation/dlrm
root@8042646de2b5:~# export MODEL_DIR=$WORKSPACE/inference/v0.5/recommendation/model
root@8042646de2b5:~# export DATA_DIR=$WORKSPACE/inference/v0.5/recommendation/fake_criteo
```

Apologies if I've got at this at the wrong end. Maybe I was only supposed to do:
```
$ export DLRM_DIR=$WORKSPACE/training/recommendation/dlrm
$ export DATA_DIR=$WORKSPACE/inference/v0.5/recommendation/fake_criteo
$ export MODEL_DIR=$WORKSPACE/inference/v0.5/recommendation/model
$ cd $WORKSPACE/inference/v0.5/recommendation
$ ./run_and_time.sh pytorch dlrm terabyte0875 gpu
```
But then this happens:
```                                                                            
mkdir: cannot create directory ‘/datasets/mlperf-inference-v0.7/inference/v0.5/recommendation/output/pytorch-gpu’: Permission denied
Sending build context to Docker daemon   28.9GB.6GB
Step 1/17 : FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu16.04
 ---> 7bb1f0b039e1
Step 2/17 : ENV PYTHON_VERSION=3.7
 ---> Using cache
...
Removing intermediate container dbbc6c71e94a
 ---> 6841183ce1be
Step 17/17 : ENTRYPOINT ["/bin/bash"]
 ---> Running in d2917a8ed2a3
Removing intermediate container d2917a8ed2a3
 ---> 4484c60feca9
Successfully built 4484c60feca9
Successfully tagged mlperf-infer-recommendation-gpu:latest
tee: /datasets/mlperf-inference-v0.7/inference/v0.5/recommendation/output/pytorch-gpu/dlrm/output.txt: No such file or directory
docker: invalid reference format.
See 'docker run --help'.
```
- `mkdir: cannot create directory ‘/datasets/mlperf-inference-v0.7/inference/v0.5/recommendation/output/pytorch-gpu’: Permission denied`: Maybe should first run `mkdir -p output` and then `mkdir -p output/pytorch-gpu`?
- `Sending build context to Docker daemon   28.9GB.6GB`: Seems to happen on every run?
- `tee: /datasets/mlperf-inference-v0.7/inference/v0.5/recommendation/output/pytorch-gpu/dlrm/output.txt: No such file or directory`: another case for a chain of `mkdir` commands rather than a single one?

Even when I create `$WORKSPACE/inference/v0.5/recommendation/output/pytorch-gpu/dlrm/` by hand, I get:
```
$ cat output/pytorch-gpu/dlrm/output.txt
docker: invalid reference format.
See 'docker run --help'.
```

Is this still an issue @psyhtest? you may want to tag Maxim. Otherwise please close this issue?