NMT add Multiprocessing to Data Preprocessing
State: closed | Pull request: yes

Tokenizing data and saving it to a tarred dataset is an embarrassingly parallel task.
We parallelize the data processing by partioning the src and tgt text lines into chunks. 
Each chunk is then written to a tarred dataset in parallel using the joblib library.

Also note that tarred datasets are to be used when the dataset is too large to be handled within system memory so
we made sure that our approach was also memory friendly. The memory consumed while multiprocessing depends 
only on the number of jobs and the chunk size (n_jobs * chunk_size).

# Results
We tested the end-to-end time of automatic data processing for NMT (do_training=False) with multiprocessing for two language pairs. We used a chunk size of 1M lines.

En-De used a parallel training set with 4524868 lines (~4.5M):
```
n_preproc_jobs: 4, elapsed seconds: 122
n_preproc_jobs: 2, elapsed seconds: 207
n_preproc_jobs: 1, elapsed seconds: 373
```
En-Es used a parallel training set with 65630789 lines (~65M):
```
n_preproc_jobs: 64, elapsed seconds: 514
n_preproc_jobs: 32, elapsed seconds: 602
n_preproc_jobs: 16, elapsed seconds: 837
n_preproc_jobs: 8, elapsed seconds: 1326
n_preproc_jobs: 1, elapsed seconds: 8168
```
As we can see, for larger datasets the speedup can save a lot of time! ~15x speed up or 2.2 hours vs 8.5 minutes.
Since we have to preprocess our dataset for different tokenizers, vocabulary sizes, and batch sizes, it's process that we end up doing quite a lot when experimenting and it can be very useful to save a couple of hours per experiment.

# Usage
Multi-processing will now happen automatically. By default NMT will attempt to use every process available (except for 2):
```
train_ds.n_preproc_jobs=-2
```
Not every process will necessarily be used as there has to be at least one chunk per process. To configure multi-processing manually simply set:
```
train_ds.n_preproc_jobs=4
```
# TODO
- [x] Testing for backwards compatibility and convergence.
- [x] General clean up
- [x] Add jenkins tests
- [x] Update script version

This pull request **introduces 4 alerts** when merging 146952ca71c98f2274c49f2472a080f786157e45 into d79d5d3ed06d86d24b82214363755174313d6d9d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f5bf2e88dcb9955a61864b2cba2d1fc2e6b13b3b)

**new alerts:**

* 2 for Variable defined multiple times
* 1 for Wrong name for an argument in a call
* 1 for Unused import

This pull request **introduces 2 alerts** when merging 7b13704b3c80a645de62d7d8fd4deabd644ad0d8 into d79d5d3ed06d86d24b82214363755174313d6d9d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2d3f41de390ac4045d4332e16f5fe044c0ca63af)

**new alerts:**

* 1 for Wrong name for an argument in a call
* 1 for Unused import

This pull request **introduces 1 alert** when merging 9743c99d762b95fa230b8c047fda37c9d10b8d7a into d79d5d3ed06d86d24b82214363755174313d6d9d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0a3215727fef1e5e725e6276711ec435e66f42d1)

**new alerts:**

* 1 for Wrong name for an argument in a call

This pull request **introduces 1 alert** when merging fca0fc66c678005af01de5970b44afd3a64c499f into 480bbc39b8c35f2f5ff0e2a2b61f4e2526de28eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-dd7a2c4804b8bbe025bce68e9d2571d5833d00e6)

**new alerts:**

* 1 for Wrong name for an argument in a call

This pull request **introduces 1 alert** when merging cdc88bc431d617643671602e4ec33dd2ce68421c into 480bbc39b8c35f2f5ff0e2a2b61f4e2526de28eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-3e6422c6ea4187dd1b86d9b399253ff4109ee7e1)

**new alerts:**

* 1 for Wrong name for an argument in a call

This pull request **introduces 1 alert** when merging 842012cb6f2bb7988a5035fad7e98ab5c8b54ec6 into 480bbc39b8c35f2f5ff0e2a2b61f4e2526de28eb - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-38a4773fbc26e6c792d7ee14e9370cc2eae2f5fd)

**new alerts:**

* 1 for Wrong name for an argument in a call