NMT automatically preprocess data in example script
State: closed | Pull request: yes

This PR will add the ability to automatically train tokenizers and preprocess data from the NeMo config for NMT. For very large datasets it can be prohibitive to train them without first creating a tarred dataset. If the user wants to change tokenizer, vocab size, or batch size, for example, they must reprocess the data with the correct configuration. With this PR, users can sweep through data configurations and the tarred dataset will be automatically created according to the model configuration.

The tarred dataset creation code is taken directly from https://github.com/NVIDIA/NeMo/blob/main/examples/nlp/machine_translation/create_tarred_parallel_dataset.py

# Automatic tokenizer training
If user specifies that they want to use the 'yttm' tokenizer but does not specify a trained tokenizer model, then the tokenizer model will automatically be trained and saved to `preproc_out_dir`. The config will then be updated with the newly trained tokenizer model. 

# Automatic data preprocessing
This PR changes the translation dataset configuration slightly. `load_from_tarred_dataset` has been removed. Now, if users want to use tarred datasets (recommended) then they should specify `use_tarred_dataset=True`. If `use_tarred_dataset` is `True`, then we will check for `train_ds.tar_files`. If `train_ds.tar_files` is `None` or does not exists, then we will automatically create the tar files and save them to `preproc_out_dir`.

Note: if user specifies `train_ds.tar_files` then those will be used and no data preprocessing will happen.

Note: if user does not specify `train_ds.tar_files` but the tarred dataset has already been created, then the existing dataset will be used. This would happen if the training script were run multiple times w/o modifying relevant dataset preprocessing configuration.

# Large Dataset Workflow
For large datasets it can be useful to process them in a different environment than the training environment.
In this case, or in any situation where the user wants to process the data but not train at the same time, 
they can set `do_training=False` and only the data will be preprocessed.

# Usage

## Automatically train tokenizer and preprocess data
User only specifies src and tgt data for training.
```
python /home/eharper/github/NeMo/examples/nlp/machine_translation/enc_dec_nmt.py -cn aayn_base \
    model.encoder_tokenizer.vocab_size=32000 \
    model.decoder_tokenizer.vocab_size=32000 \
    +model.encoder_tokenizer.bpe_dropout=0.1 \
    +model.decoder_tokenizer.bpe_dropout=0.1 \
    model.preproc_out_dir=/raid/data/en_es_preproc \
    model.train_ds.src_file_name=/raid/data/72241/parallel/train.clean.en-es.60.tok.en.shuffled \
    model.train_ds.tgt_file_name=/raid/data/72241/parallel/train.clean.en-es.60.tok.es.shuffled \
    model.train_ds.tokens_in_batch=16000 \
    model.train_ds.use_tarred_dataset=true \
    ...
```
## Use existing tokenizer model and tarred dataset
```
python /home/eharper/github/NeMo/examples/nlp/machine_translation/enc_dec_nmt.py -cn aayn_base \
    model.encoder_tokenizer.tokenizer_model=/raid/data/en_es_preproc/shared_tokenizer.32000.BPE.model \
    model.decoder_tokenizer.tokenizer_model=/raid/data/en_es_preproc/shared_tokenizer.32000.BPE.model \
    model.encoder_tokenizer.vocab_size=32000 \
    model.decoder_tokenizer.vocab_size=32000 \
    +model.encoder_tokenizer.bpe_dropout=0.1 \
    +model.decoder_tokenizer.bpe_dropout=0.1 \
    model.train_ds.tokens_in_batch=16000 \
    model.train_ds.use_tarred_dataset=true \
    model.train_ds.tar_files=/raid/data/en_es_preproc/batches.tokens.16000._OP_1..1000_CL_.tar \
    model.train_ds.metadata_file=/raid/data/en_es_preproc/metadata.tokens.16000.json \
    model.validation_ds.src_file_name=/raid/data/72241/parallel/newstest2012-en-es.clean.tok.src \
    model.validation_ds.tgt_file_name=/raid/data/72241/parallel/newstest2012-en-es.clean.tok.ref \
    ...
```
## Process data from text files during dataset creation
Should only be used for rapid prototyping with small datasets.
```
python examples/nlp/machine_translation/enc_dec_nmt.py -cn aayn_base \
	model.preproc_out_dir=/raid/data/toy_wmt_preproc \
	model.train_ds.src_file_name=/raid/data/toy_wmt/toy_data/wmt14-de-en.src \
	model.train_ds.tgt_file_name=/raid/data/toy_wmt/toy_data/wmt14-de-en.ref \
	model.validation_ds.src_file_name=/raid/data/toy_wmt/toy_data/val/wmt14-de-en.src \
	model.validation_ds.tgt_file_name=/raid/data/toy_wmt/toy_data/val/wmt14-de-en.src \
	model.test_ds.src_file_name=/raid/data/toy_wmt/toy_data/val/wmt14-de-en.src \
	model.test_ds.tgt_file_name=/raid/data/toy_wmt/toy_data/val/wmt14-de-en.src \
	model.encoder_tokenizer.vocab_size=2000 \
	model.decoder_tokenizer.vocab_size=2000 \
        ...
```
# NMT Data Preprocessing TODOs
This should be done but not necessarily in this PR.
- Manifest file should include everything needed for the tarred dataest. Config should use manifest with first priority.
- Multiprocessing should be added to the tarred dataset creation. 

This pull request **introduces 7 alerts** and **fixes 1** when merging 39305db0bbb9b46913d7af27263078f7a3e0d051 into 3c288d8a7caf667c95444c39434e3ebc5f53d911 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9d19b73f6a2e17f867cc19d0bc33bfa1fed8951a)

**new alerts:**

* 4 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 7 alerts** and **fixes 1** when merging c8771cd94d853f0e95bb10837a8151c74aefb017 into 3c288d8a7caf667c95444c39434e3ebc5f53d911 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-21eb996b3547d2da4c9db33b8b98c3e4e47b40c3)

**new alerts:**

* 4 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 7 alerts** and **fixes 1** when merging ee310bde853aa5bde392fa700bbe7c2d859a66c6 into 3c288d8a7caf667c95444c39434e3ebc5f53d911 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-af39994aab0a30451f880e37c982b9d02cfcb99e)

**new alerts:**

* 4 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 7 alerts** and **fixes 1** when merging 415d89b4bd54c104728493471adf5d6a749d4774 into dacdb1ad4286731f50fa33c0a74b04089d373f44 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-787da5da4a13eb3e92830b36c7d672d8ea97f7ed)

**new alerts:**

* 4 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 7 alerts** and **fixes 1** when merging e58ce898f08bd6a05791f2c06a25e0fbfe975608 into dacdb1ad4286731f50fa33c0a74b04089d373f44 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e03037e6d8fdca6358617d98f91d19b1c291c1a9)

**new alerts:**

* 4 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 7 alerts** and **fixes 1** when merging ea2059bdd67e734aacd14a7d8cbb4b81841ed80a into 85224dbdc40af41be5f68c2b60004073c91d9a91 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d4a50594e7480874d6701c8ae0a48e05ba857d61)

**new alerts:**

* 4 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 8 alerts** and **fixes 1** when merging 3af0fbf020445f9bc0f7dc26b1a2b44fc3fb8632 into 85224dbdc40af41be5f68c2b60004073c91d9a91 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-07995c215b3500c2750daa2c065604cacfbecbab)

**new alerts:**

* 5 for Unused import
* 3 for NotImplemented is not an Exception

**fixed alerts:**

* 1 for NotImplemented is not an Exception

This pull request **introduces 1 alert** and **fixes 2** when merging 007cdb97bf3e14feac42b696e120bbd4430b0099 into 85224dbdc40af41be5f68c2b60004073c91d9a91 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e287075de9ec0672b66629e0f7ef9b34d7dacdd2)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **introduces 1 alert** and **fixes 2** when merging 7c2a531b871532e71bfe08e1273a5e78930c9b8e into 4518b0030485bdd0634831a957cd6f3dd5c52f60 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0b459ac05528cebb1c9eb47566536d9743d75959)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **introduces 1 alert** and **fixes 2** when merging be18efa852e5f86ca9fe110db733213129ec50a3 into fc4017a08db4cf2aaad2288dd158e9c287f7107a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e9a3285512df8ba91cb05fb5406557007d0fc3a1)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **introduces 1 alert** and **fixes 2** when merging a120d6f76a06a5e8b54dd8ad14a2412367c8552f into fc4017a08db4cf2aaad2288dd158e9c287f7107a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-110ce275b7fb2de6c634a9a7de0df585f9f1db06)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This PR is good, but @ericharper please add tests (in this or subsequent PRs) for this workflow with some arbitrary small random data - 
1) from scratch tokenize, tar, train flow
2) restore model - in different env than original env
3) setup train tokekizer, etc after restore c

NMT side has woefully few tests for tarred dataset and model checkpoint testing in general 

This pull request **introduces 1 alert** and **fixes 2** when merging 1e22f113606a7fc007511dd39551484584cac6e4 into fc4017a08db4cf2aaad2288dd158e9c287f7107a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6d860e01bf512f9daaac61759de758320115eb91)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **introduces 1 alert** and **fixes 2** when merging 7a2095807c916c54c224f3320521cb209c53df75 into fc4017a08db4cf2aaad2288dd158e9c287f7107a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f820a3d19835b685676282308a050280697d5c6b)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **fixes 2 alerts** when merging 972ffeb15e5e297b955051e6f4e0d1bed950780c into fc4017a08db4cf2aaad2288dd158e9c287f7107a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-8df142ef1715cd2747ce6ce31a690804e0cbea40)

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **fixes 2 alerts** when merging 10a45a12ea8fdd28e475f251c61a44f0e265c7a2 into fc4017a08db4cf2aaad2288dd158e9c287f7107a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-360464a32a3857a4e406632ca47503627f5d168d)

**fixed alerts:**

* 2 for NotImplemented is not an Exception

This pull request **fixes 2 alerts** when merging bee4236db27a78cf7ace6b38e3e6fc57e346338d into e6439786239044e3c4b803b296b8013feb142772 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5bf19714013d312906024d42216a831fbf06f665)

**fixed alerts:**

* 2 for NotImplemented is not an Exception