Multilingual training for NMT
State: closed | Pull request: yes

This PR adds support for multilingual training for NMT.

### Usage ###
To train a model in multilingual mode, simply set the following:

* `model.multilingual=True`
* `model.encoder_tokenizer.library=sentencepiece` and `model.decoder_tokenizer.library=sentencepiece`. 
   Currently we only support sentencepiece for multilingual NMT.
* `model.src_language=['lang1','lang2']` or `model.tgt_language=['lang1','lang2']`
   This PR only supports `many-to-one` or `one-to-many` model, as a result, only either of the above can be a list.
* `model.train_ds.metadata_file = [a,b]` if using tarred dataset
    or
   `model.train_ds.src_file_name = [a.src,b.src]` and `model.train_ds.tgt_file_name = [a.ref,b.ref]`
* MultiValidation from  #2113 can be used to provide val sets from different languages

NOTE: All the lists in the configuration, including src/tgt_languages, train_set, val_set need to be in the same order. 
That is,
if `model.tgt_language=['de','fr']`, the train data should also be provided as `['de_train_data','fr_train_data']`.


### Additional Info ###
Under the hood, the model automatically adds language identifier tokens which are created as `control_symbols` in sentencepiece. 
So for instance, if `model.tgt_language=['de','fr']`, `<de>` is prepended to all German source sentences and `<fr>` is prepended to all French source sentences.

### Edit ###
The PR also does the following changes to the current `TarredTranslationDataset`:
* Adjust `__len__` of the dataset based on the world size when shard strategy is set to `scatter`.
* Change naming convention of fragment pkl files (during tar file creation in `preproc_mt_data`) to make them globally unique in the dataset.
* Remove `load_from_cached_dataset`.
* Small fix for a variable name and file ptr when only tar metadata is passed.


This pull request **introduces 3 alerts** when merging d75335ceb01cff0fcd27dbfdc171669389f6cc15 into 74f1264e049c10b012a1ea483fcd468f4d89ecf8 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6b9dd3e2f2dbe721fa62dd144bab1d74313eb4c8)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import
* 1 for Unguarded next in generator

Overall this is looking really nice. 

Could you help to explain further how we're getting multi-lingual models (many to one and one to many) by adding multiple datasets and a language identifier token?

> Overall this is looking really nice.
> 
> Could you help to explain further how we're getting multi-lingual models (many to one and one to many) by adding multiple datasets and a language identifier token?

Currently, we are only supporting a full-parameter sharing multilingual model. As a result, we use the same MTEncDecModel and feed it data from multiple languages. A multilingual model just ingests data from multiple languages and learns to translate it (much like the standard translation model). Though, in cases like one-to-many, it may be unclear for the model as to what language the user wants to translate to. In order to solve this, we use a language identifier token that guides the translation into the correct language. So in the case of en->de/fr, we add the language identifier (`<de>` or `<fr>`) token to the beginning of every English source sentence to tell the model which language the output is going to be in. 

In the future, if we want to create models that allow selective parameter sharing, we can simply create a new model class that accepts configurations from the user telling which parts of the model to share. So we can have the same decoders but different encoder, shared self-attention layers but different FFNs, so on and so forth. The data, in that case, can still be sampled in the same way it is being done currently.

This pull request **introduces 1 alert** when merging bbdd886e106b1d2e4b6988d6cad062f09f8873df into b44a1077df4579ecdba9a5386fbfe2c0bcd20667 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-661c90ded048b01baf0895e4f4e3f8c5ac28cc4d)

**new alerts:**

* 1 for Explicit export is not defined

This pull request **introduces 2 alerts** when merging a6355f19d2dbcf79656d8cd05a8b6ccbededbbbe into 046fad0f8fa72ce67d0f7718a048f3be82526402 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-69808af1ef37317492d9d69dd00b416762b99df3)

**new alerts:**

* 2 for Unused local variable

The previous few commits bring the following changes to the PR:
* Fix `TarredTranslationDataset` length. Earlier, this length was not reduced based on the world size
* MultiGPU / Multi-worker fixes (especially when provided datasets are of kind `map`)
* Fixes for changes requested earlier (like the addition of `ConcatTranslationDataset` params to config)

This pull request **introduces 1 alert** when merging edde60f444fb3d22b35f5c0789b50b310c568686 into 6d53452eb1487102c0d5ae657815ec2aecc12076 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9511778dc8a9b50737d0f0148316f4f0d1eabb83)

**new alerts:**

* 1 for Unused import