NMT adding pretrained HF encoders
State: closed | Pull request: yes

This PR will add HuggingFace Encoder support to NeMo NMT.

When training NeMo NMT models, users can now: 

- Download pretrained model from HF and use it for NMT automatically
- The NMT data preprocessor will automatically use the tokenizer associated with pretrained when preprocessing the data
- The HF encoder can be used with the pretrained weights or with randomly initialized weights
- The HF encoder can be configured from scratch.

# Implementation Details
Since our ultimate goal is to be able to use encoders from HuggingFace, Megatron-LM, and NeMo, we've added some new flags to NMT.

The `library` flag takes values: `huggingface`, `megatron` (when implemented), and `nemo`.
NMT has `library=nemo` by default to maintain backwards compatibility.

The `model_name` flag is used to indicate a _named_ model architecture. From HuggingFace, we have `bert_base_cased` for example.

We've also added the `pretrained` flag. This allows users to decide whether they want to download the named model's pretrained weights `pretrained=True` or simply instantiated the named model architecture with random weights `pretrained=False`.

If a user wants to use a model architecture from a specific library, then they can use `model_name=None` and add the custom configuration under the `encoder` configuration. For example:
```
encoder:
  _target_: transformers.BertConfig
  hidden_size: 1536
```
Also note that `shared_tokenizer` should `False` when using HuggingFace encoders since we do not support HuggingFace Decoders yet.

# Usage
We've provided a config file to use with HuggingFace encoders: examples/nlp/machine_translation/conf/huggingface.yaml

We can use that config file by specifying from the CLI: 
```
--config-path=conf \
--config-name=huggingface
```
## Pretrained HuggingFace Encoder
This will train an NMT model with a pretrained encoder model from HuggingFace and
a randomly initialized decoder from NeMo.
To train the encoder from scratch, set `encoder.pretrained=False`. 
```
MODEL_NAME = bert-base-cased
python examples/nlp/machine_translation/enc_dec_nmt.py \
	--config-path=conf \
	--config-name=huggingface \
        model.src_language=en \
        model.tgt_language=de \
	model.preproc_out_dir=/raid/data/hf_en_de_preproc_${MODEL_NAME} \
	model.shared_tokenizer=false \
	model.encoder_tokenizer.library=huggingface \
	model.encoder.library=huggingface \
	model.encoder.pretrained=true \
	model.encoder.model_name=${MODEL_NAME} \
	model.decoder_tokenizer.library=yttm \
	model.decoder_tokenizer.vocab_size=32000 \
	model.decoder.library=nemo \
	model.decoder.inner_size=4096 \
	model.decoder.num_attention_heads=16 \
	model.decoder.attn_layer_dropout=0.1 \
	model.decoder.ffn_dropout=0.1 \
	model.decoder.num_layers=6 \
        ...
```
## Custom HuggingFace Encoder
This will train an NMT model with a custom encoder model from HuggingFace and
a randomly initialized decoder from NeMo.
Note: we have to use `+` from the CLI if we are not adding the argument to the config file.
```
python examples/nlp/machine_translation/enc_dec_nmt.py \
	--config-path=conf \
	--config-name=huggingface \
        model.src_language=en \
        model.tgt_language=de \
	model.preproc_out_dir=/raid/data/hf_en_de_preproc_custom \
	model.shared_tokenizer=false \
	model.encoder_tokenizer.library=huggingface \
	model.encoder.library=huggingface \ 
        +model.encoder._target_=transformers.BertConfig \
        +model.encoder.hidden_size=1536 \
	model.decoder_tokenizer.library=yttm \
	model.decoder_tokenizer.vocab_size=32000 \
	model.decoder.library=nemo \
	model.decoder.inner_size=4096 \
	model.decoder.num_attention_heads=16 \
	model.decoder.attn_layer_dropout=0.1 \
	model.decoder.ffn_dropout=0.1 \
	model.decoder.num_layers=6 \
        ...
```

This pull request **introduces 9 alerts** when merging 8abd6c6c4439cc5ce3e1e65f5dfb1950faecceaf into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-57eaca60590a915a6de70d40a43478f84286be8b)

**new alerts:**

* 9 for Unused import

This pull request **introduces 8 alerts** when merging 25c47c98b4f926d7ecde0983fd78afa298fa6262 into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-93a20e9ba45f5b25b0d1aea9927876859d0dc06b)

**new alerts:**

* 8 for Unused import

This pull request **introduces 8 alerts** when merging 451c58ede732b03459401125db3d863ee3cde7d6 into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7b59b100be33fa5ec49e9830cfd5a50e2564af38)

**new alerts:**

* 8 for Unused import

This pull request **introduces 8 alerts** when merging bccf4359ded0623538005f8306e3e43aa0ee1901 into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e64814b5067e63dd3def67107ec3ade6870ce1c3)

**new alerts:**

* 8 for Unused import

This pull request **introduces 8 alerts** when merging e497e9c9f3cca1bc222eab775384894e52892100 into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5ce35f5f9af6e8021615d26d34756d2a441f8444)

**new alerts:**

* 8 for Unused import

This pull request **introduces 8 alerts** when merging e9e24b0642ba4442d8ecc1ed232cb6d0d22e5942 into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-707978e222e3dd83f9122e0d7f8a1868a5228ccf)

**new alerts:**

* 8 for Unused import

This pull request **introduces 9 alerts** when merging ace6054f6ff8e187a3dabb6c8579ffe37994ed67 into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4c85b3a77bb89c17a75099d929758ede363b0712)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging 840d85a2be80a1bdaaaa4ce89dac9e8d66ab210e into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5448b514e9884d39d1bf7390ab6baa7cd4ee243f)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging b5548e8ac629d09238907accd078968e0eb2343c into e8912ca6e3321347272a6a7da18e052812fb2062 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bf6f33958e2a39abcf23e1556be4677f0ecfba77)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging 4ec5233841f60524fe467e0ef47efa67c38af3df into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-64b32681122ebce805dac4871d3c8ff5e86b9e20)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging 9415668530a3b8cecc6c59f8dfc32d614e8a710b into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e8cf742f834fe689da5d110aae105f285ae04b5c)

**new alerts:**

* 9 for Unused import

This pull request **introduces 9 alerts** when merging ce91bc6f760ec8d0cb5ceb6b20e77cb9b25d833b into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2d7e75c8d23771faac96c610a683be97be76f6a7)

**new alerts:**

* 9 for Unused import

This pull request **introduces 10 alerts** when merging f716c2a62119dffda6970b180aaf8f2ac9da7297 into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4dc03d91b173b523df4671949257c290a3719471)

**new alerts:**

* 10 for Unused import

This pull request **introduces 1 alert** when merging c1ffc2f4afa9944f6e144c85cb1f7bdcc0746c90 into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-c99f08d69fe3218ec44d0ee9aa9b9c7bd82b2835)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 8c66456ce66de0e9ca0c1e59b333a35f531140e2 into e816905e286df545ce344e865ea6cf8e4b0c85ca - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2c936e8bba47f81569faf00f09ed12652cb3500e)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 4fe9ec3856069c022b594cc05467f9dcde063bd6 into a33d2794cba9cff5b29044f787e41f654325424a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-2611a24d4bcd182d90cfe48ba77f8de8a98c5443)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging d0c0ad094f0d962bf46c3a0710f3d6fabdf89cd7 into a33d2794cba9cff5b29044f787e41f654325424a - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-faeb51f6d07a71183a9f2de39e302f23d2dcc555)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 2bf7978ace5bf9e1e9915e2aebd66247719e4da1 into 5d80627ca696384dc16e462f1e17efd33917a9dd - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-db259f4d4535a335027996896f845157461243e0)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging ac50ff7f38dc0b40fadd043ba89f48115432326c into 5d80627ca696384dc16e462f1e17efd33917a9dd - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1ff35793b6715f4e03900e37765fde7ff64b2152)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 6d9d547a3f2a0c46e4d17d515d5bbc3da0872a87 into 05eb2587a95e512b5e95705d9f77582634abbd95 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-da27fb011b14e210486260eec93f4e64e784c32c)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 1a8aa0967db281d5e09a9efd400936b0e2eac727 into 646aff11aa788b7d874dae930ce6ffcd658c1f16 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-083514a3c31f38a8f876cbcabe0fcdef74f98916)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging e0cca8f60c7a3946eee8bd18010029bf33384d1a into 646aff11aa788b7d874dae930ce6ffcd658c1f16 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-814ada8b11f65d71e003845db1a54f0de081bb01)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** when merging 1876f3f6f1cd743ef30df8e34e935730892652a3 into b0f9f94469b1b7bacb10fb1851f8fabfd8b4323d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-0e99d19428b41e342b29946e4215b117bcc5121e)

**new alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** and **fixes 1** when merging ff3b03ada6ec8677b8d4295fbea0d05138103fd5 into 4b8dd8ba7d02fd52e863f554b98f6be4c470948d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b5cebddab2c1ffcba131a6c0751a71c8dec470f9)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 1 for Unused import