[NMT] Model Parallel Megatron Encoders
State: closed | Pull request: yes

This PR adds Megatron encoders to NeMo NMT.

# Usage

Below is the configuration needed to use Megatron BERT 3.9B (model parallel size 4).

The `checkpoint_file` should be the path to the directory containing the standard megatron checkpoint format:
``` bash
3.9b_bert_no_rng
├── mp_rank_00
│   └── model_optim_rng.pt
├── mp_rank_01
│   └── model_optim_rng.pt
├── mp_rank_02
│   └── model_optim_rng.pt
└── mp_rank_03
    └── model_optim_rng.pt
```

``` bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python examples/nlp/machine_translation/enc_dec_nmt.py \
        --config-path=conf \
        --config-name=megatron \
        trainer.gpus=4 \
        model.preproc_out_dir=/raid/data/megatron_en_de_preproc \
        model.encoder.checkpoint_file=/path/to/megatron/checkpoint/3.9b_bert_no_rng \
        model.encoder.hidden_size=2560 \
        model.encoder.num_attention_heads=40 \
        model.encoder.num_layers=48 \
        model.encoder.max_position_embeddings=512 \
```

This pull request **introduces 4 alerts** when merging 79fb48ed35cec7f7a6c38ddac61577ec05e621f3 into 202ccc902937b10bd9a0f37d49f8b8dc29e3ca4d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-884ab3ddf6d76bf3417f94c478f3398a5fba104d)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging 2055fa3a0ff602f7dabeccfa93eac23fb1e65fc5 into 202ccc902937b10bd9a0f37d49f8b8dc29e3ca4d - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a101decf05cf5ff4bf46256156d17bdcad21b997)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging 6868d7a15521872b3efb52bd157fc3ae0946e6c2 into 3c5aace91fb8c51267a272822ef17f3123e08ec5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-667382c6e19dad10607e3da1dc0bb37273cbfe68)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging 51edfee097fd801d71edbb88935b5779a2dc41f8 into 3c5aace91fb8c51267a272822ef17f3123e08ec5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d1dde8169089c9b6399841a80b241d4fc9998c16)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging ce0c0683b1356bcf1152665e7bf576fa659afef3 into 3c5aace91fb8c51267a272822ef17f3123e08ec5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-64a0d09ecb896a84bfe26bef25b2fa8014cbfb8d)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging e729da3695990414769ad03929cbb5132102f9f5 into b94c7a17accf40e4235b9f746caaf8d9ea5d20af - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-f307a5bbcdda47c10b7cbef0d29684502a0d7ca4)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging b961cd4e667dd4355890fe078e13bae7615b0bf9 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d15b9b2837f4dc7f6f295887d571d9b26703d096)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging 4fce81e4369eeac77acb231c7c2421a1f20ce982 into bee43e840837d7e30e17a65b972e9fe89ecb9ca5 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-15e216d46eccadf897e4e61a8d804c89307c399d)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging a3a978062899bf854c0a4ad39a83ca6d09bd0233 into 638539f9c636093d3ce634b4b0602ff98e55fb12 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-4801639e1576f5a8576cfb74fd2fe4ee441d16e2)

**new alerts:**

* 4 for Unused import

This pull request **introduces 4 alerts** when merging 6e917389dde6fc1c627787a2cb726acc54fc4e7a into fbfdc1bb4a08933a494cdc9b7cd6d22bb90c32cd - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-31c3cc7164ab40fad2b5a5f3695551dbffcdea07)

**new alerts:**

* 4 for Unused import

This pull request **introduces 5 alerts** when merging 66b4c6bc459a8dae7687b3230b4fb7459ff8f978 into fbfdc1bb4a08933a494cdc9b7cd6d22bb90c32cd - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7241914f9c17e7e499a595c1155020799d3904f4)

**new alerts:**

* 5 for Unused import

This pull request **introduces 5 alerts** when merging 3bc42ae647ba2819def3a6101d89a50f7faf8b2f into 08f3c65837adfcd092ad921c050ce1628cfcb8c4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-58b9ff4ca4d3761ccf446daae8bdb98da42d2156)

**new alerts:**

* 5 for Unused import

This pull request **introduces 5 alerts** when merging 37326ea0177061a5c05f2691ec2b08388d9cb8a1 into 08f3c65837adfcd092ad921c050ce1628cfcb8c4 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-a6909c08ac8c20e08b42ba71f218701706209d65)

**new alerts:**

* 5 for Unused import

This pull request **introduces 7 alerts** when merging a4d7ef1d1cf80744347d41f606a833d235656818 into cfe2548d4db9dcc4ca393ee63604139c9f681cd8 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-5fabcd399781336942ccc8669ea8b8c5ef5fdcaf)

**new alerts:**

* 7 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging 698d2a6477edcb121ea6b409d71cdb660a7991e7 into cfe2548d4db9dcc4ca393ee63604139c9f681cd8 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-57fdaa38e337be727b4ec20eeff26959867433a3)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging 07d6b3bd23a5742cffca22cc7b37bcb570467432 into cfe2548d4db9dcc4ca393ee63604139c9f681cd8 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b24cc1f0f578628a5f20641f7a148faf21cb8daa)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging 5bead025009ec613c0adf5de8788c3ed35cd3f7d into 5f7651d0e75b0b42d2776ba0e5d95a9e7311d081 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-78f7febb2430d1b0439d3e69c1a2c61fa1d3deff)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging 4609c546ce93952512cfb31c7d8c9c3b54cd553c into 5f7651d0e75b0b42d2776ba0e5d95a9e7311d081 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-046db73326573f9d97e4e663498da77e931b99c0)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging b82466c2ee9b75f271a81e13ca9d6f43a2da2f37 into 0aa45241ac13b9a83dc017c6e1dd2d8752bb4dc6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-42b8fb4d45deb8f7e5ed2d7aa771dd2fb8a9202b)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging 8eda5ea97856ba0eee3ec02c5ae0fd9fc9470dbf into 0aa45241ac13b9a83dc017c6e1dd2d8752bb4dc6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b6a82cf5ec2f9d62ca2782adc2f4fc3d7ed3a6f5)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 4 alerts** and **fixes 1** when merging 8117073678a5cf57810858eee956128d00ab85ad into 2ce876259b52291a18fcc4fff43b448ca150d407 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-73ae5a3d6959622e152f470ede6be77ac1034c97)

**new alerts:**

* 4 for Unused import

**fixed alerts:**

* 1 for Unused import