Megatron model parallel autoresume
State: closed | Pull request: yes

Adds training auto-resume for NLP models that use model parallel megatron-lm. The existing autoresume logic depends on there being only one checkpoint, this is not the case for model parallel models. There is a checkpoint for each model parallel rank.

We were able to add this feature with minimal changes to exp_manager. The only functional change is to log a warning rather than throw an error when exp_manager detects multiple `last` checkpoints. We also use the exp_manager to set some properties in `AppState` that we need in `NLPModel`.

# Usage
```
python examples/nlp/text_classification/text_classification_with_bert.py \
	--config-path=${CONFIG_PATH} \
	--config-name=${CONFIG_NAME} \
        ...
	exp_manager.exp_dir=${EXP_DIR}/nemo_experiments \
	exp_manager.create_checkpoint_callback=true \
	+exp_manager.create_wandb_logger=true \
	+exp_manager.wandb_logger_kwargs.name=${EXP_NAME} \
	+exp_manager.wandb_logger_kwargs.project=${PROJECT_NAME} \
	exp_manager.name=${EXP_NAME} \
	+exp_manager.resume_if_exists=true \
	+exp_manager.resume_ignore_no_checkpoint=true \
	+exp_manager.version=0 \
	+exp_manager.use_datetime_version=false \
```


This pull request **introduces 3 alerts** when merging 3dd3fe5c85e29181cf3e7d49bc36e456105d42c1 into 297a8bdbb0370a51cb74a41288025befd9cd2392 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-98ed3610935c7e6e69dbd68cfdbb8d9904f289b7)

**new alerts:**

* 2 for Unused import
* 1 for Except block handles &#39;BaseException&#39;

This pull request **fixes 2 alerts** when merging ff8734a172413bd7c4c9cca96a61043f79af907f into 9cc1d68444b748be950f0ae0b4e4c7701320bb71 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1e0b3f4360b3499ef9ed329aaba19fb125d66e23)

**fixed alerts:**

* 2 for Unused import

This pull request **fixes 1 alert** when merging 5d17e572624d6bca5380725f665afbf0a5f39068 into 9cc1d68444b748be950f0ae0b4e4c7701320bb71 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d9d75b4748659f6ed33d59a2335d9ff847fce707)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging b133b40dfc16242b2f7e8a71c0769cc7e55d9f12 into 3562fddd45ae5571616fbc35c9252cb1bf19ab05 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-083584d542a5b76d35514d028ce6b7157875b357)

**fixed alerts:**

* 1 for Unused import

This pull request **fixes 1 alert** when merging 374548c903edcee3d2f8b2443f0f0d00ac4c6615 into 3562fddd45ae5571616fbc35c9252cb1bf19ab05 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-55183500bbbe54c781a90fdf70a225d0432c20e8)

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** and **fixes 1** when merging d36ef70fb398b3cf351ab1e650f8116b2a2b4d86 into 278e85b7b33d841b08133c30e8cf40b2b0dcdc18 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ecf435074e38fdce89694f2e9f2ac95bb1d48566)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 1 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 33eda0e2d1682c42c9aa779fa1e64d8cdc786c52 into e10bb0348daff62348f84bdfb1845b1226d74f11 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b24d2ac8315a4fec3b8007f09ddc33fd1079ef99)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 17c5cbec94e77551ceaaf7eb52aa3bcab573a8bc into 08147dfb05f1622ec6809320a938b57c843de735 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ac57b2015a11a84dfcbc0a7e3a6917924e8bde58)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 2bd23ef9e58810ab2482bd8daf342ab2b7b54cdc into 630c7e81b527ee42ab3e65319aadb4a941fb5aab - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-bf1465d164a2432d0061500ae615700aaae1a4c5)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 9ce7a90de3d907dca3553cab805f8ed493a6f889 into 8a26496f415386f50adcead21ebd1e4a21a7ff42 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-10da98049da326b821085f2d23c8697aabb93f5b)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 034107251524c43ba1d549e42267080ed73b1e11 into 8ccb4f3349de7603f7607dbd74de61795a5f5f68 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-e31103cecc12db8125923b30770d00508cd514fe)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import

This pull request **introduces 1 alert** and **fixes 2** when merging 06d5192da1b2c36d63fa0f2fdb2dcdd0991178f1 into 8ccb4f3349de7603f7607dbd74de61795a5f5f68 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7e1542f9b56ef850149a5581c19ad2853bf35865)

**new alerts:**

* 1 for Unused import

**fixed alerts:**

* 2 for Unused import