Add disable/enable forward pre hook for DDP and overlap param gather
State: closed | Pull request: yes
Labels: name: skip-docs

This is currently called when saving checkpoint as follows:
```
if distributed_optimizer and overlap_param_gather:
    disable_forward_pre_hook
...
save_checkpoint
...
if distributed_optimizer and overlap_param_gather:
    enable_forward_pre_hook
```

<!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.lightning.megatron_parallel
nemo/lightning/megatron_parallel.py:238:0: C0301: Line too long (127/119) (line-too-long)
nemo/lightning/megatron_parallel.py:239:0: C0301: Line too long (140/119) (line-too-long)
nemo/lightning/megatron_parallel.py:240:0: C0301: Line too long (130/119) (line-too-long)
nemo/lightning/megatron_parallel.py:546:0: C0301: Line too long (129/119) (line-too-long)
nemo/lightning/megatron_parallel.py:553:0: C0301: Line too long (135/119) (line-too-long)
nemo/lightning/megatron_parallel.py:825:0: C0301: Line too long (137/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1055:0: C0301: Line too long (136/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1622:0: C0301: Line too long (128/119) (line-too-long)
nemo/lightning/megatron_parallel.py:1661:0: C0301: Line too long (146/119) (line-too-long)
nemo/lightning/megatron_parallel.py:64:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:65:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:67:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:102:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:106:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:306:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:330:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:356:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:382:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:518:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:561:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:565:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:621:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:656:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:662:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:669:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:676:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:710:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:718:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:734:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:761:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:773:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:795:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1315:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1490:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1496:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1502:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1506:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1511:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1516:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1544:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1590:8: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1612:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1685:0: C0115: Missing class docstring (missing-class-docstring)
nemo/lightning/megatron_parallel.py:1724:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/lightning/megatron_parallel.py:1738:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.55/10
```

---

Thank you for improving NeMo's documentation!