Issue: TimeError Occurring During Training on node 16 or more
State: closed | Pull request: no
Labels: name: bug

When training on node 16 and 64, the following TimeError occurs. This issue does not arise when using Node 4 and 8.

```
Traceback (most recent call last):
  File "/opt/NeMo/examples/multimodal/multimodal_llm/neva/neva_pretrain.py", line 19, in <module>
    from nemo.collections.multimodal.models.multimodal_llm.neva.neva_model import MegatronNevaModel
  File "/opt/NeMo/nemo/collections/multimodal/models/multimodal_llm/neva/neva_model.py", line 25, in <module>
    from pytorch_lightning.trainer.trainer import Trainer
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/__init__.py", line 27, in <module>
    from pytorch_lightning.callbacks import Callback  # noqa: E402
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/callbacks/__init__.py", line 14, in <module>
    from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/callbacks/batch_size_finder.py", line 26, in <module>
    from pytorch_lightning.callbacks.callback import Callback
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/callbacks/callback.py", line 22, in <module>
    from pytorch_lightning.utilities.types import STEP_OUTPUT
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/utilities/types.py", line 41, in <module>
    from torchmetrics import Metric
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/__init__.py", line 26, in <module>
    from torchmetrics import functional  # noqa: E402
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/functional/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio._deprecated import _permutation_invariant_training as permutation_invariant_training
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/functional/audio/__init__.py", line 14, in <module>
    from torchmetrics.functional.audio.pit import permutation_invariant_training, pit_permutate
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/functional/audio/pit.py", line 22, in <module>
    from torchmetrics.utilities import rank_zero_warn
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/__init__.py", line 14, in <module>
    from torchmetrics.utilities.checks import check_forward_full_state_property
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/checks.py", line 25, in <module>
    from torchmetrics.metric import Metric
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/metric.py", line 42, in <module>
    from torchmetrics.utilities.plot import _AX_TYPE, _PLOT_OUT_TYPE, plot_single_or_multi_val
  File "/usr/local/lib/python3.10/dist-packages/torchmetrics/utilities/plot.py", line 26, in <module>
    import matplotlib.axes
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/axes/__init__.py", line 1, in <module>
    from . import _base
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/axes/_base.py", line 13, in <module>
    from matplotlib import _api, cbook, _docstring, offsetbox
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/offsetbox.py", line 33, in <module>
    import matplotlib.text as mtext
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/text.py", line 16, in <module>
    from .font_manager import FontProperties
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/font_manager.py", line 1582, in <module>
    fontManager = _load_fontmanager()
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/font_manager.py", line 1577, in _load_fontmanager
    json_dump(fm, fm_path)
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/font_manager.py", line 963, in json_dump
    with cbook._lock_path(filename), open(filename, 'w') as fh:
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/matplotlib/cbook.py", line 1826, in _lock_path
    raise TimeoutError("""\
TimeoutError: Lock error: Matplotlib failed to acquire the following lock file:
    /root/.cache/matplotlib/fontlist-v330.json.matplotlib-lock
This maybe due to another process holding this lock file.  If you are sure no
other Matplotlib process is running, remove this file and try again.
```

My code is running with 
- Megatron-LM:  from `/opt/megatron-lm` in NeMo Framework Container dev version (nvcr.io/nvidia/nemo:dev)
- NeMo: NeMo main branch (501f0dfc76886fda7f95e934de39fd8275628e2a)