Problems running multi-gpu punctuation capitalization training 
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Running multi-gpu training without a pre-prepared cache, crashes upon initialization with the following trace
```python3
Traceback (most recent call last):
  File "examples/nlp/token_classification/punctuation_capitalization_train_evaluate.py", line 155, in <module>
    main()
  File "/workspace/nemo/nemo/core/config/hydra_runner.py", line 104, in wrapper
    _run_hydra(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 377, in _run_hydra
    run_and_report(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py", line 378, in <lambda>
    lambda: hydra.run(
  File "/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "examples/nlp/token_classification/punctuation_capitalization_train_evaluate.py", line 116, in main
    model = PunctuationCapitalizationModel(cfg.model, trainer=trainer)
  File "/workspace/nemo/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py", line 101, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/workspace/nemo/nemo/collections/nlp/models/nlp_model.py", line 98, in __init__
    super().__init__(cfg, trainer)
  File "/workspace/nemo/nemo/core/classes/modelPT.py", line 138, in __init__
    self.setup_training_data(self._cfg.train_ds)
  File "/workspace/nemo/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py", line 469, in setup_training_data
    self._train_dl = self._setup_dataloader_from_config(cfg=train_data_config, train=True)
  File "/workspace/nemo/nemo/collections/nlp/models/token_classification/punctuation_capitalization_model.py", line 773, in _setup_dataloader_from_config
    dataset = BertPunctuationCapitalizationDataset(
  File "/workspace/nemo/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py", line 993, in __init__
    features = pickle.load(self.features_pkl.open('rb'))
  File "/opt/conda/lib/python3.8/pathlib.py", line 1222, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/opt/conda/lib/python3.8/pathlib.py", line 1078, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/data/punct_v2/cached.text_train.BertTokenizer.max_seq_length512.vocab119547.all_samples.punctuation_capitalization.pkl'
```

The issue arises because, the order of starting the individual processes, does not guarantee that the process with `global_rank()=0`  will start first. There seems to be a guard 
https://github.com/NVIDIA/NeMo/blob/f9d45db36afae8d75aecc27175a854d289bffd84/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py#L984-L985
that should block the other processes, but interestingly `torch.distributed.is_initialized()` returns `False` so the guard is skipped.

Adding the parameter `model.train_ds.use_cache=false`, does not help, since the cache is loaded irrespective of the value of this parameter https://github.com/NVIDIA/NeMo/blob/f9d45db36afae8d75aecc27175a854d289bffd84/nemo/collections/nlp/data/token_classification/punctuation_capitalization_dataset.py#L987-L988

**Steps/Code to reproduce bug**

Train using multiple GPUs without a pre-prepared cache. 

**Expected behavior**

All processes except the one with `global_rank()=0` to wait for the master to finish preparing cache, and the training start afterwords.

**Environment overview (please complete the following information)**

 - Environment location: SLURM
 - Method of NeMo install: `pytorch:22.04-py3` container with `nemo:1.8.2` (`./reinstal.sh` without numba update).