When calling the "trainer.fit" it crashes in "initializing ddp"
State: closed | Pull request: no
Labels: name: bug

Hi,

I was able to run the https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/asr/10_ASR_CTC_Language_Finetuning.ipynb script in Colab with no problems. However, due to disk and GPU limitations, I took this script and ran it in my environment using JupyterLab.
When starting the training, exactly in the part `trainer.fit(char_model)` It crashes in the message:
**`"initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2"`**

The only change made to the code is in this part, where I set that I want to use 2 GPUs and the ddp accelerator:
```
if torch.cuda.is_available():
  gpus = 2
else:
  gpus = 0

EPOCHS = 50 
trainer = ptl.Trainer(gpus=gpus,
                      max_epochs=EPOCHS, 
                      accumulate_grad_batches=1,
                      checkpoint_callback=False,
                      logger=False,
                      log_every_n_steps=5,
                      check_val_every_n_epoch=1,
                      accelerator="ddp")
```

**lightning_logs.txt**
```
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
```
**nemo_log_globalrank-0_localrank-0.txt**
```
[NeMo W 2021-06-17 18:46:12 nemo_logging:349] /opt/conda/lib/python3.6/site-packages/pydub/utils.py:170: RuntimeWarning: Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work
      warn("Couldn't find ffmpeg or avconv - defaulting to ffmpeg, but may not work", RuntimeWarning)
    
[NeMo W 2021-06-17 18:46:15 experimental:28] Module <class 'nemo.collections.asr.data.audio_to_text_dali.AudioToCharDALIDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2021-06-17 18:47:06 cloud:56] Found existing object /root/.cache/torch/NeMo/NeMo_1.0.2/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo.
[NeMo I 2021-06-17 18:47:06 cloud:62] Re-using file from: /root/.cache/torch/NeMo/NeMo_1.0.2/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo
[NeMo I 2021-06-17 18:47:06 common:675] Instantiating model from pre-trained checkpoint
[NeMo W 2021-06-17 18:47:08 modelPT:138] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    manifest_filepath: /data2/voices/train_1k.json
    sample_rate: 16000
    labels:
    - ' '
    - a
    - b
    - c
    - d
    - e
    - f
    - g
    - h
    - i
    - j
    - k
    - l
    - m
    - 'n'
    - o
    - p
    - q
    - r
    - s
    - t
    - u
    - v
    - w
    - x
    - 'y'
    - z
    - ''''
    batch_size: 32
    trim_silence: true
    max_duration: 16.7
    shuffle: true
    is_tarred: false
    tarred_audio_filepaths: /asr_set_1.2/train/train_{0..1023}.tar
    num_workers: 20
    
[NeMo W 2021-06-17 18:47:08 modelPT:145] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    manifest_filepath: /data2/voices/train_1k_samp.json
    sample_rate: 16000
    labels:
    - ' '
    - a
    - b
    - c
    - d
    - e
    - f
    - g
    - h
    - i
    - j
    - k
    - l
    - m
    - 'n'
    - o
    - p
    - q
    - r
    - s
    - t
    - u
    - v
    - w
    - x
    - 'y'
    - z
    - ''''
    batch_size: 32
    shuffle: false
    
[NeMo I 2021-06-17 18:47:08 features:252] PADDING: 16
[NeMo I 2021-06-17 18:47:08 features:269] STFT using torch
[NeMo I 2021-06-17 18:47:09 modelPT:434] Model EncDecCTCModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.0.2/stt_en_quartznet15x5/16661021d16e679bdfd97a2a03944c49/stt_en_quartznet15x5.nemo.
[NeMo I 2021-06-17 18:47:09 ctc_models:333] Changed decoder to output to ['i', 'e', 'c', 'a', 'b', 'y', 'v', ' ', 'f', 'j', '5', 'â', 'ú', 'á', 'h', 'ç', "'", '9', 'é', 'p', 'n', '\x00', 'ö', 'ü', '3', 'ê', 'z', 'ó', 'õ', 'x', 'q', 'd', 'å', 'm', 'l', 'í', 's', 'o', 'u', 'à', 'g', 'k', 'è', 'r', 'ã', 'w', 'ñ', 'ò', 'ô', 't', '\ufeff', '2'] vocabulary.
[NeMo I 2021-06-17 18:47:09 <ipython-input-37-2dee6ca3573c>:7] Model encoder has been un-frozen
[NeMo I 2021-06-17 18:47:51 collections:173] Dataset loaded with 935459 files totalling 1020.24 hours
[NeMo I 2021-06-17 18:47:51 collections:174] 16725 files were filtered totalling 90.65 hours
[NeMo I 2021-06-17 18:47:52 collections:173] Dataset loaded with 2338 files totalling 10.50 hours
[NeMo I 2021-06-17 18:47:52 collections:174] 0 files were filtered totalling 0.00 hours
[NeMo I 2021-06-17 18:47:52 exp_manager:216] Experiments will be logged at /home/f2it/NeMo/dataSetMuitasHorasCompleta/experiments/lang-pt/ASR-Char-Model-Language-pt/2021-06-17_18-47-52
[NeMo I 2021-06-17 18:47:52 exp_manager:563] TensorboardLogger has been set up
[NeMo I 2021-06-17 18:47:54 modelPT:748] Optimizer config = Novograd (
    Parameter Group 0
        amsgrad: False
        betas: [0.95, 0.5]
        eps: 1e-08
        grad_averaging: False
        lr: 0.01
        weight_decay: 0.001
    )
[NeMo I 2021-06-17 18:47:54 lr_scheduler:621] Scheduler "<nemo.core.optim.lr_scheduler.CosineAnnealing object at 0x7f7498d04f98>" 
    will be used during training (effective maximum steps = 2923350) - 
    Parameters : 
    (warmup_steps: null
    warmup_ratio: 0.05
    min_lr: 1.0e-05
    last_epoch: -1
    max_steps: 2923350
    )

```

Any solution for this problem?
**Environment:**
![image](https://user-images.githubusercontent.com/49459897/122457676-47f87f80-cf85-11eb-9596-36351982c160.png)

pytorch-lightning             1.2.10
torch                         1.8.0+cu111
nemo-toolkit                  1.0.2