RuntimeError Fastpitch Training: Timed out initializing process group in store based barrier on rank: 0
State: closed | Pull request: no
Labels: name: bug

name: TTS

I am training a fastpitch model and when I use large data i.e. 21,000 audio files to trainset, I get the following error.
`15247it [58:54,  4.53it/s]Waiting in store based barrier to initialize process group for rank: 0, key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00)
14572it [58:55,  3.74it/s]Waiting in store based barrier to initialize process group for rank: 0, key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00)
15334it [59:14,  4.30it/s]Waiting in store based barrier to initialize process group for rank: 0, key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00)
13797it [59:18,  3.78it/s]Waiting in store based barrier to initialize process group for rank: 0, key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00)
13836it [59:28,  3.86it/s]Waiting in store based barrier to initialize process group for rank: 0, key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00)
15461it [59:45,  4.33it/s]Waiting in store based barrier to initialize process group for rank: 0, key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00)
13911it [59:47,  3.76it/s][NeMo W 2022-06-22 12:50:01 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py:420: UserWarning: Error handling mechanism for deadlock detection is uninitialized. Skipping check.
      rank_zero_warn("Error handling mechanism for deadlock detection is uninitialized. Skipping check.")

Error executing job with overrides: ['train_dataset=6670_manifest_other_train.json', 'validation_datasets=6670_manifest_other_dev.json']
15213it [59:40,  4.36it/s]Traceback (most recent call last):
  File "fastpitch_align.py", line 31, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 719, in _call_and_handle_interrupt
    return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1169, in _run
    self.strategy.setup_environment()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 151, in setup_environment
    self.setup_distributed()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 191, in setup_distributed
    init_dist_connection(self.cluster_environment, self._process_group_backend)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py", line 355, in init_dist_connection
    torch.distributed.init_process_group(torch_distributed_backend, rank=global_rank, world_size=world_size, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 610, in init_process_group
    _store_based_barrier(rank, store, timeout)
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 246, in _store_based_barrier
    raise RuntimeError(
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:30:00
)`
I am training this on 

- 7xA100(80G)
- PyTorch Nvidia container v21.12-py3
- Nemo 1.9.0