Training MSDD on multiple GPUs
State: closed | Pull request: no
Labels: name: bug

name: stale

**How to train MSDD on multiple GPUs?**

When training MSDD model for speaker diarization on multiple GPU units, an error is thrown in the manifest generation step.

FileNotFoundError: [Errno 2] No such file or directory: `<temp_dir>/speaker_outputs/oracle_vad_manifest.json`
where `<temp_dir>` refers to what we set up in the config for `model.train_ds.emb_dir` or `model.valid_ds.emb_dir`.

This happens in the init of the `EncDecDiarLabelModel` class. Below is the complete error trace. 

If training is initiated on a single GPU, the `EncDecDiarLabelModel` gets initialised without any problems. Is this a known limitation of the MSDD model training code or there is something else I need to do to initi the training on multiple GPUs?

Thanks

```
Error executing job with overrides: ['num_workers=20', 'batch_size=15', 'model.train_ds.manifest_filepath=/data/multispeaker-dataset/msdd_manifest_train_for_trainer.json', 'model.validation_ds.manifest_filepath=/data/multispeaker-dataset/msdd_manifest_valid_for_trainer.json', 'model.train_ds.emb_dir=/data/multispeaker-dataset/temp_dir_train', 'model.validation_ds.emb_dir=/data/multispeaker-dataset/temp_dir_valid', 'model.diarizer.speaker_embeddings.model_path=titanet_large', 'model.emb_batch_size=100', 'exp_manager.name=msdd_v1', '+exp_manager.exp_dir=/experiments', '+exp_manager.version=20230422-0827', '+exp_manager.resume_if_exists=true', '+exp_manager.resume_ignore_no_checkpoint=true', '+exp_manager.checkpoint_callback_params.always_save_nemo=true', '+exp_manager.checkpoint_callback_params.save_best_model=false', 'trainer.devices=-1', 'trainer.num_nodes=1', 'trainer.strategy=ddp', 'trainer.accelerator=gpu', '+trainer.check_val_every_n_epoch=1', 'trainer.max_epochs=1000']
Traceback (most recent call last):
  File "/examples/speaker_tasks/diarization/neural_diarizer/multiscale_diar_decoder.py", line 51, in <module>
    main()
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/config/hydra_runner.py", line 105, in wrapper
    _run_hydra(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/examples/speaker_tasks/diarization/neural_diarizer/multiscale_diar_decoder.py", line 46, in main
    msdd_model = EncDecDiarLabelModel(cfg=cfg.model, trainer=trainer)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/models/msdd_models.py", line 123, in __init__
    super().__init__(cfg=self.cfg_msdd_model, trainer=trainer)
  File "/usr/local/lib/python3.8/dist-packages/nemo/core/classes/modelPT.py", line 143, in __init__
    self.setup_training_data(self._cfg.train_ds)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/models/msdd_models.py", line 278, in setup_training_data
    self._train_dl = self.__setup_dataloader_from_config(config=train_data_config,)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/models/msdd_models.py", line 217, in __setup_dataloader_from_config
    dataset = AudioToSpeechMSDDTrainDataset(
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/data/audio_to_diar_label.py", line 774, in __init__
    super().__init__(
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/data/audio_to_diar_label.py", line 226, in __init__
    self.multiscale_timestamp_dict = prepare_split_data(
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/parts/utils/speaker_utils.py", line 1469, in prepare_split_data
    write_rttm2manifest(split_audio_rttm_map, _speaker_manifest_path, include_uniq_id=True)
  File "/usr/local/lib/python3.8/dist-packages/nemo/collections/asr/parts/utils/speaker_utils.py", line 854, in write_rttm2manifest
    with open(manifest_file, 'w') as outfile:
FileNotFoundError: [Errno 2] No such file or directory: '/data/multispeaker-dataset/temp_dir_train/speaker_outputs/oracle_vad_manifest.json'
```