[TTS] Training for VITS not possible with multiple GPU's
State: closed | Pull request: no
Labels: name: bug

name: TTS

name: stale

**Describe the bug**

When running multi-GPU training for TTS model VITS with standard config for LJSpeech dataset, the training stops after global step 97000 with an exception. Also there is a warning at the beginning of the training about `Grad strides do not match bucket view strides`.

```
... <snip>

[NeMo W 2023-02-08 20:43:43 nemo_logging:349] /home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/autograd/__init__.py:197: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed.  This is not an error, but may impair performance.
    grad.sizes() = [1, 9, 96], strides() = [22176, 96, 1]
    bucket_view.sizes() = [1, 9, 96], strides() = [864, 96, 1] (Triggered internally at ../torch/csrc/distributed/c10d/reducer.cpp:325.)
      Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass

...  <snip>

Epoch 245: 100%|██████████| 206/206 [01:56<00:00,  1.77it/s, v_num=3-0Epoch 245, global step 97908: 'loss_gen_all' was not in top 3
Epoch 246:  67%|██████▋   | 139/206 [01:18<00:37,  1.76it/s, v_num=3-00]Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/anna/tts/NeMo/examples/tts/vits.py", line 30, in main
    trainer.fit(model)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 38, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1205, in _run_train
    self.fit_loop.run()
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 213, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 90, in advance
    outputs = self.manual_loop.run(kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/manual_loop.py", line 110, in advance
    training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 351, in training_step
    return self.model(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 98, in forward
    output = self._forward_module.training_step(*inputs, **kwargs)
  File "/home/anna/tts/NeMo/nemo/utils/model_utils.py", line 362, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/models/vits.py", line 202, in training_step
    audio_pred, l_length, attn, ids_slice, text_mask, z_mask, (z, z_p, m_p, logs_p, m_q, logs_q) = self.net_g(
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/modules/vits_modules.py", line 876, in forward
    l_length = self.dp(x, text_mask, w, g=g)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/modules/vits_modules.py", line 425, in forward
    z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/modules/vits_modules.py", line 352, in forward
    x1, logabsdet = piecewise_rational_quadratic_transform(
  File "/home/anna/tts/NeMo/nemo/collections/tts/helpers/splines.py", line 313, in piecewise_rational_quadratic_transform
    outputs, logabsdet = spline_fn(
  File "/home/anna/tts/NeMo/nemo/collections/tts/helpers/splines.py", line 361, in unconstrained_rational_quadratic_spline
    outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline(
  File "/home/anna/tts/NeMo/nemo/collections/tts/helpers/splines.py", line 394, in rational_quadratic_spline
    if torch.min(inputs) < left or torch.max(inputs) > right:
RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Error executing job with overrides: []
Traceback (most recent call last):
  File "examples/tts/vits.py", line 30, in main
    trainer.fit(model)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
    return function(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1103, in _run
    results = self._run_stage()
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1182, in _run_stage
    self._run_train()
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1205, in _run_train
    self.fit_loop.run()
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 267, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 213, in advance
    batch_output = self.batch_loop.run(kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 90, in advance
    outputs = self.manual_loop.run(kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/manual_loop.py", line 110, in advance
    training_step_output = self.trainer._call_strategy_hook("training_step", *kwargs.values())
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1485, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 351, in training_step
    return self.model(*args, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1040, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1000, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 98, in forward
    output = self._forward_module.training_step(*inputs, **kwargs)
  File "/home/anna/tts/NeMo/nemo/utils/model_utils.py", line 362, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/models/vits.py", line 202, in training_step
    audio_pred, l_length, attn, ids_slice, text_mask, z_mask, (z, z_p, m_p, logs_p, m_q, logs_q) = self.net_g(
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/modules/vits_modules.py", line 876, in forward
    l_length = self.dp(x, text_mask, w, g=g)
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/modules/vits_modules.py", line 425, in forward
    z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
  File "/home/anna/tts/NeMo/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/anna/tts/NeMo/nemo/collections/tts/modules/vits_modules.py", line 352, in forward
    x1, logabsdet = piecewise_rational_quadratic_transform(
  File "/home/anna/tts/NeMo/nemo/collections/tts/helpers/splines.py", line 313, in piecewise_rational_quadratic_transform
    outputs, logabsdet = spline_fn(
  File "/home/anna/tts/NeMo/nemo/collections/tts/helpers/splines.py", line 361, in unconstrained_rational_quadratic_spline
    outputs[inside_interval_mask], logabsdet[inside_interval_mask] = rational_quadratic_spline(
  File "/home/anna/tts/NeMo/nemo/collections/tts/helpers/splines.py", line 394, in rational_quadratic_spline
    if torch.min(inputs) < left or torch.max(inputs) > right:
RuntimeError: min(): Expected reduction dim to be specified for input.numel() == 0. Specify the reduction dim with the 'dim' argument.
```

**Steps/Code to reproduce bug**

Training a LJSpeech model with standard configuration and 2 GPU's. Training with one GPU doesn't show the behaviour.

**Expected behavior**

The training runs without exceptions

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: from source

**Environment details**

- Ubuntu 20.04
- PyTorch 1.13.1
- Python 3.8.10

**Additional context**

2 x NVIDIA A6000 GPU's