Error when finetuning HifiGan for multi-speaker FastPitch
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

An error occured when trying to finetune tts_en_hifitts_hifigan_ft_fastpitch.nemo

 Using a target size (torch.Size([16, 80, 64])) that is different to the input size (torch.Size([16, 80, 129])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.


Error executing job with overrides: ['model.max_steps=100000', 'train_dataset=data/ruslan.orig/synmels/hifigan_train_manifest_phonemes_ft.json', 'validation_datasets=data/ruslan.orig/synmels/hifigan_val_manifest_phonemes_ft.json', 'exp_manager.exp_dir=data/ruslan.orig/result', '+init_from_nemo_model=pretrained/tts_en_hifitts_hifigan_ft_fastpitch.nemo', 'trainer.devices=-1', '+trainer.val_check_interval=100', 'trainer.check_val_every_n_epoch=null', 'model/train_ds=train_ds_finetune', 'model/validation_ds=val_ds_finetune']
Traceback (most recent call last):
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/clearml/binding/hydra_bind.py", line 173, in _patched_task_function
    return task_function(a_config, *a_args, **a_kwargs)
  File "tts_scripts/hifigan_finetune.py", line 34, in main
    trainer.fit(model)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 603, in fit
    call._call_and_handle_interrupt(
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 90, in launch
    return function(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 645, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1098, in _run
    results = self._run_stage()
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1177, in _run_stage
    self._run_train()
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1190, in _run_train
    self._run_sanity_check()
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1262, in _run_sanity_check
    val_loop.run()
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1480, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 360, in validation_step
    return self.model(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1129, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1083, in _run_ddp_forward
    return self.module(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1533, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 110, in forward
    return self._forward_module.validation_step(*inputs, **kwargs)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/nemo/collections/tts/models/hifigan.py", line 237, in validation_step
    loss_mel = F.l1_loss(audio_mel, audio_pred_mel)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/torch/nn/functional.py", line 3268, in l1_loss
    expanded_input, expanded_target = torch.broadcast_tensors(input, target)
  File "/home/user/.conda/envs/nemo/lib/python3.8/site-packages/torch/functional.py", line 74, in broadcast_tensors
    return _VF.broadcast_tensors(tensors)  # type: ignore[attr-defined]
RuntimeError: The size of tensor a (129) must match the size of tensor b (64) at non-singleton dimension 2

**Steps/Code to reproduce bug**

I am following the standard process of fine tuning FastPitch and HifiGan
1) I finetuned /tts_en_fastpitch_multispeaker.nemo with Russian voice (44100 sr)
2) After that I decided to finetune HifiGan from  finetune tts_en_hifitts_hifigan_ft_fastpitch.nemo

**Expected behavior**
I got the error above. And I fixed it when changed 

### Default values for dataset with sample_rate=44100
sample_rate: 44100
n_mel_channels: 80
n_window_size: 2048
n_window_stride: 256  <- changed to 256 from 512
n_fft: 2048
lowfreq: 0
highfreq: 8000
window: hann

But can you explain why this error occured?
Both my FastPitch model and tts_en_hifitts_hifigan_ft_fastpitch.nemo contain in their config: n_window_stride: 512