Runtime error when exporting FastPitch model to PyTorch
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

I tried to export FastPitch model to PyTorch but failed with a following error. It happens with all EN models  `tts_en_fastpitch_ipa`, `tts_en_fastpitch`, and `tts_en_fastpitch_multispeaker`.

**Steps/Code to reproduce bug**

I tried the same code with all the models, just changing a model label:
```
import nemo.collections.tts as nemo_tts
model = nemo_tts.models.FastPitchModel.from_pretrained("tts_en_fastpitch_multispeaker")
model.export('tts_en_fastpitch_multispeaker.pt')
```
The models were loaded successfully, e.g.:
```
[NeMo I 2024-03-08 16:00:55 cloud:68] Downloading from: https://api.ngc.nvidia.com/v2/models/nvidia/nemo/tts_en_fastpitch/versions/IPA_1.13.0/files/tts_en_fastpitch_align_ipa.nemo to /root/.cache/torch/NeMo/NeMo_1.23.0/tts_en_fastpitch_align_ipa/63969a9d8c2d5f4870d16fbf8064a5e0/tts_en_fastpitch_align_ipa.nemo
[NeMo I 2024-03-08 16:01:04 common:924] Instantiating model from pre-trained checkpoint
 NeMo-text-processing :: INFO     :: Creating ClassifyFst grammars.
INFO:NeMo-text-processing:Creating ClassifyFst grammars.
[NeMo W 2024-03-08 16:02:17 deprecated:65] Function ``g2p_backward_compatible_support`` is deprecated. But it will not be removed until a further notice. G2P object root directory `nemo_text_processing.g2p` has been replaced with `nemo.collections.tts.g2p`. Please use the latter instead as of NeMo 1.18.0.
[NeMo W 2024-03-08 16:02:17 experimental:26] `<class 'nemo.collections.tts.g2p.models.i18n_ipa.IpaG2p'>` is experimental and not ready for production yet. Use at your own risk.
[NeMo W 2024-03-08 16:02:18 i18n_ipa:124] apply_to_oov_word=None, This means that some of words will remain unchanged if they are not handled by any of the rules in self.parse_one_word(). This may be intended if phonemes and chars are both valid inputs, otherwise, you may see unexpected deletions in your input.
[NeMo W 2024-03-08 16:02:18 experimental:26] `<class 'nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.IPATokenizer'>` is experimental and not ready for production yet. Use at your own risk.
[NeMo W 2024-03-08 16:02:18 modelPT:165] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    dataset:
      _target_: nemo.collections.tts.torch.data.TTSDataset
      manifest_filepath: /data3/LJSpeech/nvidia_ljspeech_train.json
      sample_rate: 22050
      sup_data_path: /data3/LJSpeech/tmp_ignoreamb/
      sup_data_types:
      - align_prior_matrix
      - pitch
      n_fft: 1024
      win_length: 1024
      hop_length: 256
      window: hann
      n_mels: 80
      lowfreq: 0
      highfreq: 8000
      max_duration: null
      min_duration: 0.1
      ignore_file: null
      trim: false
      pitch_fmin: 65.40639132514966
      pitch_fmax: 2093.004522404789
      pitch_norm: true
      pitch_mean: 212.35873413085938
      pitch_std: 68.52806091308594
      use_beta_binomial_interpolator: true
    dataloader_params:
      drop_last: false
      shuffle: true
      batch_size: 32
      num_workers: 12
    
[NeMo W 2024-03-08 16:02:18 modelPT:172] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    dataset:
      _target_: nemo.collections.tts.torch.data.TTSDataset
      manifest_filepath: /data3/LJSpeech/nvidia_ljspeech_val.json
      sample_rate: 22050
      sup_data_path: /data3/LJSpeech/tmp_ignoreamb/
      sup_data_types:
      - align_prior_matrix
      - pitch
      n_fft: 1024
      win_length: 1024
      hop_length: 256
      window: hann
      n_mels: 80
      lowfreq: 0
      highfreq: 8000
      max_duration: null
      min_duration: null
      ignore_file: null
      trim: false
      pitch_fmin: 65.40639132514966
      pitch_fmax: 2093.004522404789
      pitch_norm: true
      pitch_mean: 212.35873413085938
      pitch_std: 68.52806091308594
      use_beta_binomial_interpolator: true
    dataloader_params:
      drop_last: false
      shuffle: false
      batch_size: 32
      num_workers: 8
    
[NeMo I 2024-03-08 16:02:18 features:289] PADDING: 1
[NeMo I 2024-03-08 16:02:19 save_restore_connector:249] Model FastPitchModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.23.0/tts_en_fastpitch_align_ipa/63969a9d8c2d5f4870d16fbf8064a5e0/tts_en_fastpitch_align_ipa.nemo.
```
But the export filed, in case of `tts_en_fastpitch_ipa` and `tts_en_fastpitch` models, the error was:
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-08f90d3e012f> in <cell line: 1>()
----> 1 model.export('tts_en_fastpitch.pt')

8 frames
/usr/local/lib/python3.10/dist-packages/nemo/collections/tts/parts/utils/helpers.py in regulate_len(durations, enc_out, pace, mel_max_len, group_size, dur_lens)
    595     mult = (reps_cumsum[:, :, :-1] <= range_) & (reps_cumsum[:, :, 1:] > range_)
    596     mult = mult.to(dtype)
--> 597     enc_rep = torch.matmul(mult, enc_out)
    598 
    599     if mel_max_len is not None:

RuntimeError: mat1 and mat2 shapes cannot be multiplied (271x44 and 1x1)
```

in case of `tts_en_fastpitch_multispeaker` model, the error was:
```
ValueError                                Traceback (most recent call last)
<ipython-input-10-1205d9628005> in <cell line: 4>()
      2 
      3 model = nemo_tts.models.FastPitchModel.from_pretrained("tts_en_fastpitch_multispeaker")
----> 4 model.export('tts_en_fastpitch_multispeaker.pt')

8 frames
/usr/local/lib/python3.10/dist-packages/nemo/collections/tts/modules/fastpitch.py in get_speaker_embedding(self, batch_size, speaker, reference_spec, reference_spec_lens)
    260         elif self.speaker_emb is not None:
    261             if speaker is None:
--> 262                 raise ValueError('Please give speaker id to get lookup speaker embedding.')
    263             spk_emb = self.speaker_emb(speaker).unsqueeze(1)
    264         else:

ValueError: Please give speaker id to get lookup speaker embedding.
```

**Environment**
The code was run in Google Colab.
The NeMO toolkit was installed using a pair lines:
```
!pip3 install torch torchaudio torchvision
!pip install nemo_toolkit['all']
```
Python 3.10
PyTorch 2.1.0+cu121



I have the same problem, exporting a FastPitch model (either pretrained or fine-tuned) to TorchScript causes the error described.

This issue seems to be fixed when `nemo/collections/tts/models/fastpitch.py:842` is changed to:

> def input_example(self, max_batch=**44**, max_dim=44):

Don't know if there are any other effects of doing this, but none seem to impact my use case.

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.