Tacotron Finetune Error(s) in loading state_dict for Tacotron2Model: size mismatch 
State: closed | Pull request: no
Labels: name: bug

Hi, Nemo Developer:
    Happy new year first. 

**Describe the bug**
I am trying to have a trial on voice clone. So I am trying to finetune a TTS model based on a test person's voice clips. Have successfully realized this using that FastPitch notebook. I am stuck on Tacotron2 finetune trial. 

**Steps/Code to reproduce bug**
I am trying to run that tacotron2_finetune.py for the training. As my audio clips are 22050hz, I modified the three parameters in the tacotron2_44100.yaml following the 22050hz original tacotron2.yaml training config file:
  n_window_size: 1024
  n_window_stride: 256
  n_fft: 1024 
Other configs untouched.


**Expected behavior**
I am running it as:
`!(python tacotron2_finetune.py  --config-name=tacotron2_22050.yaml \
  train_dataset=./TestFileList.json \
  validation_datasets=./TestFileList.json \
  exp_manager.exp_dir=./ljspeech_to_test_no_mixing \
  +init_from_nemo_model=./tts_en_tacotron2.nemo \
  trainer.max_epochs=20 \
  trainer.check_val_every_n_epoch=5 \
  model.train_ds.dataloader_params.batch_size=24 model.validation_ds.dataloader_params.batch_size=24 \
)`

So I am mainly adding the pretrained model of the original pretrained tacotron2 model.
The output is shown as below:
`[NeMo I 2022-12-29 23:33:49 features:267] PADDING: 16
[NeMo I 2022-12-29 23:33:49 save_restore_connector:243] Model Tacotron2Model was successfully restored from /media/xyzhou/extDisk2t1/DeepFake_Audio/Nemo_Clone/tts_nemo/tts_en_tacotron2.nemo.
Error executing job with overrides: ['train_dataset=./EricFileList.json', 'validation_datasets=./EricFileList.json', 'exp_manager.exp_dir=./ljspeech_to_eric_no_mixing', '+init_from_nemo_model=./tts_en_tacotron2.nemo', 'trainer.max_epochs=20', 'trainer.check_val_every_n_epoch=5', 'model.train_ds.dataloader_params.batch_size=24', 'model.validation_ds.dataloader_params.batch_size=24']
Traceback (most recent call last):
  File "tacotron2_finetune.py", line 35, in main
    model.maybe_init_from_pretrained_checkpoint(cfg=cfg)
  File "/home/xyzhou/anaconda3/envs/nemo/lib/python3.8/site-packages/lightning_utilities/core/rank_zero.py", line 24, in wrapped_fn
    return fn(*args, **kwargs)
  File "/home/xyzhou/anaconda3/envs/nemo/lib/python3.8/site-packages/nemo/core/classes/modelPT.py", line 1066, in maybe_init_from_pretrained_checkpoint
    self.load_state_dict(restored_model.state_dict(), strict=False)
  File "/home/xyzhou/anaconda3/envs/nemo/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1671, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for Tacotron2Model:
	size mismatch for text_embedding.weight: copying a param with shape torch.Size([69, 512]) from checkpoint, the shape in current model is torch.Size([114, 512]).`

The the main error shown is the final part:
RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Tacotron2Model: size mismatch for text_embedding.weight: copying a param with shape torch.Size([69, 512]) from checkpoint, the shape in current model is torch.Size([114, 512]).

Not sure how to deal with this. Thank you!