Finetuning FastPitch on mixed dataset using config `fastpitch_align_v1.05.yaml` is not successful.
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

I have a small dataset consisting of ~1 hrs of audio. I mixed the dataset and the dataset of "old speaker" (the [Nancy corpus](http://www.cstr.ed.ac.uk/projects/blizzard/2011/lessac_blizzard2011/) in my case) according to `FastPitch_Finetuning.ipynb`. 
I trained it using config `fastpitch_align.yaml` and had finetuned a Hifigan, it was successful.
Since `fastpitch_align_v1.05.yaml` comes with the text normalizer, I decided to retrain it with the same mixed manifest.
But the result model does not always produce speech by the new speaker. Most of the speeches produced are still by the old speaker, only occasionally the new speaker will come up.
I tried finetuning with another small dataset using `fastpitch_align_v1.05.yaml`, and have fine-tuned Hifigan with the spectrogram generated by the result Fastpitch model. The speech produced also does not sound right.

**Steps/Code to reproduce bug**

Training with `fastpitch_align_v1.05.yaml`:
`examples/tts/fastpitch_finetune.py --config-name=fastpitch_align_v1.05.yaml train_dataset=/root/autodl-tmp/syb-male1--with--nancy_train.normed.json validation_datasets=/root/autodl-tmp/syb-male1--with--nancy_val.normed.json +init_from_nemo_model=/root/autodl-tmp/nemo-training/nancy_22k/nancy_fastpitch-22k_new.nemo sup_data_path=/root/autodl-tmp/nemo-training/syb-male1/sup_syb-male1 model.train_ds.dataloader_params.batch_size=24 model.validation_ds.dataloader_params.batch_size=24 exp_manager.exp_dir=/root/autodl-tmp/nemo-training/syb-male1 model.n_speakers=2 pitch_mean=117.298095703125 pitch_std=54.89501953125 model.optim.lr=2e-4 ~model.optim.sched model.optim.name=adam`

Training with `fastpitch_align.yaml`:
`examples/tts/fastpitch_finetune.py --config-name=fastpitch_align.yaml train_dataset=/root/autodl-tmp/syb-male1--with--nancy_train.json validation_datasets=/root/autodl-tmp/syb-male1--with--nancy_val.json +init_from_nemo_model=/root/autodl-tmp/nemo-training/nancy_fastpitch-22k.nemo prior_folder=/root/autodl-tmp/nemo-training/multi-males/prior_syb-male1 model.train_ds.dataloader_params.batch_size=24 model.validation_ds.dataloader_params.batch_size=24 exp_manager.exp_dir=/root/autodl-tmp/nemo-training/syb-male1 model.n_speakers=2 model.pitch_avg=121.9 model.pitch_std=37.1 model.pitch_fmin=30 model.pitch_fmax=512 model.optim.lr=2e-4 ~model.optim.sched model.optim.name=adam`

(Since the Fastpitch model I trained from scratch on Nancy corpus using `fastpitch_align.yaml` (the `nancy_fastpitch-22k.nemo` in the above code) seemed unable to be used to finetune using  `fastpitch_align_v1.05.yaml`, I trained a new Fastpitch model on Nancy corpus using `fastpitch_align_v1.05.yaml` (the `nancy_fastpitch-22k_new.nemo` in the above code).)

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: Cloud
 - Method of NeMo install: from source
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 20.04
- PyTorch version: 1.10.2
- Python version: 3.8

**Additional context**

Add any other context about the problem here.
Example: GPU model
RTX 3090