Port DLE FastPitch
State: closed | Pull request: yes

This PR ports the [DLE FastPitch](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch) model ([arxiv](https://arxiv.org/abs/2006.06873)). There is an example of training in `examples/tts/fastpitch.py`, and inference with a `.nemo` model has been tested.

Issuess for full integration:
1. **Data preparation**
    Currently it assumes a dataset structure:
    ```bash
    LJSpeech-1.1
    ├── durations
    ├── pitch
    └── wavs
    ```
    where `pitch` and `durations` can be pre-calculated with a [script](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/FastPitch/scripts/prepare_dataset.sh) from DLE.
    .json dataset entries should have the following structure:
    ```bash
    {"audio_filepath": <path>, "text": <transcript>, "duration": <duration_in_seconds>, "speaker": <speaker_index_0_based>}
    ```
    Speakers are optional, single-speaker dataset should always have `"spaker": 0`.
2. **Mean and std of F0**
    in the DLE model, mean and std of F0 are being stored inside the model for pitch manipulation during inference. In order to specify pitch transformations in Hz, one needs a way to recompute pitch in standarized values. In this PR, these are empty [placeholders](https://github.com/alancucki/NeMo/blob/fastpitch/nemo/collections/tts/modules/fastpitch.py#L143) that can be used or deleted.
3. **Best checkpoints shouldn't be determined with val loss**
    The training loop saves the best checkpoint based on validation loss; it tends to increase during training and is a poor proxy for quality; the last checkpoint should be the final one.

This pull request **introduces 4 alerts** when merging e137574cbe6348159a8b4366afc9a13fe013a805 into db5a787f460d67f0cd89b400cf81826fa2cc9ed6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-d995225ae3f70723701e1bbfd5fc23850e9055a6)

**new alerts:**

* 2 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Comparison of constants

This pull request **introduces 4 alerts** when merging 9788daf44fc7ebee1bd216ae4178ba8856f2b088 into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-74022e590a7d5291a428b2ba1204b35a7b92840b)

**new alerts:**

* 2 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Comparison of constants

This pull request **introduces 4 alerts** when merging 36ecb1918b2afeec84f5daa4e27ab54bd65048e8 into efaab9e67346d633e3efd49c44a5c2b83661d1fc - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-9ae6fcf87a522bcd80a19e1b46437ecd02db8a2d)

**new alerts:**

* 2 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Comparison of constants

This pull request **introduces 4 alerts** when merging f99838327079814d6d178436697eb88c6ceaf37d into ac79c2c6a5f50186bdcaceb9e72a8391271ce120 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-7ee480fada9f0d7ebd56261140205d5fa05877cd)

**new alerts:**

* 2 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Comparison of constants

This pull request **introduces 4 alerts** when merging 27b51de0c580c2c56a09204b5d57f31ea51f0580 into 479b6155710e1564c60d7824a8e88ee3d18de1f6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ee003775be88098d3241e00c091277b56a908b46)

**new alerts:**

* 2 for Unused import
* 1 for Mismatch in multiple assignment
* 1 for Comparison of constants

This pull request **introduces 2 alerts** when merging 7b2bd34c671e7e91df031f2f3d2a31e1c9453e07 into 479b6155710e1564c60d7824a8e88ee3d18de1f6 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-418ba9d751ea9806fb6596aeb78ed06ec920e730)

**new alerts:**

* 1 for Mismatch in multiple assignment
* 1 for Unused import

This pull request **introduces 1 alert** when merging c603190bbb7be0f1b93552dcf5d0074cc55919ed into ff84f14611b64ee27f16757662ebdc9e2400cd1c - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-1ce6fcb3fb65a68bbae9845ed15db3c08cc619f6)

**new alerts:**

* 1 for Mismatch in multiple assignment

This pull request **introduces 1 alert** when merging 6ee0998abe186cc5cf05ef2972bbe5baca025455 into ff84f14611b64ee27f16757662ebdc9e2400cd1c - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-51eb8500cdb7e8126a3307c6e63f861285a6fdfa)

**new alerts:**

* 1 for Mismatch in multiple assignment

Hi, I am training FastPitch on a private dataset(text-speech). 
I did everything as mentioned, 
1) calculate pitch, durations with [this script](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/FastPitch/scripts/prepare_dataset.sh)
2) train using nemo example training file.

During training, it throws this error


`Epoch 0:   0%|                                                                                                                                      | 0/380 [00:00<?, ?it/s][W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator())
Epoch 0:   2%|██▊                                                                                                                                     | 8/380 [00:08<06:56,  1.12s/it, loss=37, v_num=1-50]Epoch 0:   2%|██▊                                                                                                                                   | 8/380 [00:08<06:56,  1.12s/it, loss=36.2, v_num=1-50]Error executing job with overrides: []
Traceback (most recent call last):
  File "train_fastPitch.py", line 31, in main
    trainer.fit(model)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit
    self._run(model)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run
    self.dispatch()
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch
    self.accelerator.start_training(self)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
    self._results = trainer.run_stage()
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage
    return self.run_train()
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train
    self.train_loop.run_training_epoch()
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 491, in run_training_epoch
    for batch_idx, (batch, is_last_batch) in train_dataloader:
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 112, in profile_iterable
    value = next(iterator)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 534, in prefetch_iterator
    for val in it:
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 464, in __next__
    return self.request_next_batch(self.loader_iters)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 478, in request_next_batch
    return apply_to_collection(loader_iters, Iterator, next)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 85, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__
    data = self._next_data()
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1179, in _next_data
    return self._process_data(data)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
    data.reraise()
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
AssertionError: Caught AssertionError in DataLoader worker process 9.
Original Traceback (most recent call last):
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/nemo/core/classes/common.py", line 792, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/nemo/core/classes/dataset.py", line 59, in collate_fn
    return self._collate_fn(batch)
  File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text.py", line 696, in _collate_fn
    assert len(durs_i) == tokens_i_len, f"{len(durs_i)}, {tokens_i_len}:  {path_i}"
AssertionError: 54, 56:  /root/shangeth/Dataset/20hrs_22khtz/wavs/b-4_2000-5164_4562.wav`

> Hi, I am training FastPitch on a private dataset(text-speech).
> I did everything as mentioned,
> 
> 1. calculate pitch, durations with [this script](https://github.com/NVIDIA/DeepLearningExamples/blob/master/PyTorch/SpeechSynthesis/FastPitch/scripts/prepare_dataset.sh)
> 2. train using nemo example training file.
> 
> During training, it throws this error
> 
> `Epoch 0: 0%| | 0/380 [00:00<?, ?it/s][W reducer.cpp:1050] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters, consider turning this flag off. Note that this warning may be a false positive your model has flow control causing later iterations to have unused parameters. (function operator()) Epoch 0: 2%|██▊ | 8/380 [00:08<06:56, 1.12s/it, loss=37, v_num=1-50]Epoch 0: 2%|██▊ | 8/380 [00:08<06:56, 1.12s/it, loss=36.2, v_num=1-50]Error executing job with overrides: [] Traceback (most recent call last): File "train_fastPitch.py", line 31, in main trainer.fit(model) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 460, in fit self._run(model) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 758, in _run self.dispatch() File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 799, in dispatch self.accelerator.start_training(self) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training self.training_type_plugin.start_training(trainer) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training self._results = trainer.run_stage() File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in run_stage return self.run_train() File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 871, in run_train self.train_loop.run_training_epoch() File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/training_loop.py", line 491, in run_training_epoch for batch_idx, (batch, is_last_batch) in train_dataloader: File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/profiler/profilers.py", line 112, in profile_iterable value = next(iterator) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 534, in prefetch_iterator for val in it: File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 464, in __next__ return self.request_next_batch(self.loader_iters) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/trainer/supporters.py", line 478, in request_next_batch return apply_to_collection(loader_iters, Iterator, next) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 85, in apply_to_collection return function(data, *args, **kwargs) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in __next__ data = self._next_data() File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1179, in _next_data return self._process_data(data) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data data.reraise() File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise raise self.exc_type(msg) AssertionError: Caught AssertionError in DataLoader worker process 9. Original Traceback (most recent call last): File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop data = fetcher.fetch(index) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch return self.collate_fn(data) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/nemo/core/classes/common.py", line 792, in __call__ outputs = wrapped(*args, **kwargs) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/nemo/core/classes/dataset.py", line 59, in collate_fn return self._collate_fn(batch) File "/root/anaconda3/envs/nemoenv/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text.py", line 696, in _collate_fn assert len(durs_i) == tokens_i_len, f"{len(durs_i)}, {tokens_i_len}: {path_i}" AssertionError: 54, 56: /root/shangeth/Dataset/20hrs_22khtz/wavs/b-4_2000-5164_4562.wav`

Please fill an issue.