[TTS][refactor] Part 8 - added model inference tests to safeguard changes.
State: closed | Pull request: yes
Labels: name: TTS

Download NGC pre-trained model checkpoints and run model inference for all models supported in `self.list_available_models()`. 

* fixed pretrained model names that are not following naming convention:
  * `tts_hifigan` --> `tts_en_hifigan`
  * `tts_waveglow_88m` --> `tts_en_waveglow_88m` 
* created `--nightly` option and `nightly` marker for pytest. Without `--nightly` option, all tests here will be skipped. We are setting it up this way on purpose to save running time from our CI, and it provides an option for QA.
* added test for fastpitch.
* added test for hifigan.
* added test for mixtts.
* added test for Tacotron2.
* added test for VITS.
* added test for UnivNet.
* added test for waveglow.
* remove old waveglow model that relies on torch_stft.
* added test for aligner.

related PR: https://github.com/NVIDIA/NeMo/pull/6128

```bash
(nemo) xueyang@DEV:~/workspace/NeMo:(add-tests-model-inference-tts)$ pytest tests/collections/tts/models/
A valid `test_data.tar.gz` test archive (10445891B) found in the `/home/xueyang/workspace/NeMo/tests/.data` folder.
Setting numba compat : True
================================================================================================ test session starts =================================================================================================
platform linux -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0 -- /home/xueyang/miniconda3/envs/nemo/bin/python
cachedir: .pytest_cache
rootdir: /home/xueyang/workspace/NeMo, configfile: pyproject.toml
plugins: hydra-core-1.2.0, anyio-3.6.1
collected 26 items

tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests ...) [  3%]
tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner_ipa] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These te...) [  7%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests are...) [ 11%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_ipa] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests...) [ 15%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_multispeaker] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, Th...) [ 19%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2102] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "n...) [ 23%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2210] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "n...) [ 26%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_multispeaker_5] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, ...) [ 30%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_es_fastpitch_multispeaker] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, Th...) [ 34%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_zh_fastpitch_sfspeech] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These ...) [ 38%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifigan] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests are mos...) [ 42%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixertts] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, Thes...) [ 46%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixerttsx] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, The...) [ 50%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifitts_hifigan_ft_fastpitch] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently...) [ 53%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2102] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "night...) [ 57%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2210] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "night...) [ 61%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hui_hifigan_ft_fastpitch_multispeaker_5] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly"...) [ 65%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_es_hifigan_ft_fastpitch_multispeaker] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Curr...) [ 69%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_zh_hifigan_sfspeech] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These test...) [ 73%]
tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixertts] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests ar...) [ 76%]
tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixerttsx] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests a...) [ 80%]
tests/collections/tts/models/test_tacotron2.py::test_inference[tts_en_tacotron2] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests are...) [ 84%]
tests/collections/tts/models/test_univNet.py::test_inference[tts_en_lj_univnet] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests are ...) [ 88%]
tests/collections/tts/models/test_univNet.py::test_inference[tts_en_libritts_univnet] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These test...) [ 92%]
tests/collections/tts/models/test_vits.py::test_inference[tts_en_lj_vits] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests are mostly...) [ 96%]
tests/collections/tts/models/test_waveGlow.py::test_inference[tts_en_waveglow_88m] SKIPPED (To run this test, pass --nightly option. It will run any tests marked with "nightly". Currently, These tests a...) [100%]

================================================================================================= slowest durations ==================================================================================================

(52 durations < 0.005s hidden.  Use -vv to show these durations.)
================================================================================================ 26 skipped in 0.18s =================================================================================================
(nemo) xueyang@DEV:~/workspace/NeMo:(add-tests-model-inference-tts)$ pytest --nightly tests/collections/tts/models/
A valid `test_data.tar.gz` test archive (10445891B) found in the `/home/xueyang/workspace/NeMo/tests/.data` folder.
Setting numba compat : True
================================================================================================ test session starts =================================================================================================
platform linux -- Python 3.8.13, pytest-7.1.3, pluggy-1.0.0 -- /home/xueyang/miniconda3/envs/nemo/bin/python
cachedir: .pytest_cache
rootdir: /home/xueyang/workspace/NeMo, configfile: pyproject.toml
plugins: hydra-core-1.2.0, anyio-3.6.1
collected 26 items

tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner] PASSED                                                                                                                     [  3%]
tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner_ipa] PASSED                                                                                                                 [  7%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch] PASSED                                                                                                                        [ 11%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_ipa] PASSED                                                                                                                    [ 15%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_multispeaker] PASSED                                                                                                           [ 19%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2102] PASSED                                                                                     [ 23%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2210] PASSED                                                                                     [ 26%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_multispeaker_5] PASSED                                                                                                         [ 30%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_es_fastpitch_multispeaker] PASSED                                                                                                           [ 34%]
tests/collections/tts/models/test_fastpitch.py::test_inference[tts_zh_fastpitch_sfspeech] PASSED                                                                                                               [ 38%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifigan] PASSED                                                                                                                            [ 42%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixertts] PASSED                                                                                                             [ 46%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixerttsx] PASSED                                                                                                            [ 50%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifitts_hifigan_ft_fastpitch] PASSED                                                                                                       [ 53%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2102] PASSED                                                                                         [ 57%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2210] PASSED                                                                                         [ 61%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hui_hifigan_ft_fastpitch_multispeaker_5] PASSED                                                                                            [ 65%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_es_hifigan_ft_fastpitch_multispeaker] PASSED                                                                                                  [ 69%]
tests/collections/tts/models/test_hifigan.py::test_inference[tts_zh_hifigan_sfspeech] PASSED                                                                                                                   [ 73%]
tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixertts] ^@PASSED                                                                                                                       [ 76%]
tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixerttsx] PASSED                                                                                                                      [ 80%]
tests/collections/tts/models/test_tacotron2.py::test_inference[tts_en_tacotron2] PASSED                                                                                                                        [ 84%]
tests/collections/tts/models/test_univNet.py::test_inference[tts_en_lj_univnet] PASSED                                                                                                                         [ 88%]
tests/collections/tts/models/test_univNet.py::test_inference[tts_en_libritts_univnet] PASSED                                                                                                                   [ 92%]
tests/collections/tts/models/test_vits.py::test_inference[tts_en_lj_vits] PASSED                                                                                                                               [ 96%]
tests/collections/tts/models/test_waveGlow.py::test_inference[tts_en_waveglow_88m] PASSED                                                                                                                      [100%]

================================================================================================= slowest durations ==================================================================================================
26.45s setup    tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixerttsx]
25.30s setup    tests/collections/tts/models/test_tacotron2.py::test_inference[tts_en_tacotron2]
24.82s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_ipa]
24.43s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch]
24.41s setup    tests/collections/tts/models/test_vits.py::test_inference[tts_en_lj_vits]
24.40s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_multispeaker]
23.52s setup    tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner]
23.17s setup    tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixertts]
22.34s setup    tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner_ipa]
20.70s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_multispeaker_5]
19.61s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2210]
16.09s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2102]
2.35s setup    tests/collections/tts/models/test_waveGlow.py::test_inference[tts_en_waveglow_88m]
2.34s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_es_fastpitch_multispeaker]
2.33s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifigan]
2.29s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixerttsx]
2.27s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixertts]
1.78s call     tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixerttsx]
1.78s call     tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner]
1.53s setup    tests/collections/tts/models/test_fastpitch.py::test_inference[tts_zh_fastpitch_sfspeech]
1.40s setup    tests/collections/tts/models/test_univNet.py::test_inference[tts_en_libritts_univnet]
0.96s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2102]
0.96s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2210]
0.95s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_zh_hifigan_sfspeech]
0.95s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hui_hifigan_ft_fastpitch_multispeaker_5]
0.95s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_es_hifigan_ft_fastpitch_multispeaker]
0.95s setup    tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifitts_hifigan_ft_fastpitch]
0.90s setup    tests/collections/tts/models/test_univNet.py::test_inference[tts_en_lj_univnet]
0.49s call     tests/collections/tts/models/test_vits.py::test_inference[tts_en_lj_vits]
0.48s call     tests/collections/tts/models/test_tacotron2.py::test_inference[tts_en_tacotron2]
0.45s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_zh_fastpitch_sfspeech]
0.28s call     tests/collections/tts/models/test_waveGlow.py::test_inference[tts_en_waveglow_88m]
0.07s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_multispeaker_5]
0.07s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2102]
0.06s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_de_fastpitch_singleSpeaker_thorstenNeutral_2210]
0.06s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch]
0.06s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_ipa]
0.06s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_en_fastpitch_multispeaker]
0.05s call     tests/collections/tts/models/test_mixerTTS.py::test_inference[tts_en_lj_mixertts]
0.05s call     tests/collections/tts/models/test_aligner.py::test_inference[tts_en_radtts_aligner_ipa]
0.03s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifigan]
0.03s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_hifitts_hifigan_ft_fastpitch]
0.03s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hui_hifigan_ft_fastpitch_multispeaker_5]
0.02s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_es_hifigan_ft_fastpitch_multispeaker]
0.02s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixertts]
0.02s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_en_lj_hifigan_ft_mixerttsx]
0.02s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2210]
0.02s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_de_hifigan_singleSpeaker_thorstenNeutral_2102]
0.02s call     tests/collections/tts/models/test_hifigan.py::test_inference[tts_zh_hifigan_sfspeech]
0.02s call     tests/collections/tts/models/test_fastpitch.py::test_inference[tts_es_fastpitch_multispeaker]
0.02s call     tests/collections/tts/models/test_univNet.py::test_inference[tts_en_lj_univnet]
0.01s call     tests/collections/tts/models/test_univNet.py::test_inference[tts_en_libritts_univnet]

(26 durations < 0.005s hidden.  Use -vv to show these durations.)
=========================================================================================== 26 passed in 302.54s (0:05:02) ===========================================================================================

```

**Collection**: [Note which collection this PR will affect]

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)