Error "In 'hifigan.yaml': Could not find 'model/validation_ds/val_ds'" when fine-tuning HiFi-GAN
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

Hello. I'm trying to fine-tune the HiFi-GAN model the same way as in Finetuning HiFi-GAN section of FastPitch_Finetuning.ipynb tutorial (https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb), but I get the following error when I try to run the script: "In 'hifigan.yaml': Could not find 'model/validation_ds/val_ds'". 

I used the following script: 

```
!(python hifigan_finetune.py \
  --config-name=hifigan.yaml \
  model.train_ds.dataloader_params.batch_size=16 \
  model.max_steps=1000 \
  model.optim.lr=0.00001 \
  ~model.optim.sched \
  train_dataset=./hifigan_train_ft.json \
  validation_datasets=./hifigan_val_ft.json \
  exp_manager.exp_dir=hifigan_ft \
  +init_from_pretrained_model=tts_en_hifigan \
  trainer.check_val_every_n_epoch=10 \
  model/train_ds=train_ds_finetune \
  model/validation_ds=val_ds_finetune \
)
```

**Steps/Code to reproduce bug**

Before running it, I: created melspectograms and manifests for both training and validation sets (just used the code from the tutorial and for validation files updated the path); downloaded hifigan_finetune.py and hifigan.yaml by running the following code:

```
# BRANCH = 'main'
!wget https://raw.githubusercontent.com/nvidia/NeMo/$BRANCH/examples/tts/hifigan_finetune.py

!cd conf && mkdir -p hifigan && cd hifigan \
&& wget https://raw.githubusercontent.com/nvidia/NeMo/$BRANCH/examples/tts/conf/hifigan/hifigan.yaml \
&& wget https://raw.githubusercontent.com/nvidia/NeMo/$BRANCH/examples/tts/conf/hifigan/hifigan_44100.yaml \
&& cd .. && cd ..
```

Here is the snippet of my folder structure:

![image](https://user-images.githubusercontent.com/73081144/226811955-1112d693-e602-4105-9c7a-f116c743249c.png)

hifigan_val_ft.json config points to these file paths: 

```
..., "mel_filepath": "9017_manifest_val_dur_5_mins_local_mels/mel_0.npy"}
..., "mel_filepath": "9017_manifest_val_dur_5_mins_local_mels/mel_1.npy"}
```

When I run the hifigan_finetune.py script, I get the following error (full):

```
[NeMo W 2023-03-22 05:09:39 optimizers:54] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-03-22 05:09:39 experimental:27] Module <class 'nemo.collections.tts.models.fastpitch_ssl.FastPitchModel_SSL'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-03-22 05:09:39 experimental:27] Module <class 'nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.IPATokenizer'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-03-22 05:09:39 experimental:27] Module <class 'nemo.collections.tts.models.radtts.RadTTSModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-03-22 05:09:40 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-03-22 05:09:41 experimental:27] Module <class 'nemo.collections.tts.models.ssl_tts.SSLDisentangler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-03-22 05:09:41 experimental:27] Module <class 'nemo.collections.tts.models.vits.VitsModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-03-22 05:09:41 nemo_logging:349] /usr/local/lib/python3.9/dist-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'hifigan.yaml': Defaults list is missing `_self_`. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
      warnings.warn(msg, UserWarning)
    
In 'hifigan.yaml': Could not find 'model/validation_ds/val_ds'

Config search path:
  provider=hydra, path=pkg://hydra.conf
  provider=main, path=file:///content/conf/hifigan
  provider=schema, path=structured://

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
```

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment overview (please complete the following information)**

 - Environment location: [Bare-metal, Docker, Cloud(specify cloud provider - AWS, Azure, GCP, Collab)]
 - Method of NeMo install: [pip install or from source]. Please specify exact commands you used to install.
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Expected behavior**

Script should run without any errors.

**Environment overview**

 - Environment location: Collab
 - Method of NeMo install: as in Collab (pip)

**Environment details**

- PyTorch version: 1.13.1+cu116
- Python version: Python 3.9.16

**Additional comments**

I feel like the solution is extremely easy. I'm not sure why the error only points to 'val_ds' but not 'train_ds'. Perhaps the problem is in this line in the script: "+init_from_pretrained_model=tts_en_hifigan" ?