Cannot fine-tune a Hifigan model
State: closed | Pull request: no
Labels: name: bug

My code looks like:

python /workspaces/TTS/nemo/examples/tts/hifigan_finetune.py --config-path /workspaces/TTS/nemo/examples/tts/conf/hifigan --config-name hifigan.yaml model.max_steps=200000 model.optim.lr=0.00001 ~model.optim.sched train_dataset=/workspaces/TTS/data/train_manifest_mel_test.json validation_datasets=/workspaces/TTS/data/val_manifest_mel_test.json exp_manager.exp_dir=experimentsHifigan +init_from_pretrained_model=tts_en_hifigan +trainer.val_check_interval=100 trainer.check_val_every_n_epoch=null model/train_ds=train_ds_finetune model/validation_ds=val_ds_finetune exp_manager.resume_if_exists=false

And I get:

[NeMo W 2023-08-04 10:01:13 optimizers:54] Apex was not found. Using the lamb or fused_adam optimizer will error out.
[NeMo W 2023-08-04 10:01:13 experimental:27] Module <class 'nemo.collections.asr.modules.audio_modules.SpectrogramToMultichannelFeatures'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-08-04 10:01:13 experimental:27] Module <class 'nemo.collections.tts.models.fastpitch_ssl.FastPitchModel_SSL'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-08-04 10:01:13 experimental:27] Module <class 'nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.IPATokenizer'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-08-04 10:01:13 experimental:27] Module <class 'nemo.collections.tts.models.radtts.RadTTSModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-08-04 10:01:14 experimental:27] Module <class 'nemo.collections.tts.models.ssl_tts.SSLDisentangler'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-08-04 10:01:14 experimental:27] Module <class 'nemo.collections.tts.models.vits.VitsModel'> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2023-08-04 10:01:14 nemo_logging:349] /opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/defaults_list.py:251: UserWarning: In 'hifigan.yaml': Defaults list is missing `_self_`. See https://hydra.cc/docs/upgrades/1.0_to_1.1/default_composition_order for more information
      warnings.warn(msg, UserWarning)
    
[NeMo W 2023-08-04 10:01:14 nemo_logging:349] /opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/hydra.py:119: UserWarning: Future Hydra versions will no longer change working directory at job runtime by default.
    See https://hydra.cc/docs/next/upgrades/1.1_to_1.2/changes_to_job_working_dir/ for more information.
      ret = run_job(
    
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
[NeMo I 2023-08-04 10:01:14 exp_manager:374] Experiments will be logged at experimentsHifigan/HifiGan/2023-08-04_10-01-14
[NeMo I 2023-08-04 10:01:14 exp_manager:797] TensorboardLogger has been set up
[NeMo W 2023-08-04 10:01:14 exp_manager:893] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to 200000. Please ensure that max_steps will run for at least 1 epochs to ensure that checkpointing will not error out.
[NeMo I 2023-08-04 10:01:14 dataset:1068] Loading dataset from /root/../workspaces/TTS/data/train_manifest_mel_test.json.
20869it [00:00, 213363.10it/s]
[NeMo I 2023-08-04 10:01:14 dataset:1092] Loaded dataset with 20869 files.
[NeMo I 2023-08-04 10:01:14 dataset:1094] Dataset contains 29.45 hours.
[NeMo I 2023-08-04 10:01:14 dataset:376] Pruned 0 files. Final dataset contains 20869 files
[NeMo I 2023-08-04 10:01:14 dataset:378] Pruned 0.00 hours. Final dataset contains 29.45 hours.
[NeMo I 2023-08-04 10:01:14 dataset:1068] Loading dataset from /root/../workspaces/TTS/data/val_manifest_mel_test.json.
460it [00:00, 199419.10it/s]
[NeMo I 2023-08-04 10:01:14 dataset:1092] Loaded dataset with 460 files.
[NeMo I 2023-08-04 10:01:14 dataset:1094] Dataset contains 0.76 hours.
[NeMo I 2023-08-04 10:01:14 dataset:376] Pruned 13 files. Final dataset contains 447 files
[NeMo I 2023-08-04 10:01:14 dataset:378] Pruned 0.01 hours. Final dataset contains 0.76 hours.
[NeMo I 2023-08-04 10:01:14 features:291] PADDING: 0
[NeMo I 2023-08-04 10:01:14 features:299] STFT using exact pad
[NeMo I 2023-08-04 10:01:14 features:291] PADDING: 0
[NeMo I 2023-08-04 10:01:14 features:299] STFT using exact pad
[NeMo I 2023-08-04 10:01:15 cloud:58] Found existing object /root/.cache/torch/NeMo/NeMo_1.19.0/tts_hifigan/e6da322f0f7e7dcf3f1900a9229a7e69/tts_hifigan.nemo.
[NeMo I 2023-08-04 10:01:15 cloud:64] Re-using file from: /root/.cache/torch/NeMo/NeMo_1.19.0/tts_hifigan/e6da322f0f7e7dcf3f1900a9229a7e69/tts_hifigan.nemo
[NeMo I 2023-08-04 10:01:15 common:913] Instantiating model from pre-trained checkpoint
[NeMo W 2023-08-04 10:01:17 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
    Train config : 
    dataset:
      _target_: nemo.collections.tts.data.datalayers.MelAudioDataset
      manifest_filepath: /home/fkreuk/data/train_finetune.txt
      min_duration: 0.75
      n_segments: 8192
    dataloader_params:
      drop_last: false
      shuffle: true
      batch_size: 64
      num_workers: 4
    
[NeMo W 2023-08-04 10:01:17 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). 
    Validation config : 
    dataset:
      _target_: nemo.collections.tts.data.datalayers.MelAudioDataset
      manifest_filepath: /home/fkreuk/data/val_finetune.txt
      min_duration: 3
      n_segments: 66150
    dataloader_params:
      drop_last: false
      shuffle: false
      batch_size: 5
      num_workers: 4
    
[NeMo W 2023-08-04 10:01:17 features:268] Using torch_stft is deprecated and has been removed. The values have been forcibly set to False for FilterbankFeatures and AudioToMelSpectrogramPreprocessor. Please set exact_pad to True as needed.
[NeMo I 2023-08-04 10:01:17 features:291] PADDING: 0
[NeMo W 2023-08-04 10:01:17 features:268] Using torch_stft is deprecated and has been removed. The values have been forcibly set to False for FilterbankFeatures and AudioToMelSpectrogramPreprocessor. Please set exact_pad to True as needed.
[NeMo I 2023-08-04 10:01:17 features:291] PADDING: 0
[NeMo I 2023-08-04 10:01:18 save_restore_connector:249] Model HifiGanModel was successfully restored from /root/.cache/torch/NeMo/NeMo_1.19.0/tts_hifigan/e6da322f0f7e7dcf3f1900a9229a7e69/tts_hifigan.nemo.
[NeMo I 2023-08-04 10:01:18 modelPT:1268] Model checkpoint restored from pretrained checkpoint with name : `tts_en_hifigan`
Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/1
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 1 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [1]

  | Name                       | Type                     | Params
------------------------------------------------------------------------
0 | audio_to_melspec_precessor | FilterbankFeatures       | 0     
1 | trg_melspec_fn             | FilterbankFeatures       | 0     
2 | generator                  | Generator                | 13.9 M
3 | mpd                        | MultiPeriodDiscriminator | 41.1 M
4 | msd                        | MultiScaleDiscriminator  | 29.6 M
5 | feature_loss               | FeatureMatchingLoss      | 0     
6 | discriminator_loss         | DiscriminatorLoss        | 0     
7 | generator_loss             | GeneratorLoss            | 0     
------------------------------------------------------------------------
84.7 M    Trainable params
0         Non-trainable params
84.7 M    Total params
338.643   Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]Error executing job with overrides: ['model.max_steps=200000', 'model.optim.lr=0.00001', '~model.optim.sched', 'train_dataset=/root/../workspaces/TTS/data/train_manifest_mel_test.json', 'validation_datasets=/root/../workspaces/TTS/data/val_manifest_mel_test.json', 'exp_manager.exp_dir=experimentsHifigan', '+init_from_pretrained_model=tts_en_hifigan', '+trainer.val_check_interval=100', 'trainer.check_val_every_n_epoch=1', 'model/train_ds=train_ds_finetune', 'model/validation_ds=val_ds_finetune', 'model.train_ds.dataloader_params.batch_size=16', 'exp_manager.resume_if_exists=false']
Traceback (most recent call last):
  File "/root/../workspaces/TTS/nemo/examples/tts/hifigan_finetune.py", line 32, in <module>
    main()  # noqa pylint: disable=no-value-for-parameter
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py", line 105, in wrapper
    _run_hydra(
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/root/../workspaces/TTS/nemo/examples/tts/hifigan_finetune.py", line 28, in main
    trainer.fit(model)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
    return function(*args, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1204, in _run_train
    self._run_sanity_check()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1276, in _run_sanity_check
    val_loop.run()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 121, in advance
    batch = next(data_fetcher)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in __next__
    return self.fetching_function()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 265, in fetching_function
    self._fetch_next_batch(self.dataloader_iter)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 280, in _fetch_next_batch
    batch = next(iterator)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/_utils.py", line 644, in reraise
    raise exception
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/envs/nemo/lib/python3.8/site-packages/nemo/collections/tts/data/dataset.py", line 1144, in __getitem__
    start = random.randint(0, mel.shape[1] - frames - 1)
  File "/opt/conda/envs/nemo/lib/python3.8/random.py", line 248, in randint
    return self.randrange(a, b+1)
  File "/opt/conda/envs/nemo/lib/python3.8/random.py", line 226, in randrange
    raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (0, -16, -16)

I am using main branch of nemo.