fastpitch_hifigan_e2e.py requires /fastpitch/durations/ and /fastpitch/pitch_char/ .pt files?
State: closed | Pull request: no
Labels: name: bug

fastpich_hifigan_e2e.py (just like fastpitch.py) requires some extra processing setup steps and I can't find what generates the missing durations and pitch_char .pt files.  Can anyone give me a hint?

There was an earlier defect closed that talked about needing to run tacotron2 training first.
I did that for a couple of epochs and none of the required files were generated.
https://github.com/NVIDIA/NeMo/issues/2352

File "/home/alansrobotlab/anaconda3/envs/nemo/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text.py", line 734, in __getitem__
    return (audio, audio_len, text, text_len, torch.load(durs_path), torch.load(pitch_path), speaker, audio_path)

audio_to_text.py __getitem__

    def __getitem__(self, item):
        audio, audio_len, text, text_len = super().__getitem__(item)  # noqa

        audio_path = self.manifest_processor.collection[item].audio_file
        durs_path = audio_path.replace('/wavs/', '/fastpitch/durations/').replace('.wav', '.pt')
        pitch_path = audio_path.replace('/wavs/', '/fastpitch/pitch_char/').replace('.wav', '.pt')
        speaker = None
        if self.manifest_processor.collection[item].speaker is not None:
            speaker = torch.zeros_like(text_len).fill_(self.manifest_processor.collection[item].speaker)

        return (audio, audio_len, text, text_len, torch.load(durs_path), torch.load(pitch_path), speaker, audio_path)



Complete error from run:
```
root@f62e1306b3c4:/workspace# python fastpitch_                                                                                                                                                                                           
fastpitch_6097.py                         fastpitch_hifigan_e2e_6097.py             fastpitch_pyt_fp32_ckpt_v1_1_21.05.0.zip  
fastpitch_durs_6097.py                    fastpitch_pitch_6097.py                   fastpitch_spect_6097.py                   
root@f62e1306b3c4:/workspace# python fastpitch_hifigan_e2e_6097.py 
[NeMo W 2021-09-07 21:52:43 experimental:27] Module <class 'nemo.collections.asr.data.audio_to_text_dali._AudioTextDALIDataset'> is experimental, not ready for production and is not fully supported. Use at your own risk.
Using native 16bit precision.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
[NeMo I 2021-09-07 21:52:43 exp_manager:220] Experiments will be logged at /workspace/nemo_experiments/FastPitch_HiFiGAN_E2E/2021-09-07_21-52-43
[NeMo I 2021-09-07 21:52:43 exp_manager:569] TensorboardLogger has been set up
[NeMo W 2021-09-07 21:52:43 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/pytorch_lightning/callbacks/model_checkpoint.py:240: LightningDeprecationWarning: `ModelCheckpoint(every_n_val_epochs)` is deprecated in v1.4 and will be removed in v1.6. Please use `every_n_epochs` instead.
      rank_zero_deprecation(
    
[NeMo I 2021-09-07 21:52:47 collections:173] Dataset loaded with 39373 files totalling 29.95 hours
[NeMo I 2021-09-07 21:52:47 collections:174] 0 files were filtered totalling 0.00 hours
[NeMo I 2021-09-07 21:52:47 collections:173] Dataset loaded with 100 files totalling 0.10 hours
[NeMo I 2021-09-07 21:52:47 collections:174] 0 files were filtered totalling 0.00 hours
[NeMo I 2021-09-07 21:52:47 features:252] PADDING: 1
[NeMo I 2021-09-07 21:52:47 features:269] STFT using torch
[NeMo I 2021-09-07 21:52:47 features:271] STFT using exact pad
[NeMo I 2021-09-07 21:52:47 features:252] PADDING: 1
[NeMo I 2021-09-07 21:52:47 features:269] STFT using torch
[NeMo I 2021-09-07 21:52:47 features:271] STFT using exact pad
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Using native 16bit precision.
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
Added key: store_based_barrier_key:1 to store for rank: 1
Added key: store_based_barrier_key:1 to store for rank: 0
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All DDP processes registered. Starting ddp with 2 processes
----------------------------------------------------------------------------------------------------

LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1]
[NeMo I 2021-09-07 21:52:56 fastpitch_hifigan_e2e:184] MAX STEPS: 462000.0

   | Name               | Type                     | Params
-----------------------------------------------------------------
0  | preprocessor       | FilterbankFeatures       | 0     
1  | melspec_fn         | FilterbankFeatures       | 0     
2  | encoder            | FFTransformerEncoder     | 21.9 M
3  | duration_predictor | TemporalPredictor        | 493 K 
4  | pitch_predictor    | TemporalPredictor        | 493 K 
5  | generator          | Generator                | 15.0 M
6  | multiperioddisc    | MultiPeriodDiscriminator | 41.1 M
7  | multiscaledisc     | MultiScaleDiscriminator  | 29.6 M
8  | mel_val_loss       | L1MelLoss                | 0     
9  | feat_matching_loss | FeatureMatchingLoss      | 0     
10 | disc_loss          | DiscriminatorLoss        | 0     
11 | gen_loss           | GeneratorLoss            | 0     
12 | pitch_emb          | Conv1d                   | 1.5 K 
13 | pitchloss          | PitchLoss                | 0     
14 | durationloss       | DurationLoss             | 0     
-----------------------------------------------------------------
108 M     Trainable params
0         Non-trainable params
108 M     Total params
434.563   Total estimated model params size (MB)
Validation sanity check:   0%|                                                                                                                                                                                      | 0/1 [00:00<?, ?it/s]Error executing job with overrides: []
Error executing job with overrides: []
Traceback (most recent call last):
  File "fastpitch_hifigan_e2e_6097.py", line 32, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 917, in _run
    self._dispatch()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 985, in _dispatch
    self.accelerator.start_training(self)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 995, in run_stage
    return self._run_train()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_train
    self._run_sanity_check(self.lightning_module)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1114, in _run_sanity_check
    self._evaluation_loop.run()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
    dl_outputs = self.epoch_loop.run(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 93, in advance
    batch_idx, batch = next(dataloader_iter)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 518, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1200, in _next_data
    return self._process_data(data)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1226, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text.py", line 734, in __getitem__
    return (audio, audio_len, text, text_len, torch.load(durs_path), torch.load(pitch_path), speaker, audio_path)
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 579, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'hi_fi_tts_v0/audio/6097_clean/fastpitch/durations/nada_lily_00_haggard_0063.pt'


Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
Traceback (most recent call last):
  File "/workspace/fastpitch_hifigan_e2e_6097.py", line 32, in main
    trainer.fit(model)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 917, in _run
    self._dispatch()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 985, in _dispatch
    self.accelerator.start_training(self)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 92, in start_training
    self.training_type_plugin.start_training(trainer)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 161, in start_training
    self._results = trainer.run_stage()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 995, in run_stage
    return self._run_train()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1030, in _run_train
    self._run_sanity_check(self.lightning_module)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1114, in _run_sanity_check
    self._evaluation_loop.run()
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 110, in advance
    dl_outputs = self.epoch_loop.run(
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 111, in run
    self.advance(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 93, in advance
    batch_idx, batch = next(dataloader_iter)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 518, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1200, in _next_data
    return self._process_data(data)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1226, in _process_data
    data.reraise()
  File "/opt/conda/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise self.exc_type(msg)
FileNotFoundError: Caught FileNotFoundError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_text.py", line 734, in __getitem__
    return (audio, audio_len, text, text_len, torch.load(durs_path), torch.load(pitch_path), speaker, audio_path)
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 579, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/opt/conda/lib/python3.8/site-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: 'hi_fi_tts_v0/audio/6097_clean/fastpitch/durations/nada_lily_00_haggard_0077.pt'


Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
```