Dataset metadata for audio_codec
State: closed | Pull request: no
Labels: name: bug

name: documentation

name: TTS

Hello, I'm trying to train an audio codec using my custom data. The configuration requires several parameters, including train_ds_meta, val_ds_meta, log_ds_meta, and log_dir. I'm curious about the meaning and detailed structure of each.

For example, in the case of Speech-to-Text (STT), the training dataset (train_ds) requires a manifest. This manifest is structured with the fields {"audio_filepath", "duration", "text"}, and for STT, the manifest can be specified as a .json file.

I would like to understand specifically how to format these parameters for my audio codec training configuration.

Please provide examples of the file formats and internal structures for train_ds_meta, val_ds_meta, log_ds_meta, and log_dir.

If you are interested, I opened a PR yesterday (https://github.com/NVIDIA/NeMo/pull/8723) which contains a tutorial notebook you could download and try https://github.com/NVIDIA/NeMo/blob/79ffef97d2993b716406227f00fc234c19541610/tutorials/tts/Audio_Codec_Training.ipynb. Change the branch from 'main' to 'codec_tutorial'.

The training manifest only needs `{"audio_filepath", "duration"}`

The metadata structure is defined at https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/data/vocoder_dataset.py#L45.  For each dataset you use it asks for the `manifest_path` to the manifest file, and the root `audio_dir` where the audio is stored. If training on multiple datasets you can specify `sample_weight` to determine how often it samples from each.

`train_ds_meta` configures training data, `val_ds_meta` configures data for computing validation statistics/loss, and `log_ds_meta` is used with this callback (https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/parts/utils/callbacks.py#L331) to configure logging of various artifacts during training (eg. reconstructed audio, embedding plots). If doing local logging then `log_dir` specifies the local directory.

A full training command (copied from the tutorial), with 1 dataset "vctk", looks like:

```
python /content/nemo/examples/tts/audio_codec.py
--config-path=/content/nemo/examples/tts/conf/audio_codec
--config-name=audio_codec_24000.yaml
max_epochs=10
weighted_sampling_steps_per_epoch=10
batch_size=4
log_dir=/content/exps/EnCodec/test_run/logs
exp_manager.exp_dir=/content/exps
+exp_manager.version=test_run
model.log_config.log_wandb=False
model.log_config.log_tensorboard=False
model.log_config.generators.0.log_dequantized=True
trainer.accelerator=gpu
+train_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/train_manifest.json
+train_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
+val_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/dev_manifest.json
+val_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
+log_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/dev_manifest.json
+log_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
```

Let me know if you have any other questions.

Thank you for your kind reply. It was so helpful to me !!



> If you are interested, I opened a PR yesterday (#8723) which contains a tutorial notebook you could download and try https://github.com/NVIDIA/NeMo/blob/79ffef97d2993b716406227f00fc234c19541610/tutorials/tts/Audio_Codec_Training.ipynb. Change the branch from 'main' to 'codec_tutorial'.
> 
> The training manifest only needs `{"audio_filepath", "duration"}`
> 
> The metadata structure is defined at https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/data/vocoder_dataset.py#L45. For each dataset you use it asks for the `manifest_path` to the manifest file, and the root `audio_dir` where the audio is stored. If training on multiple datasets you can specify `sample_weight` to determine how often it samples from each.
> 
> `train_ds_meta` configures training data, `val_ds_meta` configures data for computing validation statistics/loss, and `log_ds_meta` is used with this callback (https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/tts/parts/utils/callbacks.py#L331) to configure logging of various artifacts during training (eg. reconstructed audio, embedding plots). If doing local logging then `log_dir` specifies the local directory.
> 
> A full training command (copied from the tutorial), with 1 dataset "vctk", looks like:
> 
> ```
> python /content/nemo/examples/tts/audio_codec.py
> --config-path=/content/nemo/examples/tts/conf/audio_codec
> --config-name=audio_codec_24000.yaml
> max_epochs=10
> weighted_sampling_steps_per_epoch=10
> batch_size=4
> log_dir=/content/exps/EnCodec/test_run/logs
> exp_manager.exp_dir=/content/exps
> +exp_manager.version=test_run
> model.log_config.log_wandb=False
> model.log_config.log_tensorboard=False
> model.log_config.generators.0.log_dequantized=True
> trainer.accelerator=gpu
> +train_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/train_manifest.json
> +train_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
> +val_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/dev_manifest.json
> +val_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
> +log_ds_meta.vctk.manifest_path=/content/data/vctk_subset_multispeaker/dev_manifest.json
> +log_ds_meta.vctk.audio_dir=/content/data/vctk_subset_multispeaker/audio_preprocessed
> ```
> 
> Let me know if you have any other questions.

I'm using my custom data to fine-tune the pre-trained model. There is no problem when I start training from scratch, but when I fine-tune the pre-trained model, I get an error. Below are my dataset format and error encountered.
![image](https://github.com/NVIDIA/NeMo/assets/112935872/0b13c79d-28ae-42cd-bc06-1bc3110a715c)
![image](https://github.com/NVIDIA/NeMo/assets/112935872/fd650abf-e2bb-492f-a03a-0999eeafcc8d)
![image](https://github.com/NVIDIA/NeMo/assets/112935872/35db43cd-2013-458a-b43d-e0c0ab0b5440)