egs/tts/VALLE/README.md

VALL-E Recipe

In this recipe, we will show how to train VALL-E using Amphion's infrastructure. VALL-E is a zero-shot TTS architecture that uses a neural codec language model with discrete codes.

There are four stages in total:

1. Data preparation
2. Features extraction
3. Training
4. Inference

NOTE: You need to run every command of this recipe in the Amphion root path:

cd Amphion

1. Data Preparation

Dataset Download

You can use the commonly used TTS dataset to train VALL-E model, e.g., LibriTTS, etc. We strongly recommend you use LibriTTS to train VALL-E model for the first time. How to download dataset is detailed here.

Configuration

After downloading the dataset, you can set the dataset paths in exp_config.json. Note that you can change the dataset list to use your preferred datasets.

    "dataset": [
        "libritts",
    ],
    "dataset_path": {
        // TODO: Fill in your dataset path
        "libritts": "[LibriTTS dataset path]",
    },

2. Features Extraction

Configuration

Specify the processed_dir and the log_dir and for saving the processed data and the checkpoints in exp_config.json:

    // TODO: Fill in the output log path. The default value is "Amphion/ckpts/tts"
    "log_dir": "ckpts/tts",
    "preprocess": {
        // TODO: Fill in the output data path. The default value is "Amphion/data"
        "processed_dir": "data",
        ...
    },

Run

Run the run.sh as the preproces stage (set --stage 1):

sh egs/tts/VALLE/run.sh --stage 1

NOTE: The CUDA_VISIBLE_DEVICES is set as "0" in default. You can change it when running run.sh by specifying such as --gpu "1".

3. Training

Configuration

We provide the default hyparameters in the exp_config.json. They can work on single NVIDIA-24g GPU. You can adjust them based on your GPU machines.

"train": {
        "batch_size": 4,
    }

Run

Run the run.sh as the training stage (set --stage 2). Specify a experimental name to run the following command. The tensorboard logs and checkpoints will be saved in Amphion/ckpts/tts/[YourExptName].

Specifically, VALL-E need to train a autoregressive (AR) model and then a non-autoregressive (NAR) model. So, you can set --model_train_stage 1 to train AR model, and set --model_train_stage 2 to train NAR model, where --ar_model_ckpt_dir should be set as the ckeckpoint path to the trained AR model.

Train a AR moel, just run:

sh egs/tts/VALLE/run.sh --stage 2 --model_train_stage 1 --name [YourExptName]

Train a NAR model, just run:

sh egs/tts/VALLE/run.sh --stage 2 --model_train_stage 2 --ar_model_ckpt_dir [ARModelPath] --name [YourExptName]

NOTE: The CUDA_VISIBLE_DEVICES is set as "0" in default. You can change it when running run.sh by specifying such as --gpu "0,1,2,3".

4. Inference

Configuration

For inference, you need to specify the following configurations when running run.sh:

Parameters	Description	Example
--infer_expt_dir	The experimental directory of NAR model which contains checkpoint	Amphion/ckpts/tts/[YourExptName]
--infer_output_dir	The output directory to save inferred audios.	Amphion/ckpts/tts/[YourExptName]/result
--infer_mode	The inference mode, e.g., "single", "batch".	"single" to generate a clip of speech, "batch" to generate a batch of speech at a time.
--infer_text	The text to be synthesized.	"This is a clip of generated speech with the given text from a TTS model."
--infer_text_prompt	The text prompt for inference.	The text prompt should be aligned with the audio prompt.
--infer_audio_prompt	The audio prompt for inference.	The audio prompt should be aligned with text prompt.
--test_list_file	The test list file used for batch inference.	The format of test list file is text|text_prompt|audio_prompt.

Run

For example, if you want to generate a single clip of speech, just run:

sh egs/tts/VALLE/run.sh --stage 3 --gpu "0" \
    --infer_expt_dir Amphion/ckpts/tts/[YourExptName] \
    --infer_output_dir Amphion/ckpts/tts/[YourExptName]/result \
    --infer_mode "single" \
    --infer_text "This is a clip of generated speech with the given text from a TTS model." \
    --infer_text_prompt "But even the unsuccessful dramatist has his moments." \
    --infer_audio_prompt egs/tts/VALLE/prompt_examples/7176_92135_000004_000000.wav

We will release a pre-trained VALL-E. So you can download the pre-trained model and generate speech following the above inference instruction.

@article{wang2023neural,
  title={Neural codec language models are zero-shot text to speech synthesizers},
  author={Wang, Chengyi and Chen, Sanyuan and Wu, Yu and Zhang, Ziqiang and Zhou, Long and Liu, Shujie and Chen, Zhuo and Liu, Yanqing and Wang, Huaming and Li, Jinyu and others},
  journal={arXiv preprint arXiv:2301.02111},
  year={2023}
}