Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion

The official Pytorch implementation of DDDM-VC (AAAI 2024)

Ha-Yeong Choi*, Sang-Hoon Lee*, Seong-Whan Lee

Paper

image

Overall architecture

Diffusion-based generative models have recently exhibited powerful generative performance. However, as many attributes exist in the data distribution and owing to several limitations of sharing the model parameters across all levels of the generation process, it remains challenging to control specific styles for each attribute. To address the above problem, we introduce decoupled denoising diffusion models (DDDMs) with disentangled representations, which can enable effective style transfers for each attribute in generative models. In particular, we apply DDDMs for voice conversion (VC) tasks, tackling the intricate challenge of disentangling and individually transferring each speech attributes such as linguistic information, intonation, and timbre. First, we use a self-supervised representation to disentangle the speech representation. Subsequently, the DDDMs are applied to resynthesize the speech from the disentangled representations for style transfer with respect to each attribute. Moreover, we also propose the prior mixup for robust voice style transfer, which uses the converted representation of the mixed style as a prior distribution for the diffusion models. The experimental results reveal that our method outperforms publicly available VC models. Furthermore, we show that our method provides robust generative performance even when using a smaller model size.

πŸ“‘ Pre-trained Model

Our model checkpoints can be downloaded here.

  • model_base.pth
  • voc_ckpt.pth
  • voc_bigvgan.pth
  • f0_vqvae.pth

βš™οΈ Setup

  1. Clone this rep && Install python requirement
git clone https://github.com/hayeong0/DDDM-VC.git
pip install -r req*
  1. Download the pre-trained model checkpoint from drive.

πŸ”¨ Usage

Preprocess

  1. Data
  • Training requires both wav files and F0 features, which we extract using YAAPT through the script ./preprocess/extract_f0.py.
  • After extracting F0, create a list of files with the path to each data item, as shown in the following example:
train_wav.txt
/workspace/raid/dataset/LibriTTS_16k/train-clean-360/100/121669/100_121669_000001_000000.wav
/workspace/raid/dataset/LibriTTS_16k/train-clean-360/100/121669/100_121669_000003_000000.wav

train_f0.txt
/workspace/raid/dataset/LibriTTS_f0_norm/train-clean-360/100/121669/100_121669_000001_000000.pt
/workspace/raid/dataset/LibriTTS_f0_norm/train-clean-360/100/121669/100_121669_000003_000000.pt
  1. F0_VQVAE

πŸ” Training

  • For training, prepare a file list with the following structure:
|-- filelist 
|    |-- train_f0.txt
|    |-- train_wav.txt
|    |-- test_f0.txt
|    `-- test_wav.txt
  • Run train_dddmvc.py

πŸ”‘ Inference

  • Run infer.sh
bash infer.sh

python3 inference.py \
    --src_path './sample/src_p227_013.wav' \
    --trg_path './sample/tar_p229_005.wav' \
    --ckpt_model './ckpt/model_base.pth' \
    --ckpt_voc './vocoder/voc_ckpt.pth' \
    --ckpt_f0_vqvae './f0_vqvae/f0_vqvae.pth' \
    --output_dir './converted' \
    -t 6

🎧 Train and test it on your own dataset and share your interesting results! πŸ€—

πŸŽ“ Citation

@article{choi2023dddm,
  title={DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion},
  author={Choi, Ha-Yeong and Lee, Sang-Hoon and Lee, Seong-Whan},
  journal={arXiv preprint arXiv:2305.15816},
  year={2023}
}

πŸ’Ž Acknowledgements

License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

CC BY-NC-SA 4.0

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference API
Unable to determine this model's library. Check the docs .