openvoice plugin

.ipynb_checkpoints/LICENSE-checkpoint

@@ -0,0 +1,5 @@
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

.ipynb_checkpoints/README-checkpoint.md

@@ -0,0 +1,142 @@
+---
+language:
+- en
+tags:
+- myshell
+- speech-to-speech
+---
+<!-- might put a [width=2000 * height=xxx] img here, this size best fits git page
+<img src="resources\cover.png"> -->
+<img src="resources/dreamvoice.png">
+
+# DreamVoice: Text-guided Voice Conversion
+
+--------------------
+
+## Introduction
+
+DreamVoice is an innovative approach to voice conversion (VC) that leverages text-guided generation to create personalized and versatile voice experiences. 
+Unlike traditional VC methods, which require a target recording during inference, DreamVoice introduces a more intuitive solution by allowing users to specify desired voice timbres through text prompts.
+
+For more details, please check our interspeech paper: [DreamVoice](https://arxiv.org/abs/2406.16314)
+
+To listen to demos and download dataset, please check dreamvoice's homepage: [Homepage](https://haidog-yaqub.github.io/dreamvoice_demo/)
+
+
+# Model Usage
+
+To load the models, you need to install packages:
+
+```
+pip install -r requirements.txt
+```
+
+Then you can use the model with the following code:
+
+- NEW! DreamVoice Plugin for OpenVoice (DreamVG + [Opnevoice](https://github.com/myshell-ai/OpenVoice))
+
+```python
+import torch
+from dreamvoice import DreamVoice_Plugin
+from dreamvoice.openvoice_utils import se_extractor
+from openvoice.api import ToneColorConverter
+
+# init dreamvoice
+dreamvoice = DreamVoice_Plugin(device='cuda')
+
+# init openvoice
+ckpt_converter = 'checkpoints_v2/converter'
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+openvoice = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
+openvoice.load_ckpt(f'{ckpt_converter}/checkpoint.pth')
+
+# generate speaker
+prompt = 'cute female girl voice'
+target_se = dreamvoice.gen_spk(prompt)
+target_se = target_se.unsqueeze(-1)
+
+# content source
+source_path = 'examples/test2.wav'
+source_se = se_extractor(source_path, openvoice).to(device)
+
+# voice conversion
+encode_message = "@MyShell"
+openvoice.convert(
+    audio_src_path=source_path,
+    src_se=source_se,
+    tgt_se=target_se,
+    output_path='output.wav',
+    message=encode_message)
+```
+
+- DreamVoice Plugin for Diffusion-based VC model (DreamVG + DiffVC)
+
+```python
+from dreamvoice import DreamVoice
+
+# Initialize DreamVoice in plugin mode with CUDA device
+dreamvoice = DreamVoice(mode='plugin', device='cuda')
+# Description of the target voice
+prompt = 'young female voice, sounds young and cute'
+# Provide the path to the content audio and generate the converted audio
+gen_audio, sr = dreamvoice.genvc('examples/test1.wav', prompt)
+# Save the converted audio
+dreamvoice.save_audio('gen1.wav', gen_audio, sr)
+
+# Save the speaker embedding if you like the generated voice
+dreamvoice.save_spk_embed('voice_stash1.pt')
+# Load the saved speaker embedding
+dreamvoice.load_spk_embed('voice_stash1.pt')
+# Use the saved speaker embedding for another audio sample
+gen_audio2, sr = dreamvoice.simplevc('examples/test2.wav', use_spk_cache=True)
+dreamvoice.save_audio('gen2.wav', gen_audio2, sr)
+```
+
+- Diffusion-based End-to-end model (DreamVC)
+
+```python
+from dreamvoice import DreamVoice
+
+# Initialize DreamVoice in end-to-end mode with CUDA device
+dreamvoice = DreamVoice(mode='end2end', device='cuda')
+# Provide the path to the content audio and generate the converted audio
+gen_end2end, sr = dreamvoice.genvc('examples/test1.wav', prompt)
+# Save the converted audio
+dreamvoice.save_audio('gen_end2end.wav', gen_end2end, sr)
+
+# Note: End-to-end mode does not support saving speaker embeddings
+# To use a voice generated in end-to-end mode, switch back to plugin mode
+# and extract the speaker embedding from the generated audio
+# Switch back to plugin mode
+dreamvoice = DreamVoice(mode='plugin', device='cuda')
+# Load the speaker audio from the previously generated file
+gen_end2end2, sr = dreamvoice.simplevc('examples/test2.wav', speaker_audio='gen_end2end.wav')
+# Save the new converted audio
+dreamvoice.save_audio('gen_end2end2.wav', gen_end2end2, sr)
+```
+
+- One-shot Voice Conversion (DiffVC)
+
+```python
+from dreamvoice import DreamVoice
+
+# Plugin mode can be used for traditional one-shot voice conversion
+dreamvoice = DreamVoice(mode='plugin', device='cuda')
+# Generate audio using traditional one-shot voice conversion
+gen_tradition, sr = dreamvoice.simplevc('examples/test1.wav', speaker_audio='examples/speaker.wav')
+# Save the converted audio
+dreamvoice.save_audio('gen_tradition.wav', gen_tradition, sr)
+```
+
+## Reference
+
+If you find the code useful for your research, please consider citing:
+
+```bibtex
+@article{hai2024dreamvoice,
+  title={DreamVoice: Text-Guided Voice Conversion},
+  author={Hai, Jiarui and Thakkar, Karan and Wang, Helin and Qin, Zengyi and Elhilali, Mounya},
+  journal={arXiv preprint arXiv:2406.16314},
+  year={2024}
+}
+```

.ipynb_checkpoints/example-checkpoint.py

@@ -0,0 +1,47 @@
+from dreamvoice import DreamVoice
+
+# Plugin mode (DreamVG + ReDiffVC)
+# Initialize DreamVoice in plugin mode with CUDA device
+dreamvoice = DreamVoice(mode='plugin', device='cuda')
+# Description of the target voice
+prompt = 'young female voice, sounds young and cute'
+# Provide the path to the content audio and generate the converted audio
+gen_audio, sr = dreamvoice.genvc('examples/test1.wav', prompt)
+# Save the converted audio
+dreamvoice.save_audio('gen1.wav', gen_audio, sr)
+
+# Save the speaker embedding if you like the generated voice
+dreamvoice.save_spk_embed('voice_stash1.pt')
+# Load the saved speaker embedding
+dreamvoice.load_spk_embed('voice_stash1.pt')
+# Use the saved speaker embedding for another audio sample
+gen_audio2, sr = dreamvoice.simplevc('examples/test2.wav', use_spk_cache=True)
+dreamvoice.save_audio('gen2.wav', gen_audio2, sr)
+
+
+# End-to-end mode (DreamVC)
+# Initialize DreamVoice in end-to-end mode with CUDA device
+dreamvoice = DreamVoice(mode='end2end', device='cuda')
+# Provide the path to the content audio and generate the converted audio
+gen_end2end, sr = dreamvoice.genvc('examples/test1.wav', prompt)
+# Save the converted audio
+dreamvoice.save_audio('gen_end2end.wav', gen_end2end, sr)
+
+# Note: End-to-end mode does not support saving speaker embeddings
+# To use a voice generated in end-to-end mode, switch back to plugin mode
+# and extract the speaker embedding from the generated audio
+# Switch back to plugin mode
+dreamvoice = DreamVoice(mode='plugin', device='cuda')
+# Load the speaker audio from the previously generated file
+gen_end2end2, sr = dreamvoice.simplevc('examples/test2.wav', speaker_audio='gen_end2end.wav')
+# Save the new converted audio
+dreamvoice.save_audio('gen_end2end2.wav', gen_end2end2, sr)
+
+
+# Traditional VC
+# Plugin mode can be used for traditional one-shot voice conversion
+dreamvoice = DreamVoice(mode='plugin', device='cuda')
+# Generate audio using traditional one-shot voice conversion
+gen_tradition, sr = dreamvoice.simplevc('examples/test1.wav', speaker_audio='examples/speaker.wav')
+# Save the converted audio
+dreamvoice.save_audio('gen_tradition.wav', gen_tradition, sr)

.ipynb_checkpoints/openvoice_example-checkpoint.py

@@ -0,0 +1,32 @@
+import torch
+from dreamvoice import DreamVoice_Plugin
+from dreamvoice.openvoice_utils import se_extractor
+from openvoice.api import ToneColorConverter
+
+
+# init dreamvoice
+dreamvoice = DreamVoice_Plugin(device='cuda')
+
+# init openvoice
+ckpt_converter = 'checkpoints_v2/converter'
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+openvoice = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
+openvoice.load_ckpt(f'{ckpt_converter}/checkpoint.pth')
+
+# generate speaker
+prompt = 'cute female girl voice'
+target_se = dreamvoice.gen_spk(prompt)
+target_se = target_se.unsqueeze(-1)
+
+# content source
+source_path = 'examples/test2.wav'
+source_se = se_extractor(source_path, openvoice).to(device)
+
+# voice conversion
+encode_message = "@MyShell"
+openvoice.convert(
+    audio_src_path=source_path,
+    src_se=source_se,
+    tgt_se=target_se,
+    output_path='output.wav',
+    message=encode_message)

README.md

@@ -20,10 +20,7 @@ Unlike traditional VC methods, which require a target recording during inference
 
 For more details, please check our interspeech paper: [DreamVoice](https://arxiv.org/abs/2406.16314)
 
-
-## Demo
-
-🎵 Listen to [examples](https://haidog-yaqub.github.io/dreamvoice_demo/)
+To listen to demos and download dataset, please check dreamvoice's homepage: [Homepage](https://haidog-yaqub.github.io/dreamvoice_demo/)
 
 
 # Model Usage
@@ -36,7 +33,43 @@ pip install -r requirements.txt
 
 Then you can use the model with the following code:
 
-- Plugin mode (DreamVG + DiffVC)
+- NEW! DreamVoice Plugin for OpenVoice (DreamVG + [Opnevoice](https://github.com/myshell-ai/OpenVoice))
+
+```python
+import torch
+from dreamvoice import DreamVoice_Plugin
+from dreamvoice.openvoice_utils import se_extractor
+from openvoice.api import ToneColorConverter
+
+# init dreamvoice
+dreamvoice = DreamVoice_Plugin(device='cuda')
+
+# init openvoice
+ckpt_converter = 'checkpoints_v2/converter'
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+openvoice = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
+openvoice.load_ckpt(f'{ckpt_converter}/checkpoint.pth')
+
+# generate speaker
+prompt = 'cute female girl voice'
+target_se = dreamvoice.gen_spk(prompt)
+target_se = target_se.unsqueeze(-1)
+
+# content source
+source_path = 'examples/test2.wav'
+source_se = se_extractor(source_path, openvoice).to(device)
+
+# voice conversion
+encode_message = "@MyShell"
+openvoice.convert(
+    audio_src_path=source_path,
+    src_se=source_se,
+    tgt_se=target_se,
+    output_path='output.wav',
+    message=encode_message)
+```
+
+- DreamVoice Plugin for Diffusion-based VC model (DreamVG + DiffVC)
 
 ```python
 from dreamvoice import DreamVoice
@@ -58,7 +91,8 @@ dreamvoice.load_spk_embed('voice_stash1.pt')
 gen_audio2, sr = dreamvoice.simplevc('examples/test2.wav', use_spk_cache=True)
 dreamvoice.save_audio('gen2.wav', gen_audio2, sr)
 ```
-- End-to-end mode (DreamVC)
+
+- Diffusion-based End-to-end model (DreamVC)
 
 ```python
 from dreamvoice import DreamVoice

openvoice_example.py

@@ -0,0 +1,32 @@
+import torch
+from dreamvoice import DreamVoice_Plugin
+from dreamvoice.openvoice_utils import se_extractor
+from openvoice.api import ToneColorConverter
+
+
+# init dreamvoice
+dreamvoice = DreamVoice_Plugin(device='cuda')
+
+# init openvoice
+ckpt_converter = 'checkpoints_v2/converter'
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+openvoice = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)
+openvoice.load_ckpt(f'{ckpt_converter}/checkpoint.pth')
+
+# generate speaker
+prompt = 'cute female girl voice'
+target_se = dreamvoice.gen_spk(prompt)
+target_se = target_se.unsqueeze(-1)
+
+# content source
+source_path = 'examples/test2.wav'
+source_se = se_extractor(source_path, openvoice).to(device)
+
+# voice conversion
+encode_message = "@MyShell"
+openvoice.convert(
+    audio_src_path=source_path,
+    src_se=source_se,
+    tgt_se=target_se,
+    output_path='output.wav',
+    message=encode_message)