CosyVoice commited on
Commit
ca4e036
1 Parent(s): b6d54c0
Files changed (1) hide show
  1. README.md +87 -34
README.md CHANGED
@@ -1,4 +1,8 @@
1
  # CosyVoice
 
 
 
 
2
 
3
  ## Install
4
 
@@ -6,7 +10,10 @@
6
 
7
  - Clone the repo
8
  ``` sh
9
- git clone https://github.com/modelscope/cosyvoice.git
 
 
 
10
  ```
11
 
12
  - Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
@@ -16,72 +23,99 @@ git clone https://github.com/modelscope/cosyvoice.git
16
  conda create -n cosyvoice python=3.8
17
  conda activate cosyvoice
18
  pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
 
 
 
 
 
 
19
  ```
20
 
21
  **Model download**
22
 
23
- We strongly recommand that you download our pretrained multi_lingual and mutli_emotion model.
24
 
25
  If you are expert in this field, and you are only interested in training your own CosyVoice model from scratch, you can skip this step.
26
 
 
 
 
 
 
 
 
 
 
27
  ``` sh
 
28
  mkdir -p pretrained_models
29
- git clone https://www.modelscope.cn/CosyVoice/multi_lingual_cosytts.git pretrained_models/multi_lingual_cosytts
30
- git clone https://www.modelscope.cn/CosyVoice/multi_emotion_cosytts.git pretrained_models/multi_emotion_cosytts
 
 
 
 
 
 
 
 
 
31
  ```
32
 
33
  **Basic Usage**
34
 
35
- For zero_shot and sft inference, please use models in `pretrained_models/multi_lingual_cosytts`
 
 
 
36
 
 
 
37
  ```
 
 
38
  from cosyvoice.cli.cosyvoice import CosyVoice
39
  from cosyvoice.utils.file_utils import load_wav
40
  import torchaudio
41
 
42
- cosyvoice = CosyVoice('pretrained_models/multi_lingual_cosytts')
43
-
44
  # sft usage
45
  print(cosyvoice.list_avaliable_spks())
46
- output = cosyvoice.inference_sft('hello, my name is Jack. What is your name?', 'aishuo')
47
  torchaudio.save('sft.wav', output['tts_speech'], 22050)
48
 
 
49
  # zero_shot usage
50
- prompt_speech_22050 = load_wav('1089_134686_000002_000000.wav', 22050)
51
- output = cosyvoice.inference_zero_shot('hello, my name is Jack. What is your name?', 'It would be a gloomy secret night.', prompt_speech_22050)
52
  torchaudio.save('zero_shot.wav', output['tts_speech'], 22050)
53
- ```
54
-
55
- For instruct inference, please use models in `pretrained_models/multi_emotion_cosytts`
56
-
57
- ```
58
- from cosyvoice.cli.cosyvoice import CosyVoice
59
- from cosyvoice.utils.file_utils import load_wav
60
- import torchaudio
61
-
62
- cosyvoice = CosyVoice('pretrained_models/multi_emotion_cosytts')
63
 
 
64
  # instruct usage
65
- prompt_speech_22050 = load_wav('1089_134686_000002_000000.wav', 22050)
66
- output = cosyvoice.inference_instruct('hello, my name is Jack. What is your name?', 'It would be a gloomy secret night.', prompt_speech_22050, 'A serene woman articulates thoughtfully in a high pitch and slow tempo, exuding a peaceful and joyful aura.')
67
  torchaudio.save('instruct.wav', output['tts_speech'], 22050)
68
  ```
69
 
70
- **Advanced Usage**
71
-
72
- For advanced user, we have provided train and inference scripts in `examples/libritts/cosyvoice/run.sh`.
73
- You can get familiar with CosyVoice following this recipie.
74
-
75
  **Start web demo**
76
 
77
  You can use our web demo page to get familiar with CosyVoice quickly.
78
- We only support zero_shot/sft inference in web demo.
79
 
80
  Please see the demo website for details.
81
 
 
 
 
82
  ```
83
- python3 webui.py --port 50000 --model_dir pretrained_models/multi_lingual_cosytts
84
- ```
 
 
 
85
 
86
  **Build for deployment**
87
 
@@ -91,7 +125,26 @@ you can run following steps. Otherwise, you can just ignore this step.
91
  ``` sh
92
  cd runtime/python
93
  docker build -t cosyvoice:v1.0 .
94
- # change multi_lingual_cosytts to multi_emotion_cosytts if you want to use instruct inference
95
- docker run -d --runtime=nvidia -v `pwd`/../../pretrained_models/multi_lingual_cosytts:/opt/cosyvoice/cosyvoice/runtime/pretrained_models -p 50000:50000 cosyvoice:v1.0
96
- python3 client.py --port 50000 --mode <sft|zero_shot|instruct>
97
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  # CosyVoice
2
+ ## 👉🏻 [CosyVoice Demos](https://fun-audio-llm.github.io/) 👈🏻
3
+ [[CosyVoice Paper](https://fun-audio-llm.github.io/pdf/CosyVoice_v1.pdf)][[CosyVoice Studio](https://www.modelscope.cn/studios/iic/CosyVoice-300M)][[CosyVoice Code](https://github.com/FunAudioLLM/CosyVoice)]
4
+
5
+ For `SenseVoice`, visit [SenseVoice repo](https://github.com/FunAudioLLM/SenseVoice) and [SenseVoice space](https://www.modelscope.cn/studios/iic/SenseVoice).
6
 
7
  ## Install
8
 
 
10
 
11
  - Clone the repo
12
  ``` sh
13
+ git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git
14
+ # If you failed to clone submodule due to network failures, please run following command until success
15
+ cd CosyVoice
16
+ git submodule update --init --recursive
17
  ```
18
 
19
  - Install Conda: please see https://docs.conda.io/en/latest/miniconda.html
 
23
  conda create -n cosyvoice python=3.8
24
  conda activate cosyvoice
25
  pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com
26
+
27
+ # If you encounter sox compatibility issues
28
+ # ubuntu
29
+ sudo apt-get install sox libsox-dev
30
+ # centos
31
+ sudo yum install sox sox-devel
32
  ```
33
 
34
  **Model download**
35
 
36
+ We strongly recommand that you download our pretrained `CosyVoice-300M` `CosyVoice-300M-SFT` `CosyVoice-300M-Instruct` model and `speech_kantts_ttsfrd` resource.
37
 
38
  If you are expert in this field, and you are only interested in training your own CosyVoice model from scratch, you can skip this step.
39
 
40
+ ``` python
41
+ # SDK模型下载
42
+ from modelscope import snapshot_download
43
+ snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M')
44
+ snapshot_download('iic/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT')
45
+ snapshot_download('iic/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct')
46
+ snapshot_download('iic/speech_kantts_ttsfrd', local_dir='pretrained_models/speech_kantts_ttsfrd')
47
+ ```
48
+
49
  ``` sh
50
+ # git模型下载,请确保已安装git lfs
51
  mkdir -p pretrained_models
52
+ git clone https://www.modelscope.cn/iic/CosyVoice-300M.git pretrained_models/CosyVoice-300M
53
+ git clone https://www.modelscope.cn/iic/CosyVoice-300M-SFT.git pretrained_models/CosyVoice-300M-SFT
54
+ git clone https://www.modelscope.cn/iic/CosyVoice-300M-Instruct.git pretrained_models/CosyVoice-300M-Instruct
55
+ git clone https://www.modelscope.cn/iic/speech_kantts_ttsfrd.git pretrained_models/speech_kantts_ttsfrd
56
+ ```
57
+
58
+ Unzip `ttsfrd` resouce and install `ttsfrd` package
59
+ ``` sh
60
+ cd pretrained_models/speech_kantts_ttsfrd/
61
+ unzip resource.zip -d .
62
+ pip install ttsfrd-0.3.6-cp38-cp38-linux_x86_64.whl
63
  ```
64
 
65
  **Basic Usage**
66
 
67
+ For zero_shot/cross_lingual inference, please use `CosyVoice-300M` model.
68
+ For sft inference, please use `CosyVoice-300M-SFT` model.
69
+ For instruct inference, please use `CosyVoice-300M-Instruct` model.
70
+ First, add `third_party/AcademiCodec` and `third_party/Matcha-TTS` to your `PYTHONPATH`.
71
 
72
+ ``` sh
73
+ export PYTHONPATH=third_party/AcademiCodec:third_party/Matcha-TTS
74
  ```
75
+
76
+ ``` python
77
  from cosyvoice.cli.cosyvoice import CosyVoice
78
  from cosyvoice.utils.file_utils import load_wav
79
  import torchaudio
80
 
81
+ cosyvoice = CosyVoice('speech_tts/CosyVoice-300M-SFT')
 
82
  # sft usage
83
  print(cosyvoice.list_avaliable_spks())
84
+ output = cosyvoice.inference_sft('你好,我是通义生成式语音大模型,请问有什么可以帮您的吗?', '中文女')
85
  torchaudio.save('sft.wav', output['tts_speech'], 22050)
86
 
87
+ cosyvoice = CosyVoice('speech_tts/CosyVoice-300M')
88
  # zero_shot usage
89
+ prompt_speech_16k = load_wav('zero_shot_prompt.wav', 16000)
90
+ output = cosyvoice.inference_zero_shot('收到好友从远方寄来的生日礼物,那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐,笑容如花儿般绽放。', '希望你以后能够做的比我还好呦。', prompt_speech_16k)
91
  torchaudio.save('zero_shot.wav', output['tts_speech'], 22050)
92
+ # cross_lingual usage
93
+ prompt_speech_16k = load_wav('cross_lingual_prompt.wav', 16000)
94
+ output = cosyvoice.inference_cross_lingual('<|en|>And then later on, fully acquiring that company. So keeping management in line, interest in line with the asset that\'s coming into the family is a reason why sometimes we don\'t buy the whole thing.', prompt_speech_16k)
95
+ torchaudio.save('cross_lingual.wav', output['tts_speech'], 22050)
 
 
 
 
 
 
96
 
97
+ cosyvoice = CosyVoice('speech_tts/CosyVoice-300M-Instruct')
98
  # instruct usage
99
+ output = cosyvoice.inference_instruct('在面对挑战时,他展现了非凡的<strong>勇气</strong>与<strong>智慧</strong>。', '中文男', 'Theo \'Crimson\', is a fiery, passionate rebel leader. Fights with fervor for justice, but struggles with impulsiveness.')
 
100
  torchaudio.save('instruct.wav', output['tts_speech'], 22050)
101
  ```
102
 
 
 
 
 
 
103
  **Start web demo**
104
 
105
  You can use our web demo page to get familiar with CosyVoice quickly.
106
+ We support sft/zero_shot/cross_lingual/instruct inference in web demo.
107
 
108
  Please see the demo website for details.
109
 
110
+ ``` python
111
+ # change speech_tts/CosyVoice-300M-SFT for sft inference, or speech_tts/CosyVoice-300M-Instruct for instruct inference
112
+ python3 webui.py --port 50000 --model_dir speech_tts/CosyVoice-300M
113
  ```
114
+
115
+ **Advanced Usage**
116
+
117
+ For advanced user, we have provided train and inference scripts in `examples/libritts/cosyvoice/run.sh`.
118
+ You can get familiar with CosyVoice following this recipie.
119
 
120
  **Build for deployment**
121
 
 
125
  ``` sh
126
  cd runtime/python
127
  docker build -t cosyvoice:v1.0 .
128
+ # change speech_tts/CosyVoice-300M to speech_tts/CosyVoice-300M-Instruct if you want to use instruct inference
129
+ docker run -d --runtime=nvidia -p 50000:50000 cosyvoice:v1.0 /bin/bash -c "cd /opt/CosyVoice/CosyVoice/runtime/python && python3 server.py --port 50000 --max_conc 4 --model_dir speech_tts/CosyVoice-300M && sleep infinity"
130
+ python3 client.py --port 50000 --mode <sft|zero_shot|cross_lingual|instruct>
131
+ ```
132
+
133
+ ## Discussion & Communication
134
+
135
+ You can directly discuss on [Github Issues](https://github.com/FunAudioLLM/CosyVoice/issues).
136
+
137
+ You can also scan the QR code to join our officla Dingding chat group.
138
+
139
+ <img src="./asset/dingding.png" width="250px">
140
+
141
+ ## Acknowledge
142
+
143
+ 1. We borrowed a lot of code from [FunASR](https://github.com/modelscope/FunASR).
144
+ 2. We borrowed a lot of code from [FunCodec](https://github.com/modelscope/FunCodec).
145
+ 3. We borrowed a lot of code from [Matcha-TTS](https://github.com/shivammehta25/Matcha-TTS).
146
+ 4. We borrowed a lot of code from [AcademiCodec](https://github.com/yangdongchao/AcademiCodec).
147
+ 5. We borrowed a lot of code from [WeNet](https://github.com/wenet-e2e/wenet).
148
+
149
+ ## Disclaimer
150
+ The content provided above is for academic purposes only and is intended to demonstrate technical capabilities. Some examples are sourced from the internet. If any content infringes on your rights, please contact us to request its removal.