Files changed (1) hide show
  1. multi +195 -0
multi ADDED
@@ -0,0 +1,195 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "b6ee1ede",
6
+ "metadata": {},
7
+ "source": [
8
+ "## Cross-Lingual Voice Clone Demo"
9
+ ]
10
+ },
11
+ {
12
+ "cell_type": "code",
13
+ "execution_count": null,
14
+ "id": "b7f043ee",
15
+ "metadata": {},
16
+ "outputs": [],
17
+ "source": [
18
+ "import os\n",
19
+ "import torch\n",
20
+ "import se_extractor\n",
21
+ "from api import ToneColorConverter"
22
+ ]
23
+ },
24
+ {
25
+ "cell_type": "markdown",
26
+ "id": "15116b59",
27
+ "metadata": {},
28
+ "source": [
29
+ "### Initialization"
30
+ ]
31
+ },
32
+ {
33
+ "cell_type": "code",
34
+ "execution_count": null,
35
+ "id": "aacad912",
36
+ "metadata": {},
37
+ "outputs": [],
38
+ "source": [
39
+ "ckpt_converter = 'checkpoints/converter'\n",
40
+ "device = 'cuda:0'\n",
41
+ "output_dir = 'outputs'\n",
42
+ "\n",
43
+ "tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)\n",
44
+ "tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')\n",
45
+ "\n",
46
+ "os.makedirs(output_dir, exist_ok=True)"
47
+ ]
48
+ },
49
+ {
50
+ "cell_type": "markdown",
51
+ "id": "3db80fcf",
52
+ "metadata": {},
53
+ "source": [
54
+ "In this demo, we will use OpenAI TTS as the base speaker to produce multi-lingual speech audio. The users can flexibly change the base speaker according to their own needs. Please create a file named `.env` and place OpenAI key as `OPENAI_API_KEY=xxx`. We have also provided a Chinese base speaker model (see `demo_part1.ipynb`)."
55
+ ]
56
+ },
57
+ {
58
+ "cell_type": "code",
59
+ "execution_count": null,
60
+ "id": "3b245ca3",
61
+ "metadata": {},
62
+ "outputs": [],
63
+ "source": [
64
+ "from openai import OpenAI\n",
65
+ "from dotenv import load_dotenv\n",
66
+ "\n",
67
+ "# Please create a file named .env and place your\n",
68
+ "# OpenAI key as OPENAI_API_KEY=xxx\n",
69
+ "load_dotenv() \n",
70
+ "\n",
71
+ "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n",
72
+ "\n",
73
+ "response = client.audio.speech.create(\n",
74
+ " model=\"tts-1\",\n",
75
+ " voice=\"nova\",\n",
76
+ " input=\"This audio will be used to extract the base speaker tone color embedding. \" + \\\n",
77
+ " \"Typically a very short audio should be sufficient, but increasing the audio \" + \\\n",
78
+ " \"length will also improve the output audio quality.\"\n",
79
+ ")\n",
80
+ "\n",
81
+ "response.stream_to_file(f\"{output_dir}/openai_source_output.mp3\")"
82
+ ]
83
+ },
84
+ {
85
+ "cell_type": "markdown",
86
+ "id": "7f67740c",
87
+ "metadata": {},
88
+ "source": [
89
+ "### Obtain Tone Color Embedding"
90
+ ]
91
+ },
92
+ {
93
+ "cell_type": "markdown",
94
+ "id": "f8add279",
95
+ "metadata": {},
96
+ "source": [
97
+ "The `source_se` is the tone color embedding of the base speaker. \n",
98
+ "It is an average for multiple sentences with multiple emotions\n",
99
+ "of the base speaker. We directly provide the result here but\n",
100
+ "the readers feel free to extract `source_se` by themselves."
101
+ ]
102
+ },
103
+ {
104
+ "cell_type": "code",
105
+ "execution_count": null,
106
+ "id": "63ff6273",
107
+ "metadata": {},
108
+ "outputs": [],
109
+ "source": [
110
+ "base_speaker = f\"{output_dir}/openai_source_output.mp3\"\n",
111
+ "source_se, audio_name = se_extractor.get_se(base_speaker, tone_color_converter, vad=True)\n",
112
+ "\n",
113
+ "reference_speaker = 'resources/example_reference.mp3'\n",
114
+ "target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter, vad=True)"
115
+ ]
116
+ },
117
+ {
118
+ "cell_type": "markdown",
119
+ "id": "a40284aa",
120
+ "metadata": {},
121
+ "source": [
122
+ "### Inference"
123
+ ]
124
+ },
125
+ {
126
+ "cell_type": "code",
127
+ "execution_count": null,
128
+ "id": "73dc1259",
129
+ "metadata": {},
130
+ "outputs": [],
131
+ "source": [
132
+ "# Run the base speaker tts\n",
133
+ "text = [\n",
134
+ " \"MyShell is a decentralized and comprehensive platform for discovering, creating, and staking AI-native apps.\",\n",
135
+ " \"MyShell es una plataforma descentralizada y completa para descubrir, crear y apostar por aplicaciones nativas de IA.\",\n",
136
+ " \"MyShell est une plateforme décentralisée et complète pour découvrir, créer et miser sur des applications natives d'IA.\",\n",
137
+ " \"MyShell ist eine dezentralisierte und umfassende Plattform zum Entdecken, Erstellen und Staken von KI-nativen Apps.\",\n",
138
+ " \"MyShell è una piattaforma decentralizzata e completa per scoprire, creare e scommettere su app native di intelligenza artificiale.\",\n",
139
+ " \"MyShellは、AIネイティブアプリの発見、作成、およびステーキングのための分散型かつ包括的なプラットフォームです。\",\n",
140
+ " \"MyShell — это децентрализованная и всеобъемлющая платформа для обнаружения, создания и стейкинга AI-ориентированных приложений.\",\n",
141
+ " \"MyShell هي منصة لامركزية وشاملة لاكتشاف وإنشاء ورهان تطبيقات الذكاء الاصطناعي الأصلية.\",\n",
142
+ " \"MyShell是一个去中心化且全面的平台,用于发现、创建和投资AI原生应用程序。\",\n",
143
+ " \"MyShell एक विकेंद्रीकृत और व्यापक मंच है, जो AI-मूल ऐप्स की खोज, सृजन और स्टेकिंग के लिए है।\",\n",
144
+ " \"MyShell é uma plataforma descentralizada e abrangente para descobrir, criar e apostar em aplicativos nativos de IA.\"\n",
145
+ "]\n",
146
+ "src_path = f'{output_dir}/tmp.wav'\n",
147
+ "\n",
148
+ "for i, t in enumerate(text):\n",
149
+ "\n",
150
+ " response = client.audio.speech.create(\n",
151
+ " model=\"tts-1\",\n",
152
+ " voice=\"alloy\",\n",
153
+ " input=t,\n",
154
+ " )\n",
155
+ "\n",
156
+ " response.stream_to_file(src_path)\n",
157
+ "\n",
158
+ " save_path = f'{output_dir}/output_crosslingual_{i}.wav'\n",
159
+ "\n",
160
+ " # Run the tone color converter\n",
161
+ " encode_message = \"@MyShell\"\n",
162
+ " tone_color_converter.convert(\n",
163
+ " audio_src_path=src_path, \n",
164
+ " src_se=source_se, \n",
165
+ " tgt_se=target_se, \n",
166
+ " output_path=save_path,\n",
167
+ " message=encode_message)"
168
+ ]
169
+ }
170
+ ],
171
+ "metadata": {
172
+ "interpreter": {
173
+ "hash": "9d70c38e1c0b038dbdffdaa4f8bfa1f6767c43760905c87a9fbe7800d18c6c35"
174
+ },
175
+ "kernelspec": {
176
+ "display_name": "Python 3.9.18 ('openvoice')",
177
+ "language": "python",
178
+ "name": "python3"
179
+ },
180
+ "language_info": {
181
+ "codemirror_mode": {
182
+ "name": "ipython",
183
+ "version": 3
184
+ },
185
+ "file_extension": ".py",
186
+ "mimetype": "text/x-python",
187
+ "name": "python",
188
+ "nbconvert_exporter": "python",
189
+ "pygments_lexer": "ipython3",
190
+ "version": "3.9.18"
191
+ }
192
+ },
193
+ "nbformat": 4,
194
+ "nbformat_minor": 5
195
+ }