File size: 6,738 Bytes
4318c41
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b6ee1ede",
   "metadata": {},
   "source": [
    "## Cross-Lingual Voice Clone Demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7f043ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import torch\n",
    "import se_extractor\n",
    "from api import ToneColorConverter"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15116b59",
   "metadata": {},
   "source": [
    "### Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aacad912",
   "metadata": {},
   "outputs": [],
   "source": [
    "ckpt_converter = 'checkpoints/converter'\n",
    "device = 'cuda:0'\n",
    "output_dir = 'outputs'\n",
    "\n",
    "tone_color_converter = ToneColorConverter(f'{ckpt_converter}/config.json', device=device)\n",
    "tone_color_converter.load_ckpt(f'{ckpt_converter}/checkpoint.pth')\n",
    "\n",
    "os.makedirs(output_dir, exist_ok=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3db80fcf",
   "metadata": {},
   "source": [
    "In this demo, we will use OpenAI TTS as the base speaker to produce multi-lingual speech audio. The users can flexibly change the base speaker according to their own needs. Please create a file named `.env` and place OpenAI key as `OPENAI_API_KEY=xxx`. We have also provided a Chinese base speaker model (see `demo_part1.ipynb`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b245ca3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Please create a file named .env and place your\n",
    "# OpenAI key as OPENAI_API_KEY=xxx\n",
    "load_dotenv() \n",
    "\n",
    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\"))\n",
    "\n",
    "response = client.audio.speech.create(\n",
    "    model=\"tts-1\",\n",
    "    voice=\"nova\",\n",
    "    input=\"This audio will be used to extract the base speaker tone color embedding. \" + \\\n",
    "        \"Typically a very short audio should be sufficient, but increasing the audio \" + \\\n",
    "        \"length will also improve the output audio quality.\"\n",
    ")\n",
    "\n",
    "response.stream_to_file(f\"{output_dir}/openai_source_output.mp3\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f67740c",
   "metadata": {},
   "source": [
    "### Obtain Tone Color Embedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8add279",
   "metadata": {},
   "source": [
    "The `source_se` is the tone color embedding of the base speaker. \n",
    "It is an average for multiple sentences with multiple emotions\n",
    "of the base speaker. We directly provide the result here but\n",
    "the readers feel free to extract `source_se` by themselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63ff6273",
   "metadata": {},
   "outputs": [],
   "source": [
    "base_speaker = f\"{output_dir}/openai_source_output.mp3\"\n",
    "source_se, audio_name = se_extractor.get_se(base_speaker, tone_color_converter, vad=True)\n",
    "\n",
    "reference_speaker = 'resources/example_reference.mp3'\n",
    "target_se, audio_name = se_extractor.get_se(reference_speaker, tone_color_converter, vad=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a40284aa",
   "metadata": {},
   "source": [
    "### Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "73dc1259",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run the base speaker tts\n",
    "text = [\n",
    "    \"MyShell is a decentralized and comprehensive platform for discovering, creating, and staking AI-native apps.\",\n",
    "    \"MyShell es una plataforma descentralizada y completa para descubrir, crear y apostar por aplicaciones nativas de IA.\",\n",
    "    \"MyShell est une plateforme décentralisée et complète pour découvrir, créer et miser sur des applications natives d'IA.\",\n",
    "    \"MyShell ist eine dezentralisierte und umfassende Plattform zum Entdecken, Erstellen und Staken von KI-nativen Apps.\",\n",
    "    \"MyShell è una piattaforma decentralizzata e completa per scoprire, creare e scommettere su app native di intelligenza artificiale.\",\n",
    "    \"MyShellは、AIネイティブアプリの発見、作成、およびステーキングのための分散型かつ包括的なプラットフォームです。\",\n",
    "    \"MyShell — это децентрализованная и всеобъемлющая платформа для обнаружения, создания и стейкинга AI-ориентированных приложений.\",\n",
    "    \"MyShell هي منصة لامركزية وشاملة لاكتشاف وإنشاء ورهان تطبيقات الذكاء الاصطناعي الأصلية.\",\n",
    "    \"MyShell是一个去中心化且全面的平台,用于发现、创建和投资AI原生应用程序。\",\n",
    "    \"MyShell एक विकेंद्रीकृत और व्यापक मंच है, जो AI-मूल ऐप्स की खोज, सृजन और स्टेकिंग के लिए है।\",\n",
    "    \"MyShell é uma plataforma descentralizada e abrangente para descobrir, criar e apostar em aplicativos nativos de IA.\"\n",
    "]\n",
    "src_path = f'{output_dir}/tmp.wav'\n",
    "\n",
    "for i, t in enumerate(text):\n",
    "\n",
    "    response = client.audio.speech.create(\n",
    "        model=\"tts-1\",\n",
    "        voice=\"alloy\",\n",
    "        input=t,\n",
    "    )\n",
    "\n",
    "    response.stream_to_file(src_path)\n",
    "\n",
    "    save_path = f'{output_dir}/output_crosslingual_{i}.wav'\n",
    "\n",
    "    # Run the tone color converter\n",
    "    encode_message = \"@MyShell\"\n",
    "    tone_color_converter.convert(\n",
    "        audio_src_path=src_path, \n",
    "        src_se=source_se, \n",
    "        tgt_se=target_se, \n",
    "        output_path=save_path,\n",
    "        message=encode_message)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "9d70c38e1c0b038dbdffdaa4f8bfa1f6767c43760905c87a9fbe7800d18c6c35"
  },
  "kernelspec": {
   "display_name": "Python 3.9.18 ('openvoice')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}