Roi Feng commited on
Commit
e526878
1 Parent(s): 6b0bbcb

initialize hanser

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .pre-commit-config.yaml +25 -0
  2. LICENSE +661 -0
  3. README.md +1 -1
  4. app.py +228 -0
  5. attentions.py +464 -0
  6. bert/bert-base-japanese-v3/README.md +53 -0
  7. bert/bert-base-japanese-v3/config.json +19 -0
  8. bert/bert-base-japanese-v3/flax_model.msgpack +3 -0
  9. bert/bert-base-japanese-v3/pytorch_model.bin +3 -0
  10. bert/bert-base-japanese-v3/tf_model.h5 +3 -0
  11. bert/bert-base-japanese-v3/tokenizer_config.json +10 -0
  12. bert/bert-base-japanese-v3/vocab.txt +0 -0
  13. bert/chinese-roberta-wwm-ext-large/.gitattributes +9 -0
  14. bert/chinese-roberta-wwm-ext-large/.gitignore +1 -0
  15. bert/chinese-roberta-wwm-ext-large/README.md +57 -0
  16. bert/chinese-roberta-wwm-ext-large/added_tokens.json +1 -0
  17. bert/chinese-roberta-wwm-ext-large/config.json +28 -0
  18. bert/chinese-roberta-wwm-ext-large/flax_model.msgpack +3 -0
  19. bert/chinese-roberta-wwm-ext-large/pytorch_model.bin +3 -0
  20. bert/chinese-roberta-wwm-ext-large/special_tokens_map.json +1 -0
  21. bert/chinese-roberta-wwm-ext-large/tf_model.h5 +3 -0
  22. bert/chinese-roberta-wwm-ext-large/tokenizer.json +0 -0
  23. bert/chinese-roberta-wwm-ext-large/tokenizer_config.json +1 -0
  24. bert/chinese-roberta-wwm-ext-large/vocab.txt +0 -0
  25. bert_gen.py +59 -0
  26. commons.py +160 -0
  27. configs/config.json +95 -0
  28. data_utils.py +406 -0
  29. logs/hanser/G_8000.pth +3 -0
  30. losses.py +58 -0
  31. mel_processing.py +139 -0
  32. models.py +986 -0
  33. modules.py +597 -0
  34. monotonic_align/__init__.py +16 -0
  35. monotonic_align/core.py +46 -0
  36. preprocess_text.py +105 -0
  37. requirements.txt +23 -0
  38. resample.py +48 -0
  39. server.py +170 -0
  40. text/__init__.py +28 -0
  41. text/chinese.py +198 -0
  42. text/chinese_bert.py +100 -0
  43. text/cleaner.py +28 -0
  44. text/cmudict.rep +0 -0
  45. text/cmudict_cache.pickle +3 -0
  46. text/english.py +214 -0
  47. text/english_bert_mock.py +5 -0
  48. text/japanese.py +586 -0
  49. text/japanese_bert.py +38 -0
  50. text/opencpop-strict.txt +429 -0
.pre-commit-config.yaml ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ repos:
2
+ - repo: https://github.com/pre-commit/pre-commit-hooks
3
+ rev: v4.4.0
4
+ hooks:
5
+ - id: check-yaml
6
+ - id: end-of-file-fixer
7
+ - id: trailing-whitespace
8
+
9
+ - repo: https://github.com/astral-sh/ruff-pre-commit
10
+ rev: v0.0.292
11
+ hooks:
12
+ - id: ruff
13
+ args: [ --fix ]
14
+
15
+ - repo: https://github.com/psf/black
16
+ rev: 23.9.1
17
+ hooks:
18
+ - id: black
19
+
20
+ - repo: https://github.com/codespell-project/codespell
21
+ rev: v2.2.6
22
+ hooks:
23
+ - id: codespell
24
+ files: ^.*\.(py|md|rst|yml)$
25
+ args: [-L=fro]
LICENSE ADDED
@@ -0,0 +1,661 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ GNU AFFERO GENERAL PUBLIC LICENSE
2
+ Version 3, 19 November 2007
3
+
4
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
5
+ Everyone is permitted to copy and distribute verbatim copies
6
+ of this license document, but changing it is not allowed.
7
+
8
+ Preamble
9
+
10
+ The GNU Affero General Public License is a free, copyleft license for
11
+ software and other kinds of works, specifically designed to ensure
12
+ cooperation with the community in the case of network server software.
13
+
14
+ The licenses for most software and other practical works are designed
15
+ to take away your freedom to share and change the works. By contrast,
16
+ our General Public Licenses are intended to guarantee your freedom to
17
+ share and change all versions of a program--to make sure it remains free
18
+ software for all its users.
19
+
20
+ When we speak of free software, we are referring to freedom, not
21
+ price. Our General Public Licenses are designed to make sure that you
22
+ have the freedom to distribute copies of free software (and charge for
23
+ them if you wish), that you receive source code or can get it if you
24
+ want it, that you can change the software or use pieces of it in new
25
+ free programs, and that you know you can do these things.
26
+
27
+ Developers that use our General Public Licenses protect your rights
28
+ with two steps: (1) assert copyright on the software, and (2) offer
29
+ you this License which gives you legal permission to copy, distribute
30
+ and/or modify the software.
31
+
32
+ A secondary benefit of defending all users' freedom is that
33
+ improvements made in alternate versions of the program, if they
34
+ receive widespread use, become available for other developers to
35
+ incorporate. Many developers of free software are heartened and
36
+ encouraged by the resulting cooperation. However, in the case of
37
+ software used on network servers, this result may fail to come about.
38
+ The GNU General Public License permits making a modified version and
39
+ letting the public access it on a server without ever releasing its
40
+ source code to the public.
41
+
42
+ The GNU Affero General Public License is designed specifically to
43
+ ensure that, in such cases, the modified source code becomes available
44
+ to the community. It requires the operator of a network server to
45
+ provide the source code of the modified version running there to the
46
+ users of that server. Therefore, public use of a modified version, on
47
+ a publicly accessible server, gives the public access to the source
48
+ code of the modified version.
49
+
50
+ An older license, called the Affero General Public License and
51
+ published by Affero, was designed to accomplish similar goals. This is
52
+ a different license, not a version of the Affero GPL, but Affero has
53
+ released a new version of the Affero GPL which permits relicensing under
54
+ this license.
55
+
56
+ The precise terms and conditions for copying, distribution and
57
+ modification follow.
58
+
59
+ TERMS AND CONDITIONS
60
+
61
+ 0. Definitions.
62
+
63
+ "This License" refers to version 3 of the GNU Affero General Public License.
64
+
65
+ "Copyright" also means copyright-like laws that apply to other kinds of
66
+ works, such as semiconductor masks.
67
+
68
+ "The Program" refers to any copyrightable work licensed under this
69
+ License. Each licensee is addressed as "you". "Licensees" and
70
+ "recipients" may be individuals or organizations.
71
+
72
+ To "modify" a work means to copy from or adapt all or part of the work
73
+ in a fashion requiring copyright permission, other than the making of an
74
+ exact copy. The resulting work is called a "modified version" of the
75
+ earlier work or a work "based on" the earlier work.
76
+
77
+ A "covered work" means either the unmodified Program or a work based
78
+ on the Program.
79
+
80
+ To "propagate" a work means to do anything with it that, without
81
+ permission, would make you directly or secondarily liable for
82
+ infringement under applicable copyright law, except executing it on a
83
+ computer or modifying a private copy. Propagation includes copying,
84
+ distribution (with or without modification), making available to the
85
+ public, and in some countries other activities as well.
86
+
87
+ To "convey" a work means any kind of propagation that enables other
88
+ parties to make or receive copies. Mere interaction with a user through
89
+ a computer network, with no transfer of a copy, is not conveying.
90
+
91
+ An interactive user interface displays "Appropriate Legal Notices"
92
+ to the extent that it includes a convenient and prominently visible
93
+ feature that (1) displays an appropriate copyright notice, and (2)
94
+ tells the user that there is no warranty for the work (except to the
95
+ extent that warranties are provided), that licensees may convey the
96
+ work under this License, and how to view a copy of this License. If
97
+ the interface presents a list of user commands or options, such as a
98
+ menu, a prominent item in the list meets this criterion.
99
+
100
+ 1. Source Code.
101
+
102
+ The "source code" for a work means the preferred form of the work
103
+ for making modifications to it. "Object code" means any non-source
104
+ form of a work.
105
+
106
+ A "Standard Interface" means an interface that either is an official
107
+ standard defined by a recognized standards body, or, in the case of
108
+ interfaces specified for a particular programming language, one that
109
+ is widely used among developers working in that language.
110
+
111
+ The "System Libraries" of an executable work include anything, other
112
+ than the work as a whole, that (a) is included in the normal form of
113
+ packaging a Major Component, but which is not part of that Major
114
+ Component, and (b) serves only to enable use of the work with that
115
+ Major Component, or to implement a Standard Interface for which an
116
+ implementation is available to the public in source code form. A
117
+ "Major Component", in this context, means a major essential component
118
+ (kernel, window system, and so on) of the specific operating system
119
+ (if any) on which the executable work runs, or a compiler used to
120
+ produce the work, or an object code interpreter used to run it.
121
+
122
+ The "Corresponding Source" for a work in object code form means all
123
+ the source code needed to generate, install, and (for an executable
124
+ work) run the object code and to modify the work, including scripts to
125
+ control those activities. However, it does not include the work's
126
+ System Libraries, or general-purpose tools or generally available free
127
+ programs which are used unmodified in performing those activities but
128
+ which are not part of the work. For example, Corresponding Source
129
+ includes interface definition files associated with source files for
130
+ the work, and the source code for shared libraries and dynamically
131
+ linked subprograms that the work is specifically designed to require,
132
+ such as by intimate data communication or control flow between those
133
+ subprograms and other parts of the work.
134
+
135
+ The Corresponding Source need not include anything that users
136
+ can regenerate automatically from other parts of the Corresponding
137
+ Source.
138
+
139
+ The Corresponding Source for a work in source code form is that
140
+ same work.
141
+
142
+ 2. Basic Permissions.
143
+
144
+ All rights granted under this License are granted for the term of
145
+ copyright on the Program, and are irrevocable provided the stated
146
+ conditions are met. This License explicitly affirms your unlimited
147
+ permission to run the unmodified Program. The output from running a
148
+ covered work is covered by this License only if the output, given its
149
+ content, constitutes a covered work. This License acknowledges your
150
+ rights of fair use or other equivalent, as provided by copyright law.
151
+
152
+ You may make, run and propagate covered works that you do not
153
+ convey, without conditions so long as your license otherwise remains
154
+ in force. You may convey covered works to others for the sole purpose
155
+ of having them make modifications exclusively for you, or provide you
156
+ with facilities for running those works, provided that you comply with
157
+ the terms of this License in conveying all material for which you do
158
+ not control copyright. Those thus making or running the covered works
159
+ for you must do so exclusively on your behalf, under your direction
160
+ and control, on terms that prohibit them from making any copies of
161
+ your copyrighted material outside their relationship with you.
162
+
163
+ Conveying under any other circumstances is permitted solely under
164
+ the conditions stated below. Sublicensing is not allowed; section 10
165
+ makes it unnecessary.
166
+
167
+ 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
168
+
169
+ No covered work shall be deemed part of an effective technological
170
+ measure under any applicable law fulfilling obligations under article
171
+ 11 of the WIPO copyright treaty adopted on 20 December 1996, or
172
+ similar laws prohibiting or restricting circumvention of such
173
+ measures.
174
+
175
+ When you convey a covered work, you waive any legal power to forbid
176
+ circumvention of technological measures to the extent such circumvention
177
+ is effected by exercising rights under this License with respect to
178
+ the covered work, and you disclaim any intention to limit operation or
179
+ modification of the work as a means of enforcing, against the work's
180
+ users, your or third parties' legal rights to forbid circumvention of
181
+ technological measures.
182
+
183
+ 4. Conveying Verbatim Copies.
184
+
185
+ You may convey verbatim copies of the Program's source code as you
186
+ receive it, in any medium, provided that you conspicuously and
187
+ appropriately publish on each copy an appropriate copyright notice;
188
+ keep intact all notices stating that this License and any
189
+ non-permissive terms added in accord with section 7 apply to the code;
190
+ keep intact all notices of the absence of any warranty; and give all
191
+ recipients a copy of this License along with the Program.
192
+
193
+ You may charge any price or no price for each copy that you convey,
194
+ and you may offer support or warranty protection for a fee.
195
+
196
+ 5. Conveying Modified Source Versions.
197
+
198
+ You may convey a work based on the Program, or the modifications to
199
+ produce it from the Program, in the form of source code under the
200
+ terms of section 4, provided that you also meet all of these conditions:
201
+
202
+ a) The work must carry prominent notices stating that you modified
203
+ it, and giving a relevant date.
204
+
205
+ b) The work must carry prominent notices stating that it is
206
+ released under this License and any conditions added under section
207
+ 7. This requirement modifies the requirement in section 4 to
208
+ "keep intact all notices".
209
+
210
+ c) You must license the entire work, as a whole, under this
211
+ License to anyone who comes into possession of a copy. This
212
+ License will therefore apply, along with any applicable section 7
213
+ additional terms, to the whole of the work, and all its parts,
214
+ regardless of how they are packaged. This License gives no
215
+ permission to license the work in any other way, but it does not
216
+ invalidate such permission if you have separately received it.
217
+
218
+ d) If the work has interactive user interfaces, each must display
219
+ Appropriate Legal Notices; however, if the Program has interactive
220
+ interfaces that do not display Appropriate Legal Notices, your
221
+ work need not make them do so.
222
+
223
+ A compilation of a covered work with other separate and independent
224
+ works, which are not by their nature extensions of the covered work,
225
+ and which are not combined with it such as to form a larger program,
226
+ in or on a volume of a storage or distribution medium, is called an
227
+ "aggregate" if the compilation and its resulting copyright are not
228
+ used to limit the access or legal rights of the compilation's users
229
+ beyond what the individual works permit. Inclusion of a covered work
230
+ in an aggregate does not cause this License to apply to the other
231
+ parts of the aggregate.
232
+
233
+ 6. Conveying Non-Source Forms.
234
+
235
+ You may convey a covered work in object code form under the terms
236
+ of sections 4 and 5, provided that you also convey the
237
+ machine-readable Corresponding Source under the terms of this License,
238
+ in one of these ways:
239
+
240
+ a) Convey the object code in, or embodied in, a physical product
241
+ (including a physical distribution medium), accompanied by the
242
+ Corresponding Source fixed on a durable physical medium
243
+ customarily used for software interchange.
244
+
245
+ b) Convey the object code in, or embodied in, a physical product
246
+ (including a physical distribution medium), accompanied by a
247
+ written offer, valid for at least three years and valid for as
248
+ long as you offer spare parts or customer support for that product
249
+ model, to give anyone who possesses the object code either (1) a
250
+ copy of the Corresponding Source for all the software in the
251
+ product that is covered by this License, on a durable physical
252
+ medium customarily used for software interchange, for a price no
253
+ more than your reasonable cost of physically performing this
254
+ conveying of source, or (2) access to copy the
255
+ Corresponding Source from a network server at no charge.
256
+
257
+ c) Convey individual copies of the object code with a copy of the
258
+ written offer to provide the Corresponding Source. This
259
+ alternative is allowed only occasionally and noncommercially, and
260
+ only if you received the object code with such an offer, in accord
261
+ with subsection 6b.
262
+
263
+ d) Convey the object code by offering access from a designated
264
+ place (gratis or for a charge), and offer equivalent access to the
265
+ Corresponding Source in the same way through the same place at no
266
+ further charge. You need not require recipients to copy the
267
+ Corresponding Source along with the object code. If the place to
268
+ copy the object code is a network server, the Corresponding Source
269
+ may be on a different server (operated by you or a third party)
270
+ that supports equivalent copying facilities, provided you maintain
271
+ clear directions next to the object code saying where to find the
272
+ Corresponding Source. Regardless of what server hosts the
273
+ Corresponding Source, you remain obligated to ensure that it is
274
+ available for as long as needed to satisfy these requirements.
275
+
276
+ e) Convey the object code using peer-to-peer transmission, provided
277
+ you inform other peers where the object code and Corresponding
278
+ Source of the work are being offered to the general public at no
279
+ charge under subsection 6d.
280
+
281
+ A separable portion of the object code, whose source code is excluded
282
+ from the Corresponding Source as a System Library, need not be
283
+ included in conveying the object code work.
284
+
285
+ A "User Product" is either (1) a "consumer product", which means any
286
+ tangible personal property which is normally used for personal, family,
287
+ or household purposes, or (2) anything designed or sold for incorporation
288
+ into a dwelling. In determining whether a product is a consumer product,
289
+ doubtful cases shall be resolved in favor of coverage. For a particular
290
+ product received by a particular user, "normally used" refers to a
291
+ typical or common use of that class of product, regardless of the status
292
+ of the particular user or of the way in which the particular user
293
+ actually uses, or expects or is expected to use, the product. A product
294
+ is a consumer product regardless of whether the product has substantial
295
+ commercial, industrial or non-consumer uses, unless such uses represent
296
+ the only significant mode of use of the product.
297
+
298
+ "Installation Information" for a User Product means any methods,
299
+ procedures, authorization keys, or other information required to install
300
+ and execute modified versions of a covered work in that User Product from
301
+ a modified version of its Corresponding Source. The information must
302
+ suffice to ensure that the continued functioning of the modified object
303
+ code is in no case prevented or interfered with solely because
304
+ modification has been made.
305
+
306
+ If you convey an object code work under this section in, or with, or
307
+ specifically for use in, a User Product, and the conveying occurs as
308
+ part of a transaction in which the right of possession and use of the
309
+ User Product is transferred to the recipient in perpetuity or for a
310
+ fixed term (regardless of how the transaction is characterized), the
311
+ Corresponding Source conveyed under this section must be accompanied
312
+ by the Installation Information. But this requirement does not apply
313
+ if neither you nor any third party retains the ability to install
314
+ modified object code on the User Product (for example, the work has
315
+ been installed in ROM).
316
+
317
+ The requirement to provide Installation Information does not include a
318
+ requirement to continue to provide support service, warranty, or updates
319
+ for a work that has been modified or installed by the recipient, or for
320
+ the User Product in which it has been modified or installed. Access to a
321
+ network may be denied when the modification itself materially and
322
+ adversely affects the operation of the network or violates the rules and
323
+ protocols for communication across the network.
324
+
325
+ Corresponding Source conveyed, and Installation Information provided,
326
+ in accord with this section must be in a format that is publicly
327
+ documented (and with an implementation available to the public in
328
+ source code form), and must require no special password or key for
329
+ unpacking, reading or copying.
330
+
331
+ 7. Additional Terms.
332
+
333
+ "Additional permissions" are terms that supplement the terms of this
334
+ License by making exceptions from one or more of its conditions.
335
+ Additional permissions that are applicable to the entire Program shall
336
+ be treated as though they were included in this License, to the extent
337
+ that they are valid under applicable law. If additional permissions
338
+ apply only to part of the Program, that part may be used separately
339
+ under those permissions, but the entire Program remains governed by
340
+ this License without regard to the additional permissions.
341
+
342
+ When you convey a copy of a covered work, you may at your option
343
+ remove any additional permissions from that copy, or from any part of
344
+ it. (Additional permissions may be written to require their own
345
+ removal in certain cases when you modify the work.) You may place
346
+ additional permissions on material, added by you to a covered work,
347
+ for which you have or can give appropriate copyright permission.
348
+
349
+ Notwithstanding any other provision of this License, for material you
350
+ add to a covered work, you may (if authorized by the copyright holders of
351
+ that material) supplement the terms of this License with terms:
352
+
353
+ a) Disclaiming warranty or limiting liability differently from the
354
+ terms of sections 15 and 16 of this License; or
355
+
356
+ b) Requiring preservation of specified reasonable legal notices or
357
+ author attributions in that material or in the Appropriate Legal
358
+ Notices displayed by works containing it; or
359
+
360
+ c) Prohibiting misrepresentation of the origin of that material, or
361
+ requiring that modified versions of such material be marked in
362
+ reasonable ways as different from the original version; or
363
+
364
+ d) Limiting the use for publicity purposes of names of licensors or
365
+ authors of the material; or
366
+
367
+ e) Declining to grant rights under trademark law for use of some
368
+ trade names, trademarks, or service marks; or
369
+
370
+ f) Requiring indemnification of licensors and authors of that
371
+ material by anyone who conveys the material (or modified versions of
372
+ it) with contractual assumptions of liability to the recipient, for
373
+ any liability that these contractual assumptions directly impose on
374
+ those licensors and authors.
375
+
376
+ All other non-permissive additional terms are considered "further
377
+ restrictions" within the meaning of section 10. If the Program as you
378
+ received it, or any part of it, contains a notice stating that it is
379
+ governed by this License along with a term that is a further
380
+ restriction, you may remove that term. If a license document contains
381
+ a further restriction but permits relicensing or conveying under this
382
+ License, you may add to a covered work material governed by the terms
383
+ of that license document, provided that the further restriction does
384
+ not survive such relicensing or conveying.
385
+
386
+ If you add terms to a covered work in accord with this section, you
387
+ must place, in the relevant source files, a statement of the
388
+ additional terms that apply to those files, or a notice indicating
389
+ where to find the applicable terms.
390
+
391
+ Additional terms, permissive or non-permissive, may be stated in the
392
+ form of a separately written license, or stated as exceptions;
393
+ the above requirements apply either way.
394
+
395
+ 8. Termination.
396
+
397
+ You may not propagate or modify a covered work except as expressly
398
+ provided under this License. Any attempt otherwise to propagate or
399
+ modify it is void, and will automatically terminate your rights under
400
+ this License (including any patent licenses granted under the third
401
+ paragraph of section 11).
402
+
403
+ However, if you cease all violation of this License, then your
404
+ license from a particular copyright holder is reinstated (a)
405
+ provisionally, unless and until the copyright holder explicitly and
406
+ finally terminates your license, and (b) permanently, if the copyright
407
+ holder fails to notify you of the violation by some reasonable means
408
+ prior to 60 days after the cessation.
409
+
410
+ Moreover, your license from a particular copyright holder is
411
+ reinstated permanently if the copyright holder notifies you of the
412
+ violation by some reasonable means, this is the first time you have
413
+ received notice of violation of this License (for any work) from that
414
+ copyright holder, and you cure the violation prior to 30 days after
415
+ your receipt of the notice.
416
+
417
+ Termination of your rights under this section does not terminate the
418
+ licenses of parties who have received copies or rights from you under
419
+ this License. If your rights have been terminated and not permanently
420
+ reinstated, you do not qualify to receive new licenses for the same
421
+ material under section 10.
422
+
423
+ 9. Acceptance Not Required for Having Copies.
424
+
425
+ You are not required to accept this License in order to receive or
426
+ run a copy of the Program. Ancillary propagation of a covered work
427
+ occurring solely as a consequence of using peer-to-peer transmission
428
+ to receive a copy likewise does not require acceptance. However,
429
+ nothing other than this License grants you permission to propagate or
430
+ modify any covered work. These actions infringe copyright if you do
431
+ not accept this License. Therefore, by modifying or propagating a
432
+ covered work, you indicate your acceptance of this License to do so.
433
+
434
+ 10. Automatic Licensing of Downstream Recipients.
435
+
436
+ Each time you convey a covered work, the recipient automatically
437
+ receives a license from the original licensors, to run, modify and
438
+ propagate that work, subject to this License. You are not responsible
439
+ for enforcing compliance by third parties with this License.
440
+
441
+ An "entity transaction" is a transaction transferring control of an
442
+ organization, or substantially all assets of one, or subdividing an
443
+ organization, or merging organizations. If propagation of a covered
444
+ work results from an entity transaction, each party to that
445
+ transaction who receives a copy of the work also receives whatever
446
+ licenses to the work the party's predecessor in interest had or could
447
+ give under the previous paragraph, plus a right to possession of the
448
+ Corresponding Source of the work from the predecessor in interest, if
449
+ the predecessor has it or can get it with reasonable efforts.
450
+
451
+ You may not impose any further restrictions on the exercise of the
452
+ rights granted or affirmed under this License. For example, you may
453
+ not impose a license fee, royalty, or other charge for exercise of
454
+ rights granted under this License, and you may not initiate litigation
455
+ (including a cross-claim or counterclaim in a lawsuit) alleging that
456
+ any patent claim is infringed by making, using, selling, offering for
457
+ sale, or importing the Program or any portion of it.
458
+
459
+ 11. Patents.
460
+
461
+ A "contributor" is a copyright holder who authorizes use under this
462
+ License of the Program or a work on which the Program is based. The
463
+ work thus licensed is called the contributor's "contributor version".
464
+
465
+ A contributor's "essential patent claims" are all patent claims
466
+ owned or controlled by the contributor, whether already acquired or
467
+ hereafter acquired, that would be infringed by some manner, permitted
468
+ by this License, of making, using, or selling its contributor version,
469
+ but do not include claims that would be infringed only as a
470
+ consequence of further modification of the contributor version. For
471
+ purposes of this definition, "control" includes the right to grant
472
+ patent sublicenses in a manner consistent with the requirements of
473
+ this License.
474
+
475
+ Each contributor grants you a non-exclusive, worldwide, royalty-free
476
+ patent license under the contributor's essential patent claims, to
477
+ make, use, sell, offer for sale, import and otherwise run, modify and
478
+ propagate the contents of its contributor version.
479
+
480
+ In the following three paragraphs, a "patent license" is any express
481
+ agreement or commitment, however denominated, not to enforce a patent
482
+ (such as an express permission to practice a patent or covenant not to
483
+ sue for patent infringement). To "grant" such a patent license to a
484
+ party means to make such an agreement or commitment not to enforce a
485
+ patent against the party.
486
+
487
+ If you convey a covered work, knowingly relying on a patent license,
488
+ and the Corresponding Source of the work is not available for anyone
489
+ to copy, free of charge and under the terms of this License, through a
490
+ publicly available network server or other readily accessible means,
491
+ then you must either (1) cause the Corresponding Source to be so
492
+ available, or (2) arrange to deprive yourself of the benefit of the
493
+ patent license for this particular work, or (3) arrange, in a manner
494
+ consistent with the requirements of this License, to extend the patent
495
+ license to downstream recipients. "Knowingly relying" means you have
496
+ actual knowledge that, but for the patent license, your conveying the
497
+ covered work in a country, or your recipient's use of the covered work
498
+ in a country, would infringe one or more identifiable patents in that
499
+ country that you have reason to believe are valid.
500
+
501
+ If, pursuant to or in connection with a single transaction or
502
+ arrangement, you convey, or propagate by procuring conveyance of, a
503
+ covered work, and grant a patent license to some of the parties
504
+ receiving the covered work authorizing them to use, propagate, modify
505
+ or convey a specific copy of the covered work, then the patent license
506
+ you grant is automatically extended to all recipients of the covered
507
+ work and works based on it.
508
+
509
+ A patent license is "discriminatory" if it does not include within
510
+ the scope of its coverage, prohibits the exercise of, or is
511
+ conditioned on the non-exercise of one or more of the rights that are
512
+ specifically granted under this License. You may not convey a covered
513
+ work if you are a party to an arrangement with a third party that is
514
+ in the business of distributing software, under which you make payment
515
+ to the third party based on the extent of your activity of conveying
516
+ the work, and under which the third party grants, to any of the
517
+ parties who would receive the covered work from you, a discriminatory
518
+ patent license (a) in connection with copies of the covered work
519
+ conveyed by you (or copies made from those copies), or (b) primarily
520
+ for and in connection with specific products or compilations that
521
+ contain the covered work, unless you entered into that arrangement,
522
+ or that patent license was granted, prior to 28 March 2007.
523
+
524
+ Nothing in this License shall be construed as excluding or limiting
525
+ any implied license or other defenses to infringement that may
526
+ otherwise be available to you under applicable patent law.
527
+
528
+ 12. No Surrender of Others' Freedom.
529
+
530
+ If conditions are imposed on you (whether by court order, agreement or
531
+ otherwise) that contradict the conditions of this License, they do not
532
+ excuse you from the conditions of this License. If you cannot convey a
533
+ covered work so as to satisfy simultaneously your obligations under this
534
+ License and any other pertinent obligations, then as a consequence you may
535
+ not convey it at all. For example, if you agree to terms that obligate you
536
+ to collect a royalty for further conveying from those to whom you convey
537
+ the Program, the only way you could satisfy both those terms and this
538
+ License would be to refrain entirely from conveying the Program.
539
+
540
+ 13. Remote Network Interaction; Use with the GNU General Public License.
541
+
542
+ Notwithstanding any other provision of this License, if you modify the
543
+ Program, your modified version must prominently offer all users
544
+ interacting with it remotely through a computer network (if your version
545
+ supports such interaction) an opportunity to receive the Corresponding
546
+ Source of your version by providing access to the Corresponding Source
547
+ from a network server at no charge, through some standard or customary
548
+ means of facilitating copying of software. This Corresponding Source
549
+ shall include the Corresponding Source for any work covered by version 3
550
+ of the GNU General Public License that is incorporated pursuant to the
551
+ following paragraph.
552
+
553
+ Notwithstanding any other provision of this License, you have
554
+ permission to link or combine any covered work with a work licensed
555
+ under version 3 of the GNU General Public License into a single
556
+ combined work, and to convey the resulting work. The terms of this
557
+ License will continue to apply to the part which is the covered work,
558
+ but the work with which it is combined will remain governed by version
559
+ 3 of the GNU General Public License.
560
+
561
+ 14. Revised Versions of this License.
562
+
563
+ The Free Software Foundation may publish revised and/or new versions of
564
+ the GNU Affero General Public License from time to time. Such new versions
565
+ will be similar in spirit to the present version, but may differ in detail to
566
+ address new problems or concerns.
567
+
568
+ Each version is given a distinguishing version number. If the
569
+ Program specifies that a certain numbered version of the GNU Affero General
570
+ Public License "or any later version" applies to it, you have the
571
+ option of following the terms and conditions either of that numbered
572
+ version or of any later version published by the Free Software
573
+ Foundation. If the Program does not specify a version number of the
574
+ GNU Affero General Public License, you may choose any version ever published
575
+ by the Free Software Foundation.
576
+
577
+ If the Program specifies that a proxy can decide which future
578
+ versions of the GNU Affero General Public License can be used, that proxy's
579
+ public statement of acceptance of a version permanently authorizes you
580
+ to choose that version for the Program.
581
+
582
+ Later license versions may give you additional or different
583
+ permissions. However, no additional obligations are imposed on any
584
+ author or copyright holder as a result of your choosing to follow a
585
+ later version.
586
+
587
+ 15. Disclaimer of Warranty.
588
+
589
+ THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
590
+ APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
591
+ HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
592
+ OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
593
+ THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
594
+ PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
595
+ IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
596
+ ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
597
+
598
+ 16. Limitation of Liability.
599
+
600
+ IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
601
+ WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
602
+ THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
603
+ GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
604
+ USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
605
+ DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
606
+ PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
607
+ EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
608
+ SUCH DAMAGES.
609
+
610
+ 17. Interpretation of Sections 15 and 16.
611
+
612
+ If the disclaimer of warranty and limitation of liability provided
613
+ above cannot be given local legal effect according to their terms,
614
+ reviewing courts shall apply local law that most closely approximates
615
+ an absolute waiver of all civil liability in connection with the
616
+ Program, unless a warranty or assumption of liability accompanies a
617
+ copy of the Program in return for a fee.
618
+
619
+ END OF TERMS AND CONDITIONS
620
+
621
+ How to Apply These Terms to Your New Programs
622
+
623
+ If you develop a new program, and you want it to be of the greatest
624
+ possible use to the public, the best way to achieve this is to make it
625
+ free software which everyone can redistribute and change under these terms.
626
+
627
+ To do so, attach the following notices to the program. It is safest
628
+ to attach them to the start of each source file to most effectively
629
+ state the exclusion of warranty; and each file should have at least
630
+ the "copyright" line and a pointer to where the full notice is found.
631
+
632
+ <one line to give the program's name and a brief idea of what it does.>
633
+ Copyright (C) <year> <name of author>
634
+
635
+ This program is free software: you can redistribute it and/or modify
636
+ it under the terms of the GNU Affero General Public License as published
637
+ by the Free Software Foundation, either version 3 of the License, or
638
+ (at your option) any later version.
639
+
640
+ This program is distributed in the hope that it will be useful,
641
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
642
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
643
+ GNU Affero General Public License for more details.
644
+
645
+ You should have received a copy of the GNU Affero General Public License
646
+ along with this program. If not, see <https://www.gnu.org/licenses/>.
647
+
648
+ Also add information on how to contact you by electronic and paper mail.
649
+
650
+ If your software can interact with users remotely through a computer
651
+ network, you should also make sure that it provides a way for users to
652
+ get its source. For example, if your program is a web application, its
653
+ interface could display a "Source" link that leads users to an archive
654
+ of the code. There are many ways you could offer source, and different
655
+ solutions will be better for different programs; see section 13 for the
656
+ specific requirements.
657
+
658
+ You should also get your employer (if you work as a programmer) or school,
659
+ if any, to sign a "copyright disclaimer" for the program, if necessary.
660
+ For more information on this, and how to apply and follow the GNU AGPL, see
661
+ <https://www.gnu.org/licenses/>.
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- title: Hanser Bert VITS2
3
  emoji: 📊
4
  colorFrom: yellow
5
  colorTo: indigo
 
1
  ---
2
+ title: AI Hanser
3
  emoji: 📊
4
  colorFrom: yellow
5
  colorTo: indigo
app.py ADDED
@@ -0,0 +1,228 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # flake8: noqa: E402
2
+
3
+ import sys, os
4
+ import logging
5
+
6
+ logging.getLogger("numba").setLevel(logging.WARNING)
7
+ logging.getLogger("markdown_it").setLevel(logging.WARNING)
8
+ logging.getLogger("urllib3").setLevel(logging.WARNING)
9
+ logging.getLogger("matplotlib").setLevel(logging.WARNING)
10
+
11
+ logging.basicConfig(
12
+ level=logging.INFO, format="| %(name)s | %(levelname)s | %(message)s"
13
+ )
14
+
15
+ logger = logging.getLogger(__name__)
16
+
17
+ import torch
18
+ import argparse
19
+ import commons
20
+ import utils
21
+ from models import SynthesizerTrn
22
+ from text.symbols import symbols
23
+ from text import cleaned_text_to_sequence, get_bert
24
+ from text.cleaner import clean_text
25
+ import gradio as gr
26
+ import webbrowser
27
+ import numpy as np
28
+
29
+ net_g = None
30
+
31
+ if sys.platform == "darwin" and torch.backends.mps.is_available():
32
+ device = "mps"
33
+ os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"
34
+ else:
35
+ device = "cuda"
36
+
37
+
38
+ def get_text(text, language_str, hps):
39
+ norm_text, phone, tone, word2ph = clean_text(text, language_str)
40
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
41
+
42
+ if hps.data.add_blank:
43
+ phone = commons.intersperse(phone, 0)
44
+ tone = commons.intersperse(tone, 0)
45
+ language = commons.intersperse(language, 0)
46
+ for i in range(len(word2ph)):
47
+ word2ph[i] = word2ph[i] * 2
48
+ word2ph[0] += 1
49
+ bert = get_bert(norm_text, word2ph, language_str, device)
50
+ del word2ph
51
+ assert bert.shape[-1] == len(phone), phone
52
+
53
+ if language_str == "ZH":
54
+ bert = bert
55
+ ja_bert = torch.zeros(768, len(phone))
56
+ elif language_str == "JP":
57
+ ja_bert = bert
58
+ bert = torch.zeros(1024, len(phone))
59
+ else:
60
+ bert = torch.zeros(1024, len(phone))
61
+ ja_bert = torch.zeros(768, len(phone))
62
+
63
+ assert bert.shape[-1] == len(
64
+ phone
65
+ ), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
66
+
67
+ phone = torch.LongTensor(phone)
68
+ tone = torch.LongTensor(tone)
69
+ language = torch.LongTensor(language)
70
+ return bert, ja_bert, phone, tone, language
71
+
72
+
73
+ def infer(text, sdp_ratio, noise_scale, noise_scale_w, length_scale, sid, language):
74
+ global net_g
75
+ bert, ja_bert, phones, tones, lang_ids = get_text(text, language, hps)
76
+ with torch.no_grad():
77
+ x_tst = phones.to(device).unsqueeze(0)
78
+ tones = tones.to(device).unsqueeze(0)
79
+ lang_ids = lang_ids.to(device).unsqueeze(0)
80
+ bert = bert.to(device).unsqueeze(0)
81
+ ja_bert = ja_bert.to(device).unsqueeze(0)
82
+ x_tst_lengths = torch.LongTensor([phones.size(0)]).to(device)
83
+ del phones
84
+ speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(device)
85
+ audio = (
86
+ net_g.infer(
87
+ x_tst,
88
+ x_tst_lengths,
89
+ speakers,
90
+ tones,
91
+ lang_ids,
92
+ bert,
93
+ ja_bert,
94
+ sdp_ratio=sdp_ratio,
95
+ noise_scale=noise_scale,
96
+ noise_scale_w=noise_scale_w,
97
+ length_scale=length_scale,
98
+ )[0][0, 0]
99
+ .data.cpu()
100
+ .float()
101
+ .numpy()
102
+ )
103
+ del x_tst, tones, lang_ids, bert, x_tst_lengths, speakers
104
+ torch.cuda.empty_cache()
105
+ return audio
106
+
107
+
108
+ def tts_fn(text, speaker, sdp_ratio, noise_scale, noise_scale_w, length_scale, language):
109
+ slices = text.split("|")
110
+ audio_list = []
111
+ with torch.no_grad():
112
+ for slice in slices:
113
+ audio = infer(slice, sdp_ratio=sdp_ratio, noise_scale=noise_scale, noise_scale_w=noise_scale_w, length_scale=length_scale, sid=speaker, language=language)
114
+ audio_list.append(audio)
115
+ silence = np.zeros(hps.data.sampling_rate) # 生成1秒的静音
116
+ audio_list.append(silence) # 将静音添加到列表中
117
+ audio_concat = np.concatenate(audio_list)
118
+ return "Success", (hps.data.sampling_rate, audio_concat)
119
+
120
+ if __name__ == "__main__":
121
+ parser = argparse.ArgumentParser()
122
+ parser.add_argument(
123
+ "-m", "--model", default="./logs/hanser/G_8000.pth", help="path of your model"
124
+ )
125
+ parser.add_argument(
126
+ "-c",
127
+ "--config",
128
+ default="./configs/config.json",
129
+ help="path of your config file",
130
+ )
131
+ parser.add_argument(
132
+ "--share", default=False, help="make link public", action="store_true"
133
+ )
134
+ parser.add_argument(
135
+ "-d", "--debug", action="store_true", help="enable DEBUG-LEVEL log"
136
+ )
137
+
138
+ args = parser.parse_args()
139
+ if args.debug:
140
+ logger.info("Enable DEBUG-LEVEL log")
141
+ logging.basicConfig(level=logging.DEBUG)
142
+ hps = utils.get_hparams_from_file(args.config)
143
+
144
+ device = (
145
+ "cuda:0"
146
+ if torch.cuda.is_available()
147
+ else (
148
+ "mps"
149
+ if sys.platform == "darwin" and torch.backends.mps.is_available()
150
+ else "cpu"
151
+ )
152
+ )
153
+ net_g = SynthesizerTrn(
154
+ len(symbols),
155
+ hps.data.filter_length // 2 + 1,
156
+ hps.train.segment_size // hps.data.hop_length,
157
+ n_speakers=hps.data.n_speakers,
158
+ **hps.model,
159
+ ).to(device)
160
+ _ = net_g.eval()
161
+
162
+ _ = utils.load_checkpoint(args.model, net_g, None, skip_optimizer=True)
163
+
164
+ speaker_ids = hps.data.spk2id
165
+ speakers = list(speaker_ids.keys())
166
+ languages = ["ZH", "JP"]
167
+ with gr.Blocks() as app:
168
+ with gr.Row():
169
+ with gr.Column():
170
+ gr.Markdown(value="""
171
+ 🤖 【AI Hanser】在线语音合成 Bert-Vits2 🤖\n
172
+ 📝 作者:Rayzggz 📰博客 https://roi.moe 📺B站 https://space.bilibili.com/10501326 📝\n
173
+ 🎤 声音来源:Hanser https://space.bilibili.com/11073 🎤\n
174
+ 🔗 Bert-VITS2:https://github.com/fishaudio/Bert-VITS2 🔗\n
175
+ ✅ 使用本模型请遵守中华人民共和国和美利坚合众国法律 ✅\n
176
+ 🏷️ 使用基于本模型的所有生成内容均需标注「使用Bert-VITS2 AI生成」、「本项目地址」、「作者名称」和「声音来源」 🏷️\n
177
+ """)
178
+ text = gr.TextArea(
179
+ label="Text",
180
+ placeholder="Input Text Here",
181
+ value="澡后挖耳朵,湿湿一大坨",
182
+ )
183
+ speaker = gr.Dropdown(
184
+ choices=speakers, value=speakers[0], label="Speaker"
185
+ )
186
+ sdp_ratio = gr.Slider(
187
+ minimum=0, maximum=1, value=0.2, step=0.1, label="SDP Ratio"
188
+ )
189
+ noise_scale = gr.Slider(
190
+ minimum=0.1, maximum=2, value=0.6, step=0.1, label="Noise Scale"
191
+ )
192
+ noise_scale_w = gr.Slider(
193
+ minimum=0.1, maximum=2, value=0.8, step=0.1, label="Noise Scale W"
194
+ )
195
+ length_scale = gr.Slider(
196
+ minimum=0.1, maximum=2, value=1, step=0.1, label="Length Scale"
197
+ )
198
+ language = gr.Dropdown(
199
+ choices=languages, value=languages[0], label="Language"
200
+ )
201
+ btn = gr.Button("Generate 生成!", variant="primary")
202
+ with gr.Column():
203
+ text_output = gr.Textbox(label="Message")
204
+ audio_output = gr.Audio(label="Output Audio")
205
+ gr.Markdown(value="""
206
+ 👏 鸣谢: 👏\n
207
+ 👤 团子是咸鱼 https://space.bilibili.com/10685437 👤\n
208
+ 👤 领航员未鸟 https://space.bilibili.com/2403955 👤\n
209
+ 👤 Xz乔希 https://space.bilibili.com/5859321 👤\n
210
+ 🧠 Google Colab https://colab.research.google.com/ 🧠\n
211
+ 📧 如果你是“Hanser”,并且希望对此模型主张权利,请通过上方“作者”部分的联系方式联系,我将积极配合处理。📧 \n
212
+ """)
213
+
214
+ btn.click(
215
+ tts_fn,
216
+ inputs=[
217
+ text,
218
+ speaker,
219
+ sdp_ratio,
220
+ noise_scale,
221
+ noise_scale_w,
222
+ length_scale,
223
+ language,
224
+ ],
225
+ outputs=[text_output, audio_output],
226
+ )
227
+
228
+ app.launch(show_error=True)
attentions.py ADDED
@@ -0,0 +1,464 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import torch
3
+ from torch import nn
4
+ from torch.nn import functional as F
5
+
6
+ import commons
7
+ import logging
8
+
9
+ logger = logging.getLogger(__name__)
10
+
11
+
12
+ class LayerNorm(nn.Module):
13
+ def __init__(self, channels, eps=1e-5):
14
+ super().__init__()
15
+ self.channels = channels
16
+ self.eps = eps
17
+
18
+ self.gamma = nn.Parameter(torch.ones(channels))
19
+ self.beta = nn.Parameter(torch.zeros(channels))
20
+
21
+ def forward(self, x):
22
+ x = x.transpose(1, -1)
23
+ x = F.layer_norm(x, (self.channels,), self.gamma, self.beta, self.eps)
24
+ return x.transpose(1, -1)
25
+
26
+
27
+ @torch.jit.script
28
+ def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
29
+ n_channels_int = n_channels[0]
30
+ in_act = input_a + input_b
31
+ t_act = torch.tanh(in_act[:, :n_channels_int, :])
32
+ s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
33
+ acts = t_act * s_act
34
+ return acts
35
+
36
+
37
+ class Encoder(nn.Module):
38
+ def __init__(
39
+ self,
40
+ hidden_channels,
41
+ filter_channels,
42
+ n_heads,
43
+ n_layers,
44
+ kernel_size=1,
45
+ p_dropout=0.0,
46
+ window_size=4,
47
+ isflow=True,
48
+ **kwargs
49
+ ):
50
+ super().__init__()
51
+ self.hidden_channels = hidden_channels
52
+ self.filter_channels = filter_channels
53
+ self.n_heads = n_heads
54
+ self.n_layers = n_layers
55
+ self.kernel_size = kernel_size
56
+ self.p_dropout = p_dropout
57
+ self.window_size = window_size
58
+ # if isflow:
59
+ # cond_layer = torch.nn.Conv1d(256, 2*hidden_channels*n_layers, 1)
60
+ # self.cond_pre = torch.nn.Conv1d(hidden_channels, 2*hidden_channels, 1)
61
+ # self.cond_layer = weight_norm(cond_layer, name='weight')
62
+ # self.gin_channels = 256
63
+ self.cond_layer_idx = self.n_layers
64
+ if "gin_channels" in kwargs:
65
+ self.gin_channels = kwargs["gin_channels"]
66
+ if self.gin_channels != 0:
67
+ self.spk_emb_linear = nn.Linear(self.gin_channels, self.hidden_channels)
68
+ # vits2 says 3rd block, so idx is 2 by default
69
+ self.cond_layer_idx = (
70
+ kwargs["cond_layer_idx"] if "cond_layer_idx" in kwargs else 2
71
+ )
72
+ logging.debug(self.gin_channels, self.cond_layer_idx)
73
+ assert (
74
+ self.cond_layer_idx < self.n_layers
75
+ ), "cond_layer_idx should be less than n_layers"
76
+ self.drop = nn.Dropout(p_dropout)
77
+ self.attn_layers = nn.ModuleList()
78
+ self.norm_layers_1 = nn.ModuleList()
79
+ self.ffn_layers = nn.ModuleList()
80
+ self.norm_layers_2 = nn.ModuleList()
81
+ for i in range(self.n_layers):
82
+ self.attn_layers.append(
83
+ MultiHeadAttention(
84
+ hidden_channels,
85
+ hidden_channels,
86
+ n_heads,
87
+ p_dropout=p_dropout,
88
+ window_size=window_size,
89
+ )
90
+ )
91
+ self.norm_layers_1.append(LayerNorm(hidden_channels))
92
+ self.ffn_layers.append(
93
+ FFN(
94
+ hidden_channels,
95
+ hidden_channels,
96
+ filter_channels,
97
+ kernel_size,
98
+ p_dropout=p_dropout,
99
+ )
100
+ )
101
+ self.norm_layers_2.append(LayerNorm(hidden_channels))
102
+
103
+ def forward(self, x, x_mask, g=None):
104
+ attn_mask = x_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
105
+ x = x * x_mask
106
+ for i in range(self.n_layers):
107
+ if i == self.cond_layer_idx and g is not None:
108
+ g = self.spk_emb_linear(g.transpose(1, 2))
109
+ g = g.transpose(1, 2)
110
+ x = x + g
111
+ x = x * x_mask
112
+ y = self.attn_layers[i](x, x, attn_mask)
113
+ y = self.drop(y)
114
+ x = self.norm_layers_1[i](x + y)
115
+
116
+ y = self.ffn_layers[i](x, x_mask)
117
+ y = self.drop(y)
118
+ x = self.norm_layers_2[i](x + y)
119
+ x = x * x_mask
120
+ return x
121
+
122
+
123
+ class Decoder(nn.Module):
124
+ def __init__(
125
+ self,
126
+ hidden_channels,
127
+ filter_channels,
128
+ n_heads,
129
+ n_layers,
130
+ kernel_size=1,
131
+ p_dropout=0.0,
132
+ proximal_bias=False,
133
+ proximal_init=True,
134
+ **kwargs
135
+ ):
136
+ super().__init__()
137
+ self.hidden_channels = hidden_channels
138
+ self.filter_channels = filter_channels
139
+ self.n_heads = n_heads
140
+ self.n_layers = n_layers
141
+ self.kernel_size = kernel_size
142
+ self.p_dropout = p_dropout
143
+ self.proximal_bias = proximal_bias
144
+ self.proximal_init = proximal_init
145
+
146
+ self.drop = nn.Dropout(p_dropout)
147
+ self.self_attn_layers = nn.ModuleList()
148
+ self.norm_layers_0 = nn.ModuleList()
149
+ self.encdec_attn_layers = nn.ModuleList()
150
+ self.norm_layers_1 = nn.ModuleList()
151
+ self.ffn_layers = nn.ModuleList()
152
+ self.norm_layers_2 = nn.ModuleList()
153
+ for i in range(self.n_layers):
154
+ self.self_attn_layers.append(
155
+ MultiHeadAttention(
156
+ hidden_channels,
157
+ hidden_channels,
158
+ n_heads,
159
+ p_dropout=p_dropout,
160
+ proximal_bias=proximal_bias,
161
+ proximal_init=proximal_init,
162
+ )
163
+ )
164
+ self.norm_layers_0.append(LayerNorm(hidden_channels))
165
+ self.encdec_attn_layers.append(
166
+ MultiHeadAttention(
167
+ hidden_channels, hidden_channels, n_heads, p_dropout=p_dropout
168
+ )
169
+ )
170
+ self.norm_layers_1.append(LayerNorm(hidden_channels))
171
+ self.ffn_layers.append(
172
+ FFN(
173
+ hidden_channels,
174
+ hidden_channels,
175
+ filter_channels,
176
+ kernel_size,
177
+ p_dropout=p_dropout,
178
+ causal=True,
179
+ )
180
+ )
181
+ self.norm_layers_2.append(LayerNorm(hidden_channels))
182
+
183
+ def forward(self, x, x_mask, h, h_mask):
184
+ """
185
+ x: decoder input
186
+ h: encoder output
187
+ """
188
+ self_attn_mask = commons.subsequent_mask(x_mask.size(2)).to(
189
+ device=x.device, dtype=x.dtype
190
+ )
191
+ encdec_attn_mask = h_mask.unsqueeze(2) * x_mask.unsqueeze(-1)
192
+ x = x * x_mask
193
+ for i in range(self.n_layers):
194
+ y = self.self_attn_layers[i](x, x, self_attn_mask)
195
+ y = self.drop(y)
196
+ x = self.norm_layers_0[i](x + y)
197
+
198
+ y = self.encdec_attn_layers[i](x, h, encdec_attn_mask)
199
+ y = self.drop(y)
200
+ x = self.norm_layers_1[i](x + y)
201
+
202
+ y = self.ffn_layers[i](x, x_mask)
203
+ y = self.drop(y)
204
+ x = self.norm_layers_2[i](x + y)
205
+ x = x * x_mask
206
+ return x
207
+
208
+
209
+ class MultiHeadAttention(nn.Module):
210
+ def __init__(
211
+ self,
212
+ channels,
213
+ out_channels,
214
+ n_heads,
215
+ p_dropout=0.0,
216
+ window_size=None,
217
+ heads_share=True,
218
+ block_length=None,
219
+ proximal_bias=False,
220
+ proximal_init=False,
221
+ ):
222
+ super().__init__()
223
+ assert channels % n_heads == 0
224
+
225
+ self.channels = channels
226
+ self.out_channels = out_channels
227
+ self.n_heads = n_heads
228
+ self.p_dropout = p_dropout
229
+ self.window_size = window_size
230
+ self.heads_share = heads_share
231
+ self.block_length = block_length
232
+ self.proximal_bias = proximal_bias
233
+ self.proximal_init = proximal_init
234
+ self.attn = None
235
+
236
+ self.k_channels = channels // n_heads
237
+ self.conv_q = nn.Conv1d(channels, channels, 1)
238
+ self.conv_k = nn.Conv1d(channels, channels, 1)
239
+ self.conv_v = nn.Conv1d(channels, channels, 1)
240
+ self.conv_o = nn.Conv1d(channels, out_channels, 1)
241
+ self.drop = nn.Dropout(p_dropout)
242
+
243
+ if window_size is not None:
244
+ n_heads_rel = 1 if heads_share else n_heads
245
+ rel_stddev = self.k_channels**-0.5
246
+ self.emb_rel_k = nn.Parameter(
247
+ torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels)
248
+ * rel_stddev
249
+ )
250
+ self.emb_rel_v = nn.Parameter(
251
+ torch.randn(n_heads_rel, window_size * 2 + 1, self.k_channels)
252
+ * rel_stddev
253
+ )
254
+
255
+ nn.init.xavier_uniform_(self.conv_q.weight)
256
+ nn.init.xavier_uniform_(self.conv_k.weight)
257
+ nn.init.xavier_uniform_(self.conv_v.weight)
258
+ if proximal_init:
259
+ with torch.no_grad():
260
+ self.conv_k.weight.copy_(self.conv_q.weight)
261
+ self.conv_k.bias.copy_(self.conv_q.bias)
262
+
263
+ def forward(self, x, c, attn_mask=None):
264
+ q = self.conv_q(x)
265
+ k = self.conv_k(c)
266
+ v = self.conv_v(c)
267
+
268
+ x, self.attn = self.attention(q, k, v, mask=attn_mask)
269
+
270
+ x = self.conv_o(x)
271
+ return x
272
+
273
+ def attention(self, query, key, value, mask=None):
274
+ # reshape [b, d, t] -> [b, n_h, t, d_k]
275
+ b, d, t_s, t_t = (*key.size(), query.size(2))
276
+ query = query.view(b, self.n_heads, self.k_channels, t_t).transpose(2, 3)
277
+ key = key.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3)
278
+ value = value.view(b, self.n_heads, self.k_channels, t_s).transpose(2, 3)
279
+
280
+ scores = torch.matmul(query / math.sqrt(self.k_channels), key.transpose(-2, -1))
281
+ if self.window_size is not None:
282
+ assert (
283
+ t_s == t_t
284
+ ), "Relative attention is only available for self-attention."
285
+ key_relative_embeddings = self._get_relative_embeddings(self.emb_rel_k, t_s)
286
+ rel_logits = self._matmul_with_relative_keys(
287
+ query / math.sqrt(self.k_channels), key_relative_embeddings
288
+ )
289
+ scores_local = self._relative_position_to_absolute_position(rel_logits)
290
+ scores = scores + scores_local
291
+ if self.proximal_bias:
292
+ assert t_s == t_t, "Proximal bias is only available for self-attention."
293
+ scores = scores + self._attention_bias_proximal(t_s).to(
294
+ device=scores.device, dtype=scores.dtype
295
+ )
296
+ if mask is not None:
297
+ scores = scores.masked_fill(mask == 0, -1e4)
298
+ if self.block_length is not None:
299
+ assert (
300
+ t_s == t_t
301
+ ), "Local attention is only available for self-attention."
302
+ block_mask = (
303
+ torch.ones_like(scores)
304
+ .triu(-self.block_length)
305
+ .tril(self.block_length)
306
+ )
307
+ scores = scores.masked_fill(block_mask == 0, -1e4)
308
+ p_attn = F.softmax(scores, dim=-1) # [b, n_h, t_t, t_s]
309
+ p_attn = self.drop(p_attn)
310
+ output = torch.matmul(p_attn, value)
311
+ if self.window_size is not None:
312
+ relative_weights = self._absolute_position_to_relative_position(p_attn)
313
+ value_relative_embeddings = self._get_relative_embeddings(
314
+ self.emb_rel_v, t_s
315
+ )
316
+ output = output + self._matmul_with_relative_values(
317
+ relative_weights, value_relative_embeddings
318
+ )
319
+ output = (
320
+ output.transpose(2, 3).contiguous().view(b, d, t_t)
321
+ ) # [b, n_h, t_t, d_k] -> [b, d, t_t]
322
+ return output, p_attn
323
+
324
+ def _matmul_with_relative_values(self, x, y):
325
+ """
326
+ x: [b, h, l, m]
327
+ y: [h or 1, m, d]
328
+ ret: [b, h, l, d]
329
+ """
330
+ ret = torch.matmul(x, y.unsqueeze(0))
331
+ return ret
332
+
333
+ def _matmul_with_relative_keys(self, x, y):
334
+ """
335
+ x: [b, h, l, d]
336
+ y: [h or 1, m, d]
337
+ ret: [b, h, l, m]
338
+ """
339
+ ret = torch.matmul(x, y.unsqueeze(0).transpose(-2, -1))
340
+ return ret
341
+
342
+ def _get_relative_embeddings(self, relative_embeddings, length):
343
+ 2 * self.window_size + 1
344
+ # Pad first before slice to avoid using cond ops.
345
+ pad_length = max(length - (self.window_size + 1), 0)
346
+ slice_start_position = max((self.window_size + 1) - length, 0)
347
+ slice_end_position = slice_start_position + 2 * length - 1
348
+ if pad_length > 0:
349
+ padded_relative_embeddings = F.pad(
350
+ relative_embeddings,
351
+ commons.convert_pad_shape([[0, 0], [pad_length, pad_length], [0, 0]]),
352
+ )
353
+ else:
354
+ padded_relative_embeddings = relative_embeddings
355
+ used_relative_embeddings = padded_relative_embeddings[
356
+ :, slice_start_position:slice_end_position
357
+ ]
358
+ return used_relative_embeddings
359
+
360
+ def _relative_position_to_absolute_position(self, x):
361
+ """
362
+ x: [b, h, l, 2*l-1]
363
+ ret: [b, h, l, l]
364
+ """
365
+ batch, heads, length, _ = x.size()
366
+ # Concat columns of pad to shift from relative to absolute indexing.
367
+ x = F.pad(x, commons.convert_pad_shape([[0, 0], [0, 0], [0, 0], [0, 1]]))
368
+
369
+ # Concat extra elements so to add up to shape (len+1, 2*len-1).
370
+ x_flat = x.view([batch, heads, length * 2 * length])
371
+ x_flat = F.pad(
372
+ x_flat, commons.convert_pad_shape([[0, 0], [0, 0], [0, length - 1]])
373
+ )
374
+
375
+ # Reshape and slice out the padded elements.
376
+ x_final = x_flat.view([batch, heads, length + 1, 2 * length - 1])[
377
+ :, :, :length, length - 1 :
378
+ ]
379
+ return x_final
380
+
381
+ def _absolute_position_to_relative_position(self, x):
382
+ """
383
+ x: [b, h, l, l]
384
+ ret: [b, h, l, 2*l-1]
385
+ """
386
+ batch, heads, length, _ = x.size()
387
+ # pad along column
388
+ x = F.pad(
389
+ x, commons.convert_pad_shape([[0, 0], [0, 0], [0, 0], [0, length - 1]])
390
+ )
391
+ x_flat = x.view([batch, heads, length**2 + length * (length - 1)])
392
+ # add 0's in the beginning that will skew the elements after reshape
393
+ x_flat = F.pad(x_flat, commons.convert_pad_shape([[0, 0], [0, 0], [length, 0]]))
394
+ x_final = x_flat.view([batch, heads, length, 2 * length])[:, :, :, 1:]
395
+ return x_final
396
+
397
+ def _attention_bias_proximal(self, length):
398
+ """Bias for self-attention to encourage attention to close positions.
399
+ Args:
400
+ length: an integer scalar.
401
+ Returns:
402
+ a Tensor with shape [1, 1, length, length]
403
+ """
404
+ r = torch.arange(length, dtype=torch.float32)
405
+ diff = torch.unsqueeze(r, 0) - torch.unsqueeze(r, 1)
406
+ return torch.unsqueeze(torch.unsqueeze(-torch.log1p(torch.abs(diff)), 0), 0)
407
+
408
+
409
+ class FFN(nn.Module):
410
+ def __init__(
411
+ self,
412
+ in_channels,
413
+ out_channels,
414
+ filter_channels,
415
+ kernel_size,
416
+ p_dropout=0.0,
417
+ activation=None,
418
+ causal=False,
419
+ ):
420
+ super().__init__()
421
+ self.in_channels = in_channels
422
+ self.out_channels = out_channels
423
+ self.filter_channels = filter_channels
424
+ self.kernel_size = kernel_size
425
+ self.p_dropout = p_dropout
426
+ self.activation = activation
427
+ self.causal = causal
428
+
429
+ if causal:
430
+ self.padding = self._causal_padding
431
+ else:
432
+ self.padding = self._same_padding
433
+
434
+ self.conv_1 = nn.Conv1d(in_channels, filter_channels, kernel_size)
435
+ self.conv_2 = nn.Conv1d(filter_channels, out_channels, kernel_size)
436
+ self.drop = nn.Dropout(p_dropout)
437
+
438
+ def forward(self, x, x_mask):
439
+ x = self.conv_1(self.padding(x * x_mask))
440
+ if self.activation == "gelu":
441
+ x = x * torch.sigmoid(1.702 * x)
442
+ else:
443
+ x = torch.relu(x)
444
+ x = self.drop(x)
445
+ x = self.conv_2(self.padding(x * x_mask))
446
+ return x * x_mask
447
+
448
+ def _causal_padding(self, x):
449
+ if self.kernel_size == 1:
450
+ return x
451
+ pad_l = self.kernel_size - 1
452
+ pad_r = 0
453
+ padding = [[0, 0], [0, 0], [pad_l, pad_r]]
454
+ x = F.pad(x, commons.convert_pad_shape(padding))
455
+ return x
456
+
457
+ def _same_padding(self, x):
458
+ if self.kernel_size == 1:
459
+ return x
460
+ pad_l = (self.kernel_size - 1) // 2
461
+ pad_r = self.kernel_size // 2
462
+ padding = [[0, 0], [0, 0], [pad_l, pad_r]]
463
+ x = F.pad(x, commons.convert_pad_shape(padding))
464
+ return x
bert/bert-base-japanese-v3/README.md ADDED
@@ -0,0 +1,53 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ datasets:
4
+ - cc100
5
+ - wikipedia
6
+ language:
7
+ - ja
8
+ widget:
9
+ - text: 東北大学で[MASK]の研究をしています。
10
+ ---
11
+
12
+ # BERT base Japanese (unidic-lite with whole word masking, CC-100 and jawiki-20230102)
13
+
14
+ This is a [BERT](https://github.com/google-research/bert) model pretrained on texts in the Japanese language.
15
+
16
+ This version of the model processes input texts with word-level tokenization based on the Unidic 2.1.2 dictionary (available in [unidic-lite](https://pypi.org/project/unidic-lite/) package), followed by the WordPiece subword tokenization.
17
+ Additionally, the model is trained with the whole word masking enabled for the masked language modeling (MLM) objective.
18
+
19
+ The codes for the pretraining are available at [cl-tohoku/bert-japanese](https://github.com/cl-tohoku/bert-japanese/).
20
+
21
+ ## Model architecture
22
+
23
+ The model architecture is the same as the original BERT base model; 12 layers, 768 dimensions of hidden states, and 12 attention heads.
24
+
25
+ ## Training Data
26
+
27
+ The model is trained on the Japanese portion of [CC-100 dataset](https://data.statmt.org/cc-100/) and the Japanese version of Wikipedia.
28
+ For Wikipedia, we generated a text corpus from the [Wikipedia Cirrussearch dump file](https://dumps.wikimedia.org/other/cirrussearch/) as of January 2, 2023.
29
+ The corpus files generated from CC-100 and Wikipedia are 74.3GB and 4.9GB in size and consist of approximately 392M and 34M sentences, respectively.
30
+
31
+ For the purpose of splitting texts into sentences, we used [fugashi](https://github.com/polm/fugashi) with [mecab-ipadic-NEologd](https://github.com/neologd/mecab-ipadic-neologd) dictionary (v0.0.7).
32
+
33
+ ## Tokenization
34
+
35
+ The texts are first tokenized by MeCab with the Unidic 2.1.2 dictionary and then split into subwords by the WordPiece algorithm.
36
+ The vocabulary size is 32768.
37
+
38
+ We used [fugashi](https://github.com/polm/fugashi) and [unidic-lite](https://github.com/polm/unidic-lite) packages for the tokenization.
39
+
40
+ ## Training
41
+
42
+ We trained the model first on the CC-100 corpus for 1M steps and then on the Wikipedia corpus for another 1M steps.
43
+ For training of the MLM (masked language modeling) objective, we introduced whole word masking in which all of the subword tokens corresponding to a single word (tokenized by MeCab) are masked at once.
44
+
45
+ For training of each model, we used a v3-8 instance of Cloud TPUs provided by [TPU Research Cloud](https://sites.research.google/trc/about/).
46
+
47
+ ## Licenses
48
+
49
+ The pretrained models are distributed under the Apache License 2.0.
50
+
51
+ ## Acknowledgments
52
+
53
+ This model is trained with Cloud TPUs provided by [TPU Research Cloud](https://sites.research.google/trc/about/) program.
bert/bert-base-japanese-v3/config.json ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BertForPreTraining"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "hidden_act": "gelu",
7
+ "hidden_dropout_prob": 0.1,
8
+ "hidden_size": 768,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 3072,
11
+ "layer_norm_eps": 1e-12,
12
+ "max_position_embeddings": 512,
13
+ "model_type": "bert",
14
+ "num_attention_heads": 12,
15
+ "num_hidden_layers": 12,
16
+ "pad_token_id": 0,
17
+ "type_vocab_size": 2,
18
+ "vocab_size": 32768
19
+ }
bert/bert-base-japanese-v3/flax_model.msgpack ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7dce0b8b350432362a184b9f8bb90ffb0f2ff0c394ab43b915e318926f4e7569
3
+ size 447341816
bert/bert-base-japanese-v3/pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e172862e0674054d65e0ba40d67df2a4687982f589db44aa27091c386e5450a4
3
+ size 447406217
bert/bert-base-japanese-v3/tf_model.h5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:71920d0dc0174d0a0ce32b934fe65f15320b2d53aa7e671718b33065748cb712
3
+ size 549871840
bert/bert-base-japanese-v3/tokenizer_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "BertJapaneseTokenizer",
3
+ "model_max_length": 512,
4
+ "do_lower_case": false,
5
+ "word_tokenizer_type": "mecab",
6
+ "subword_tokenizer_type": "wordpiece",
7
+ "mecab_kwargs": {
8
+ "mecab_dic": "unidic_lite"
9
+ }
10
+ }
bert/bert-base-japanese-v3/vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
bert/chinese-roberta-wwm-ext-large/.gitattributes ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ *.bin.* filter=lfs diff=lfs merge=lfs -text
2
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.h5 filter=lfs diff=lfs merge=lfs -text
5
+ *.tflite filter=lfs diff=lfs merge=lfs -text
6
+ *.tar.gz filter=lfs diff=lfs merge=lfs -text
7
+ *.ot filter=lfs diff=lfs merge=lfs -text
8
+ *.onnx filter=lfs diff=lfs merge=lfs -text
9
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
bert/chinese-roberta-wwm-ext-large/.gitignore ADDED
@@ -0,0 +1 @@
 
 
1
+ *.bin
bert/chinese-roberta-wwm-ext-large/README.md ADDED
@@ -0,0 +1,57 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - zh
4
+ tags:
5
+ - bert
6
+ license: "apache-2.0"
7
+ ---
8
+
9
+ # Please use 'Bert' related functions to load this model!
10
+
11
+ ## Chinese BERT with Whole Word Masking
12
+ For further accelerating Chinese natural language processing, we provide **Chinese pre-trained BERT with Whole Word Masking**.
13
+
14
+ **[Pre-Training with Whole Word Masking for Chinese BERT](https://arxiv.org/abs/1906.08101)**
15
+ Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, Guoping Hu
16
+
17
+ This repository is developed based on:https://github.com/google-research/bert
18
+
19
+ You may also interested in,
20
+ - Chinese BERT series: https://github.com/ymcui/Chinese-BERT-wwm
21
+ - Chinese MacBERT: https://github.com/ymcui/MacBERT
22
+ - Chinese ELECTRA: https://github.com/ymcui/Chinese-ELECTRA
23
+ - Chinese XLNet: https://github.com/ymcui/Chinese-XLNet
24
+ - Knowledge Distillation Toolkit - TextBrewer: https://github.com/airaria/TextBrewer
25
+
26
+ More resources by HFL: https://github.com/ymcui/HFL-Anthology
27
+
28
+ ## Citation
29
+ If you find the technical report or resource is useful, please cite the following technical report in your paper.
30
+ - Primary: https://arxiv.org/abs/2004.13922
31
+ ```
32
+ @inproceedings{cui-etal-2020-revisiting,
33
+ title = "Revisiting Pre-Trained Models for {C}hinese Natural Language Processing",
34
+ author = "Cui, Yiming and
35
+ Che, Wanxiang and
36
+ Liu, Ting and
37
+ Qin, Bing and
38
+ Wang, Shijin and
39
+ Hu, Guoping",
40
+ booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings",
41
+ month = nov,
42
+ year = "2020",
43
+ address = "Online",
44
+ publisher = "Association for Computational Linguistics",
45
+ url = "https://www.aclweb.org/anthology/2020.findings-emnlp.58",
46
+ pages = "657--668",
47
+ }
48
+ ```
49
+ - Secondary: https://arxiv.org/abs/1906.08101
50
+ ```
51
+ @article{chinese-bert-wwm,
52
+ title={Pre-Training with Whole Word Masking for Chinese BERT},
53
+ author={Cui, Yiming and Che, Wanxiang and Liu, Ting and Qin, Bing and Yang, Ziqing and Wang, Shijin and Hu, Guoping},
54
+ journal={arXiv preprint arXiv:1906.08101},
55
+ year={2019}
56
+ }
57
+ ```
bert/chinese-roberta-wwm-ext-large/added_tokens.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {}
bert/chinese-roberta-wwm-ext-large/config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BertForMaskedLM"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "bos_token_id": 0,
7
+ "directionality": "bidi",
8
+ "eos_token_id": 2,
9
+ "hidden_act": "gelu",
10
+ "hidden_dropout_prob": 0.1,
11
+ "hidden_size": 1024,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 4096,
14
+ "layer_norm_eps": 1e-12,
15
+ "max_position_embeddings": 512,
16
+ "model_type": "bert",
17
+ "num_attention_heads": 16,
18
+ "num_hidden_layers": 24,
19
+ "output_past": true,
20
+ "pad_token_id": 0,
21
+ "pooler_fc_size": 768,
22
+ "pooler_num_attention_heads": 12,
23
+ "pooler_num_fc_layers": 3,
24
+ "pooler_size_per_head": 128,
25
+ "pooler_type": "first_token_transform",
26
+ "type_vocab_size": 2,
27
+ "vocab_size": 21128
28
+ }
bert/chinese-roberta-wwm-ext-large/flax_model.msgpack ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a46a510fe646213c728b80c9d0d5691d05235523d67f9ac3c3ce4e67deabf926
3
+ size 1302196529
bert/chinese-roberta-wwm-ext-large/pytorch_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ac62d49144d770c5ca9a5d1d3039c4995665a080febe63198189857c6bd11cd
3
+ size 1306484351
bert/chinese-roberta-wwm-ext-large/special_tokens_map.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
bert/chinese-roberta-wwm-ext-large/tf_model.h5 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:72d18616fb285b720cb869c25aa9f4d7371033dfd5d8ba82aca448fdd28132bf
3
+ size 1302594480
bert/chinese-roberta-wwm-ext-large/tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
bert/chinese-roberta-wwm-ext-large/tokenizer_config.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"init_inputs": []}
bert/chinese-roberta-wwm-ext-large/vocab.txt ADDED
The diff for this file is too large to render. See raw diff
 
bert_gen.py ADDED
@@ -0,0 +1,59 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from multiprocessing import Pool
3
+ import commons
4
+ import utils
5
+ from tqdm import tqdm
6
+ from text import cleaned_text_to_sequence, get_bert
7
+ import argparse
8
+ import torch.multiprocessing as mp
9
+
10
+
11
+ def process_line(line):
12
+ rank = mp.current_process()._identity
13
+ rank = rank[0] if len(rank) > 0 else 0
14
+ if torch.cuda.is_available():
15
+ gpu_id = rank % torch.cuda.device_count()
16
+ device = torch.device(f"cuda:{gpu_id}")
17
+ wav_path, _, language_str, text, phones, tone, word2ph = line.strip().split("|")
18
+ phone = phones.split(" ")
19
+ tone = [int(i) for i in tone.split(" ")]
20
+ word2ph = [int(i) for i in word2ph.split(" ")]
21
+ word2ph = [i for i in word2ph]
22
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
23
+
24
+ phone = commons.intersperse(phone, 0)
25
+ tone = commons.intersperse(tone, 0)
26
+ language = commons.intersperse(language, 0)
27
+ for i in range(len(word2ph)):
28
+ word2ph[i] = word2ph[i] * 2
29
+ word2ph[0] += 1
30
+
31
+ bert_path = wav_path.replace(".wav", ".bert.pt")
32
+
33
+ try:
34
+ bert = torch.load(bert_path)
35
+ assert bert.shape[-1] == len(phone)
36
+ except Exception:
37
+ bert = get_bert(text, word2ph, language_str, device)
38
+ assert bert.shape[-1] == len(phone)
39
+ torch.save(bert, bert_path)
40
+
41
+
42
+ if __name__ == "__main__":
43
+ parser = argparse.ArgumentParser()
44
+ parser.add_argument("-c", "--config", type=str, default="configs/config.json")
45
+ parser.add_argument("--num_processes", type=int, default=2)
46
+ args = parser.parse_args()
47
+ config_path = args.config
48
+ hps = utils.get_hparams_from_file(config_path)
49
+ lines = []
50
+ with open(hps.data.training_files, encoding="utf-8") as f:
51
+ lines.extend(f.readlines())
52
+
53
+ with open(hps.data.validation_files, encoding="utf-8") as f:
54
+ lines.extend(f.readlines())
55
+
56
+ num_processes = args.num_processes
57
+ with Pool(processes=num_processes) as pool:
58
+ for _ in tqdm(pool.imap_unordered(process_line, lines), total=len(lines)):
59
+ pass
commons.py ADDED
@@ -0,0 +1,160 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import torch
3
+ from torch.nn import functional as F
4
+
5
+
6
+ def init_weights(m, mean=0.0, std=0.01):
7
+ classname = m.__class__.__name__
8
+ if classname.find("Conv") != -1:
9
+ m.weight.data.normal_(mean, std)
10
+
11
+
12
+ def get_padding(kernel_size, dilation=1):
13
+ return int((kernel_size * dilation - dilation) / 2)
14
+
15
+
16
+ def convert_pad_shape(pad_shape):
17
+ layer = pad_shape[::-1]
18
+ pad_shape = [item for sublist in layer for item in sublist]
19
+ return pad_shape
20
+
21
+
22
+ def intersperse(lst, item):
23
+ result = [item] * (len(lst) * 2 + 1)
24
+ result[1::2] = lst
25
+ return result
26
+
27
+
28
+ def kl_divergence(m_p, logs_p, m_q, logs_q):
29
+ """KL(P||Q)"""
30
+ kl = (logs_q - logs_p) - 0.5
31
+ kl += (
32
+ 0.5 * (torch.exp(2.0 * logs_p) + ((m_p - m_q) ** 2)) * torch.exp(-2.0 * logs_q)
33
+ )
34
+ return kl
35
+
36
+
37
+ def rand_gumbel(shape):
38
+ """Sample from the Gumbel distribution, protect from overflows."""
39
+ uniform_samples = torch.rand(shape) * 0.99998 + 0.00001
40
+ return -torch.log(-torch.log(uniform_samples))
41
+
42
+
43
+ def rand_gumbel_like(x):
44
+ g = rand_gumbel(x.size()).to(dtype=x.dtype, device=x.device)
45
+ return g
46
+
47
+
48
+ def slice_segments(x, ids_str, segment_size=4):
49
+ ret = torch.zeros_like(x[:, :, :segment_size])
50
+ for i in range(x.size(0)):
51
+ idx_str = ids_str[i]
52
+ idx_end = idx_str + segment_size
53
+ ret[i] = x[i, :, idx_str:idx_end]
54
+ return ret
55
+
56
+
57
+ def rand_slice_segments(x, x_lengths=None, segment_size=4):
58
+ b, d, t = x.size()
59
+ if x_lengths is None:
60
+ x_lengths = t
61
+ ids_str_max = x_lengths - segment_size + 1
62
+ ids_str = (torch.rand([b]).to(device=x.device) * ids_str_max).to(dtype=torch.long)
63
+ ret = slice_segments(x, ids_str, segment_size)
64
+ return ret, ids_str
65
+
66
+
67
+ def get_timing_signal_1d(length, channels, min_timescale=1.0, max_timescale=1.0e4):
68
+ position = torch.arange(length, dtype=torch.float)
69
+ num_timescales = channels // 2
70
+ log_timescale_increment = math.log(float(max_timescale) / float(min_timescale)) / (
71
+ num_timescales - 1
72
+ )
73
+ inv_timescales = min_timescale * torch.exp(
74
+ torch.arange(num_timescales, dtype=torch.float) * -log_timescale_increment
75
+ )
76
+ scaled_time = position.unsqueeze(0) * inv_timescales.unsqueeze(1)
77
+ signal = torch.cat([torch.sin(scaled_time), torch.cos(scaled_time)], 0)
78
+ signal = F.pad(signal, [0, 0, 0, channels % 2])
79
+ signal = signal.view(1, channels, length)
80
+ return signal
81
+
82
+
83
+ def add_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4):
84
+ b, channels, length = x.size()
85
+ signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale)
86
+ return x + signal.to(dtype=x.dtype, device=x.device)
87
+
88
+
89
+ def cat_timing_signal_1d(x, min_timescale=1.0, max_timescale=1.0e4, axis=1):
90
+ b, channels, length = x.size()
91
+ signal = get_timing_signal_1d(length, channels, min_timescale, max_timescale)
92
+ return torch.cat([x, signal.to(dtype=x.dtype, device=x.device)], axis)
93
+
94
+
95
+ def subsequent_mask(length):
96
+ mask = torch.tril(torch.ones(length, length)).unsqueeze(0).unsqueeze(0)
97
+ return mask
98
+
99
+
100
+ @torch.jit.script
101
+ def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
102
+ n_channels_int = n_channels[0]
103
+ in_act = input_a + input_b
104
+ t_act = torch.tanh(in_act[:, :n_channels_int, :])
105
+ s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
106
+ acts = t_act * s_act
107
+ return acts
108
+
109
+
110
+ def convert_pad_shape(pad_shape):
111
+ layer = pad_shape[::-1]
112
+ pad_shape = [item for sublist in layer for item in sublist]
113
+ return pad_shape
114
+
115
+
116
+ def shift_1d(x):
117
+ x = F.pad(x, convert_pad_shape([[0, 0], [0, 0], [1, 0]]))[:, :, :-1]
118
+ return x
119
+
120
+
121
+ def sequence_mask(length, max_length=None):
122
+ if max_length is None:
123
+ max_length = length.max()
124
+ x = torch.arange(max_length, dtype=length.dtype, device=length.device)
125
+ return x.unsqueeze(0) < length.unsqueeze(1)
126
+
127
+
128
+ def generate_path(duration, mask):
129
+ """
130
+ duration: [b, 1, t_x]
131
+ mask: [b, 1, t_y, t_x]
132
+ """
133
+
134
+ b, _, t_y, t_x = mask.shape
135
+ cum_duration = torch.cumsum(duration, -1)
136
+
137
+ cum_duration_flat = cum_duration.view(b * t_x)
138
+ path = sequence_mask(cum_duration_flat, t_y).to(mask.dtype)
139
+ path = path.view(b, t_x, t_y)
140
+ path = path - F.pad(path, convert_pad_shape([[0, 0], [1, 0], [0, 0]]))[:, :-1]
141
+ path = path.unsqueeze(1).transpose(2, 3) * mask
142
+ return path
143
+
144
+
145
+ def clip_grad_value_(parameters, clip_value, norm_type=2):
146
+ if isinstance(parameters, torch.Tensor):
147
+ parameters = [parameters]
148
+ parameters = list(filter(lambda p: p.grad is not None, parameters))
149
+ norm_type = float(norm_type)
150
+ if clip_value is not None:
151
+ clip_value = float(clip_value)
152
+
153
+ total_norm = 0
154
+ for p in parameters:
155
+ param_norm = p.grad.data.norm(norm_type)
156
+ total_norm += param_norm.item() ** norm_type
157
+ if clip_value is not None:
158
+ p.grad.data.clamp_(min=-clip_value, max=clip_value)
159
+ total_norm = total_norm ** (1.0 / norm_type)
160
+ return total_norm
configs/config.json ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "train": {
3
+ "log_interval": 200,
4
+ "eval_interval": 1000,
5
+ "seed": 52,
6
+ "epochs": 10000,
7
+ "learning_rate": 0.0003,
8
+ "betas": [
9
+ 0.8,
10
+ 0.99
11
+ ],
12
+ "eps": 1e-09,
13
+ "batch_size": 8,
14
+ "fp16_run": false,
15
+ "lr_decay": 0.999875,
16
+ "segment_size": 16384,
17
+ "init_lr_ratio": 1,
18
+ "warmup_epochs": 0,
19
+ "c_mel": 45,
20
+ "c_kl": 1.0,
21
+ "skip_optimizer": true
22
+ },
23
+ "data": {
24
+ "training_files": "filelists/train.list",
25
+ "validation_files": "filelists/val.list",
26
+ "max_wav_value": 32768.0,
27
+ "sampling_rate": 44100,
28
+ "filter_length": 2048,
29
+ "hop_length": 512,
30
+ "win_length": 2048,
31
+ "n_mel_channels": 128,
32
+ "mel_fmin": 0.0,
33
+ "mel_fmax": null,
34
+ "add_blank": true,
35
+ "n_speakers": 256,
36
+ "cleaned_text": true,
37
+ "spk2id": {
38
+ "hanser": 0
39
+ }
40
+ },
41
+ "model": {
42
+ "use_spk_conditioned_encoder": true,
43
+ "use_noise_scaled_mas": true,
44
+ "use_mel_posterior_encoder": false,
45
+ "use_duration_discriminator": true,
46
+ "inter_channels": 192,
47
+ "hidden_channels": 192,
48
+ "filter_channels": 768,
49
+ "n_heads": 2,
50
+ "n_layers": 6,
51
+ "kernel_size": 3,
52
+ "p_dropout": 0.1,
53
+ "resblock": "1",
54
+ "resblock_kernel_sizes": [
55
+ 3,
56
+ 7,
57
+ 11
58
+ ],
59
+ "resblock_dilation_sizes": [
60
+ [
61
+ 1,
62
+ 3,
63
+ 5
64
+ ],
65
+ [
66
+ 1,
67
+ 3,
68
+ 5
69
+ ],
70
+ [
71
+ 1,
72
+ 3,
73
+ 5
74
+ ]
75
+ ],
76
+ "upsample_rates": [
77
+ 8,
78
+ 8,
79
+ 2,
80
+ 2,
81
+ 2
82
+ ],
83
+ "upsample_initial_channel": 512,
84
+ "upsample_kernel_sizes": [
85
+ 16,
86
+ 16,
87
+ 8,
88
+ 2,
89
+ 2
90
+ ],
91
+ "n_layers_q": 3,
92
+ "use_spectral_norm": false,
93
+ "gin_channels": 256
94
+ }
95
+ }
data_utils.py ADDED
@@ -0,0 +1,406 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import random
3
+ import torch
4
+ import torch.utils.data
5
+ from tqdm import tqdm
6
+ from loguru import logger
7
+ import commons
8
+ from mel_processing import spectrogram_torch, mel_spectrogram_torch
9
+ from utils import load_wav_to_torch, load_filepaths_and_text
10
+ from text import cleaned_text_to_sequence, get_bert
11
+
12
+ """Multi speaker version"""
13
+
14
+
15
+ class TextAudioSpeakerLoader(torch.utils.data.Dataset):
16
+ """
17
+ 1) loads audio, speaker_id, text pairs
18
+ 2) normalizes text and converts them to sequences of integers
19
+ 3) computes spectrograms from audio files.
20
+ """
21
+
22
+ def __init__(self, audiopaths_sid_text, hparams):
23
+ self.audiopaths_sid_text = load_filepaths_and_text(audiopaths_sid_text)
24
+ self.max_wav_value = hparams.max_wav_value
25
+ self.sampling_rate = hparams.sampling_rate
26
+ self.filter_length = hparams.filter_length
27
+ self.hop_length = hparams.hop_length
28
+ self.win_length = hparams.win_length
29
+ self.sampling_rate = hparams.sampling_rate
30
+ self.spk_map = hparams.spk2id
31
+ self.hparams = hparams
32
+
33
+ self.use_mel_spec_posterior = getattr(
34
+ hparams, "use_mel_posterior_encoder", False
35
+ )
36
+ if self.use_mel_spec_posterior:
37
+ self.n_mel_channels = getattr(hparams, "n_mel_channels", 80)
38
+
39
+ self.cleaned_text = getattr(hparams, "cleaned_text", False)
40
+
41
+ self.add_blank = hparams.add_blank
42
+ self.min_text_len = getattr(hparams, "min_text_len", 1)
43
+ self.max_text_len = getattr(hparams, "max_text_len", 300)
44
+
45
+ random.seed(1234)
46
+ random.shuffle(self.audiopaths_sid_text)
47
+ self._filter()
48
+
49
+ def _filter(self):
50
+ """
51
+ Filter text & store spec lengths
52
+ """
53
+ # Store spectrogram lengths for Bucketing
54
+ # wav_length ~= file_size / (wav_channels * Bytes per dim) = file_size / (1 * 2)
55
+ # spec_length = wav_length // hop_length
56
+
57
+ audiopaths_sid_text_new = []
58
+ lengths = []
59
+ skipped = 0
60
+ logger.info("Init dataset...")
61
+ for _id, spk, language, text, phones, tone, word2ph in tqdm(
62
+ self.audiopaths_sid_text
63
+ ):
64
+ audiopath = f"{_id}"
65
+ if self.min_text_len <= len(phones) and len(phones) <= self.max_text_len:
66
+ phones = phones.split(" ")
67
+ tone = [int(i) for i in tone.split(" ")]
68
+ word2ph = [int(i) for i in word2ph.split(" ")]
69
+ audiopaths_sid_text_new.append(
70
+ [audiopath, spk, language, text, phones, tone, word2ph]
71
+ )
72
+ lengths.append(os.path.getsize(audiopath) // (2 * self.hop_length))
73
+ else:
74
+ skipped += 1
75
+ logger.info(
76
+ "skipped: "
77
+ + str(skipped)
78
+ + ", total: "
79
+ + str(len(self.audiopaths_sid_text))
80
+ )
81
+ self.audiopaths_sid_text = audiopaths_sid_text_new
82
+ self.lengths = lengths
83
+
84
+ def get_audio_text_speaker_pair(self, audiopath_sid_text):
85
+ # separate filename, speaker_id and text
86
+ audiopath, sid, language, text, phones, tone, word2ph = audiopath_sid_text
87
+
88
+ bert, ja_bert, phones, tone, language = self.get_text(
89
+ text, word2ph, phones, tone, language, audiopath
90
+ )
91
+
92
+ spec, wav = self.get_audio(audiopath)
93
+ sid = torch.LongTensor([int(self.spk_map[sid])])
94
+ return (phones, spec, wav, sid, tone, language, bert, ja_bert)
95
+
96
+ def get_audio(self, filename):
97
+ audio, sampling_rate = load_wav_to_torch(filename)
98
+ if sampling_rate != self.sampling_rate:
99
+ raise ValueError(
100
+ "{} {} SR doesn't match target {} SR".format(
101
+ filename, sampling_rate, self.sampling_rate
102
+ )
103
+ )
104
+ audio_norm = audio / self.max_wav_value
105
+ audio_norm = audio_norm.unsqueeze(0)
106
+ spec_filename = filename.replace(".wav", ".spec.pt")
107
+ if self.use_mel_spec_posterior:
108
+ spec_filename = spec_filename.replace(".spec.pt", ".mel.pt")
109
+ try:
110
+ spec = torch.load(spec_filename)
111
+ except:
112
+ if self.use_mel_spec_posterior:
113
+ spec = mel_spectrogram_torch(
114
+ audio_norm,
115
+ self.filter_length,
116
+ self.n_mel_channels,
117
+ self.sampling_rate,
118
+ self.hop_length,
119
+ self.win_length,
120
+ self.hparams.mel_fmin,
121
+ self.hparams.mel_fmax,
122
+ center=False,
123
+ )
124
+ else:
125
+ spec = spectrogram_torch(
126
+ audio_norm,
127
+ self.filter_length,
128
+ self.sampling_rate,
129
+ self.hop_length,
130
+ self.win_length,
131
+ center=False,
132
+ )
133
+ spec = torch.squeeze(spec, 0)
134
+ torch.save(spec, spec_filename)
135
+ return spec, audio_norm
136
+
137
+ def get_text(self, text, word2ph, phone, tone, language_str, wav_path):
138
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
139
+ if self.add_blank:
140
+ phone = commons.intersperse(phone, 0)
141
+ tone = commons.intersperse(tone, 0)
142
+ language = commons.intersperse(language, 0)
143
+ for i in range(len(word2ph)):
144
+ word2ph[i] = word2ph[i] * 2
145
+ word2ph[0] += 1
146
+ bert_path = wav_path.replace(".wav", ".bert.pt")
147
+ try:
148
+ bert = torch.load(bert_path)
149
+ assert bert.shape[-1] == len(phone)
150
+ except:
151
+ bert = get_bert(text, word2ph, language_str)
152
+ torch.save(bert, bert_path)
153
+ assert bert.shape[-1] == len(phone), phone
154
+
155
+ if language_str == "ZH":
156
+ bert = bert
157
+ ja_bert = torch.zeros(768, len(phone))
158
+ elif language_str == "JP":
159
+ ja_bert = bert
160
+ bert = torch.zeros(1024, len(phone))
161
+ else:
162
+ bert = torch.zeros(1024, len(phone))
163
+ ja_bert = torch.zeros(768, len(phone))
164
+ assert bert.shape[-1] == len(phone), (
165
+ bert.shape,
166
+ len(phone),
167
+ sum(word2ph),
168
+ p1,
169
+ p2,
170
+ t1,
171
+ t2,
172
+ pold,
173
+ pold2,
174
+ word2ph,
175
+ text,
176
+ w2pho,
177
+ )
178
+ phone = torch.LongTensor(phone)
179
+ tone = torch.LongTensor(tone)
180
+ language = torch.LongTensor(language)
181
+ return bert, ja_bert, phone, tone, language
182
+
183
+ def get_sid(self, sid):
184
+ sid = torch.LongTensor([int(sid)])
185
+ return sid
186
+
187
+ def __getitem__(self, index):
188
+ return self.get_audio_text_speaker_pair(self.audiopaths_sid_text[index])
189
+
190
+ def __len__(self):
191
+ return len(self.audiopaths_sid_text)
192
+
193
+
194
+ class TextAudioSpeakerCollate:
195
+ """Zero-pads model inputs and targets"""
196
+
197
+ def __init__(self, return_ids=False):
198
+ self.return_ids = return_ids
199
+
200
+ def __call__(self, batch):
201
+ """Collate's training batch from normalized text, audio and speaker identities
202
+ PARAMS
203
+ ------
204
+ batch: [text_normalized, spec_normalized, wav_normalized, sid]
205
+ """
206
+ # Right zero-pad all one-hot text sequences to max input length
207
+ _, ids_sorted_decreasing = torch.sort(
208
+ torch.LongTensor([x[1].size(1) for x in batch]), dim=0, descending=True
209
+ )
210
+
211
+ max_text_len = max([len(x[0]) for x in batch])
212
+ max_spec_len = max([x[1].size(1) for x in batch])
213
+ max_wav_len = max([x[2].size(1) for x in batch])
214
+
215
+ text_lengths = torch.LongTensor(len(batch))
216
+ spec_lengths = torch.LongTensor(len(batch))
217
+ wav_lengths = torch.LongTensor(len(batch))
218
+ sid = torch.LongTensor(len(batch))
219
+
220
+ text_padded = torch.LongTensor(len(batch), max_text_len)
221
+ tone_padded = torch.LongTensor(len(batch), max_text_len)
222
+ language_padded = torch.LongTensor(len(batch), max_text_len)
223
+ bert_padded = torch.FloatTensor(len(batch), 1024, max_text_len)
224
+ ja_bert_padded = torch.FloatTensor(len(batch), 768, max_text_len)
225
+
226
+ spec_padded = torch.FloatTensor(len(batch), batch[0][1].size(0), max_spec_len)
227
+ wav_padded = torch.FloatTensor(len(batch), 1, max_wav_len)
228
+ text_padded.zero_()
229
+ tone_padded.zero_()
230
+ language_padded.zero_()
231
+ spec_padded.zero_()
232
+ wav_padded.zero_()
233
+ bert_padded.zero_()
234
+ ja_bert_padded.zero_()
235
+ for i in range(len(ids_sorted_decreasing)):
236
+ row = batch[ids_sorted_decreasing[i]]
237
+
238
+ text = row[0]
239
+ text_padded[i, : text.size(0)] = text
240
+ text_lengths[i] = text.size(0)
241
+
242
+ spec = row[1]
243
+ spec_padded[i, :, : spec.size(1)] = spec
244
+ spec_lengths[i] = spec.size(1)
245
+
246
+ wav = row[2]
247
+ wav_padded[i, :, : wav.size(1)] = wav
248
+ wav_lengths[i] = wav.size(1)
249
+
250
+ sid[i] = row[3]
251
+
252
+ tone = row[4]
253
+ tone_padded[i, : tone.size(0)] = tone
254
+
255
+ language = row[5]
256
+ language_padded[i, : language.size(0)] = language
257
+
258
+ bert = row[6]
259
+ bert_padded[i, :, : bert.size(1)] = bert
260
+
261
+ ja_bert = row[7]
262
+ ja_bert_padded[i, :, : ja_bert.size(1)] = ja_bert
263
+
264
+ return (
265
+ text_padded,
266
+ text_lengths,
267
+ spec_padded,
268
+ spec_lengths,
269
+ wav_padded,
270
+ wav_lengths,
271
+ sid,
272
+ tone_padded,
273
+ language_padded,
274
+ bert_padded,
275
+ ja_bert_padded,
276
+ )
277
+
278
+
279
+ class DistributedBucketSampler(torch.utils.data.distributed.DistributedSampler):
280
+ """
281
+ Maintain similar input lengths in a batch.
282
+ Length groups are specified by boundaries.
283
+ Ex) boundaries = [b1, b2, b3] -> any batch is included either {x | b1 < length(x) <=b2} or {x | b2 < length(x) <= b3}.
284
+
285
+ It removes samples which are not included in the boundaries.
286
+ Ex) boundaries = [b1, b2, b3] -> any x s.t. length(x) <= b1 or length(x) > b3 are discarded.
287
+ """
288
+
289
+ def __init__(
290
+ self,
291
+ dataset,
292
+ batch_size,
293
+ boundaries,
294
+ num_replicas=None,
295
+ rank=None,
296
+ shuffle=True,
297
+ ):
298
+ super().__init__(dataset, num_replicas=num_replicas, rank=rank, shuffle=shuffle)
299
+ self.lengths = dataset.lengths
300
+ self.batch_size = batch_size
301
+ self.boundaries = boundaries
302
+
303
+ self.buckets, self.num_samples_per_bucket = self._create_buckets()
304
+ self.total_size = sum(self.num_samples_per_bucket)
305
+ self.num_samples = self.total_size // self.num_replicas
306
+
307
+ def _create_buckets(self):
308
+ buckets = [[] for _ in range(len(self.boundaries) - 1)]
309
+ for i in range(len(self.lengths)):
310
+ length = self.lengths[i]
311
+ idx_bucket = self._bisect(length)
312
+ if idx_bucket != -1:
313
+ buckets[idx_bucket].append(i)
314
+
315
+ try:
316
+ for i in range(len(buckets) - 1, 0, -1):
317
+ if len(buckets[i]) == 0:
318
+ buckets.pop(i)
319
+ self.boundaries.pop(i + 1)
320
+ assert all(len(bucket) > 0 for bucket in buckets)
321
+ # When one bucket is not traversed
322
+ except Exception as e:
323
+ print("Bucket warning ", e)
324
+ for i in range(len(buckets) - 1, -1, -1):
325
+ if len(buckets[i]) == 0:
326
+ buckets.pop(i)
327
+ self.boundaries.pop(i + 1)
328
+
329
+ num_samples_per_bucket = []
330
+ for i in range(len(buckets)):
331
+ len_bucket = len(buckets[i])
332
+ total_batch_size = self.num_replicas * self.batch_size
333
+ rem = (
334
+ total_batch_size - (len_bucket % total_batch_size)
335
+ ) % total_batch_size
336
+ num_samples_per_bucket.append(len_bucket + rem)
337
+ return buckets, num_samples_per_bucket
338
+
339
+ def __iter__(self):
340
+ # deterministically shuffle based on epoch
341
+ g = torch.Generator()
342
+ g.manual_seed(self.epoch)
343
+
344
+ indices = []
345
+ if self.shuffle:
346
+ for bucket in self.buckets:
347
+ indices.append(torch.randperm(len(bucket), generator=g).tolist())
348
+ else:
349
+ for bucket in self.buckets:
350
+ indices.append(list(range(len(bucket))))
351
+
352
+ batches = []
353
+ for i in range(len(self.buckets)):
354
+ bucket = self.buckets[i]
355
+ len_bucket = len(bucket)
356
+ if len_bucket == 0:
357
+ continue
358
+ ids_bucket = indices[i]
359
+ num_samples_bucket = self.num_samples_per_bucket[i]
360
+
361
+ # add extra samples to make it evenly divisible
362
+ rem = num_samples_bucket - len_bucket
363
+ ids_bucket = (
364
+ ids_bucket
365
+ + ids_bucket * (rem // len_bucket)
366
+ + ids_bucket[: (rem % len_bucket)]
367
+ )
368
+
369
+ # subsample
370
+ ids_bucket = ids_bucket[self.rank :: self.num_replicas]
371
+
372
+ # batching
373
+ for j in range(len(ids_bucket) // self.batch_size):
374
+ batch = [
375
+ bucket[idx]
376
+ for idx in ids_bucket[
377
+ j * self.batch_size : (j + 1) * self.batch_size
378
+ ]
379
+ ]
380
+ batches.append(batch)
381
+
382
+ if self.shuffle:
383
+ batch_ids = torch.randperm(len(batches), generator=g).tolist()
384
+ batches = [batches[i] for i in batch_ids]
385
+ self.batches = batches
386
+
387
+ assert len(self.batches) * self.batch_size == self.num_samples
388
+ return iter(self.batches)
389
+
390
+ def _bisect(self, x, lo=0, hi=None):
391
+ if hi is None:
392
+ hi = len(self.boundaries) - 1
393
+
394
+ if hi > lo:
395
+ mid = (hi + lo) // 2
396
+ if self.boundaries[mid] < x and x <= self.boundaries[mid + 1]:
397
+ return mid
398
+ elif x <= self.boundaries[mid]:
399
+ return self._bisect(x, lo, mid)
400
+ else:
401
+ return self._bisect(x, mid + 1, hi)
402
+ else:
403
+ return -1
404
+
405
+ def __len__(self):
406
+ return self.num_samples // self.batch_size
logs/hanser/G_8000.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:782249a78ac551bda545bdf2126bd20acec871d0f69a027f5047b6f87564e68d
3
+ size 857912255
losses.py ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+
3
+
4
+ def feature_loss(fmap_r, fmap_g):
5
+ loss = 0
6
+ for dr, dg in zip(fmap_r, fmap_g):
7
+ for rl, gl in zip(dr, dg):
8
+ rl = rl.float().detach()
9
+ gl = gl.float()
10
+ loss += torch.mean(torch.abs(rl - gl))
11
+
12
+ return loss * 2
13
+
14
+
15
+ def discriminator_loss(disc_real_outputs, disc_generated_outputs):
16
+ loss = 0
17
+ r_losses = []
18
+ g_losses = []
19
+ for dr, dg in zip(disc_real_outputs, disc_generated_outputs):
20
+ dr = dr.float()
21
+ dg = dg.float()
22
+ r_loss = torch.mean((1 - dr) ** 2)
23
+ g_loss = torch.mean(dg**2)
24
+ loss += r_loss + g_loss
25
+ r_losses.append(r_loss.item())
26
+ g_losses.append(g_loss.item())
27
+
28
+ return loss, r_losses, g_losses
29
+
30
+
31
+ def generator_loss(disc_outputs):
32
+ loss = 0
33
+ gen_losses = []
34
+ for dg in disc_outputs:
35
+ dg = dg.float()
36
+ l = torch.mean((1 - dg) ** 2)
37
+ gen_losses.append(l)
38
+ loss += l
39
+
40
+ return loss, gen_losses
41
+
42
+
43
+ def kl_loss(z_p, logs_q, m_p, logs_p, z_mask):
44
+ """
45
+ z_p, logs_q: [b, h, t_t]
46
+ m_p, logs_p: [b, h, t_t]
47
+ """
48
+ z_p = z_p.float()
49
+ logs_q = logs_q.float()
50
+ m_p = m_p.float()
51
+ logs_p = logs_p.float()
52
+ z_mask = z_mask.float()
53
+
54
+ kl = logs_p - logs_q - 0.5
55
+ kl += 0.5 * ((z_p - m_p) ** 2) * torch.exp(-2.0 * logs_p)
56
+ kl = torch.sum(kl * z_mask)
57
+ l = kl / torch.sum(z_mask)
58
+ return l
mel_processing.py ADDED
@@ -0,0 +1,139 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ import torch.utils.data
3
+ from librosa.filters import mel as librosa_mel_fn
4
+
5
+ MAX_WAV_VALUE = 32768.0
6
+
7
+
8
+ def dynamic_range_compression_torch(x, C=1, clip_val=1e-5):
9
+ """
10
+ PARAMS
11
+ ------
12
+ C: compression factor
13
+ """
14
+ return torch.log(torch.clamp(x, min=clip_val) * C)
15
+
16
+
17
+ def dynamic_range_decompression_torch(x, C=1):
18
+ """
19
+ PARAMS
20
+ ------
21
+ C: compression factor used to compress
22
+ """
23
+ return torch.exp(x) / C
24
+
25
+
26
+ def spectral_normalize_torch(magnitudes):
27
+ output = dynamic_range_compression_torch(magnitudes)
28
+ return output
29
+
30
+
31
+ def spectral_de_normalize_torch(magnitudes):
32
+ output = dynamic_range_decompression_torch(magnitudes)
33
+ return output
34
+
35
+
36
+ mel_basis = {}
37
+ hann_window = {}
38
+
39
+
40
+ def spectrogram_torch(y, n_fft, sampling_rate, hop_size, win_size, center=False):
41
+ if torch.min(y) < -1.0:
42
+ print("min value is ", torch.min(y))
43
+ if torch.max(y) > 1.0:
44
+ print("max value is ", torch.max(y))
45
+
46
+ global hann_window
47
+ dtype_device = str(y.dtype) + "_" + str(y.device)
48
+ wnsize_dtype_device = str(win_size) + "_" + dtype_device
49
+ if wnsize_dtype_device not in hann_window:
50
+ hann_window[wnsize_dtype_device] = torch.hann_window(win_size).to(
51
+ dtype=y.dtype, device=y.device
52
+ )
53
+
54
+ y = torch.nn.functional.pad(
55
+ y.unsqueeze(1),
56
+ (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)),
57
+ mode="reflect",
58
+ )
59
+ y = y.squeeze(1)
60
+
61
+ spec = torch.stft(
62
+ y,
63
+ n_fft,
64
+ hop_length=hop_size,
65
+ win_length=win_size,
66
+ window=hann_window[wnsize_dtype_device],
67
+ center=center,
68
+ pad_mode="reflect",
69
+ normalized=False,
70
+ onesided=True,
71
+ return_complex=False,
72
+ )
73
+
74
+ spec = torch.sqrt(spec.pow(2).sum(-1) + 1e-6)
75
+ return spec
76
+
77
+
78
+ def spec_to_mel_torch(spec, n_fft, num_mels, sampling_rate, fmin, fmax):
79
+ global mel_basis
80
+ dtype_device = str(spec.dtype) + "_" + str(spec.device)
81
+ fmax_dtype_device = str(fmax) + "_" + dtype_device
82
+ if fmax_dtype_device not in mel_basis:
83
+ mel = librosa_mel_fn(sampling_rate, n_fft, num_mels, fmin, fmax)
84
+ mel_basis[fmax_dtype_device] = torch.from_numpy(mel).to(
85
+ dtype=spec.dtype, device=spec.device
86
+ )
87
+ spec = torch.matmul(mel_basis[fmax_dtype_device], spec)
88
+ spec = spectral_normalize_torch(spec)
89
+ return spec
90
+
91
+
92
+ def mel_spectrogram_torch(
93
+ y, n_fft, num_mels, sampling_rate, hop_size, win_size, fmin, fmax, center=False
94
+ ):
95
+ if torch.min(y) < -1.0:
96
+ print("min value is ", torch.min(y))
97
+ if torch.max(y) > 1.0:
98
+ print("max value is ", torch.max(y))
99
+
100
+ global mel_basis, hann_window
101
+ dtype_device = str(y.dtype) + "_" + str(y.device)
102
+ fmax_dtype_device = str(fmax) + "_" + dtype_device
103
+ wnsize_dtype_device = str(win_size) + "_" + dtype_device
104
+ if fmax_dtype_device not in mel_basis:
105
+ mel = librosa_mel_fn(sampling_rate, n_fft, num_mels, fmin, fmax)
106
+ mel_basis[fmax_dtype_device] = torch.from_numpy(mel).to(
107
+ dtype=y.dtype, device=y.device
108
+ )
109
+ if wnsize_dtype_device not in hann_window:
110
+ hann_window[wnsize_dtype_device] = torch.hann_window(win_size).to(
111
+ dtype=y.dtype, device=y.device
112
+ )
113
+
114
+ y = torch.nn.functional.pad(
115
+ y.unsqueeze(1),
116
+ (int((n_fft - hop_size) / 2), int((n_fft - hop_size) / 2)),
117
+ mode="reflect",
118
+ )
119
+ y = y.squeeze(1)
120
+
121
+ spec = torch.stft(
122
+ y,
123
+ n_fft,
124
+ hop_length=hop_size,
125
+ win_length=win_size,
126
+ window=hann_window[wnsize_dtype_device],
127
+ center=center,
128
+ pad_mode="reflect",
129
+ normalized=False,
130
+ onesided=True,
131
+ return_complex=False,
132
+ )
133
+
134
+ spec = torch.sqrt(spec.pow(2).sum(-1) + 1e-6)
135
+
136
+ spec = torch.matmul(mel_basis[fmax_dtype_device], spec)
137
+ spec = spectral_normalize_torch(spec)
138
+
139
+ return spec
models.py ADDED
@@ -0,0 +1,986 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import torch
3
+ from torch import nn
4
+ from torch.nn import functional as F
5
+
6
+ import commons
7
+ import modules
8
+ import attentions
9
+ import monotonic_align
10
+
11
+ from torch.nn import Conv1d, ConvTranspose1d, Conv2d
12
+ from torch.nn.utils import weight_norm, remove_weight_norm, spectral_norm
13
+
14
+ from commons import init_weights, get_padding
15
+ from text import symbols, num_tones, num_languages
16
+
17
+
18
+ class DurationDiscriminator(nn.Module): # vits2
19
+ def __init__(
20
+ self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
21
+ ):
22
+ super().__init__()
23
+
24
+ self.in_channels = in_channels
25
+ self.filter_channels = filter_channels
26
+ self.kernel_size = kernel_size
27
+ self.p_dropout = p_dropout
28
+ self.gin_channels = gin_channels
29
+
30
+ self.drop = nn.Dropout(p_dropout)
31
+ self.conv_1 = nn.Conv1d(
32
+ in_channels, filter_channels, kernel_size, padding=kernel_size // 2
33
+ )
34
+ self.norm_1 = modules.LayerNorm(filter_channels)
35
+ self.conv_2 = nn.Conv1d(
36
+ filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
37
+ )
38
+ self.norm_2 = modules.LayerNorm(filter_channels)
39
+ self.dur_proj = nn.Conv1d(1, filter_channels, 1)
40
+
41
+ self.pre_out_conv_1 = nn.Conv1d(
42
+ 2 * filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
43
+ )
44
+ self.pre_out_norm_1 = modules.LayerNorm(filter_channels)
45
+ self.pre_out_conv_2 = nn.Conv1d(
46
+ filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
47
+ )
48
+ self.pre_out_norm_2 = modules.LayerNorm(filter_channels)
49
+
50
+ if gin_channels != 0:
51
+ self.cond = nn.Conv1d(gin_channels, in_channels, 1)
52
+
53
+ self.output_layer = nn.Sequential(nn.Linear(filter_channels, 1), nn.Sigmoid())
54
+
55
+ def forward_probability(self, x, x_mask, dur, g=None):
56
+ dur = self.dur_proj(dur)
57
+ x = torch.cat([x, dur], dim=1)
58
+ x = self.pre_out_conv_1(x * x_mask)
59
+ x = torch.relu(x)
60
+ x = self.pre_out_norm_1(x)
61
+ x = self.drop(x)
62
+ x = self.pre_out_conv_2(x * x_mask)
63
+ x = torch.relu(x)
64
+ x = self.pre_out_norm_2(x)
65
+ x = self.drop(x)
66
+ x = x * x_mask
67
+ x = x.transpose(1, 2)
68
+ output_prob = self.output_layer(x)
69
+ return output_prob
70
+
71
+ def forward(self, x, x_mask, dur_r, dur_hat, g=None):
72
+ x = torch.detach(x)
73
+ if g is not None:
74
+ g = torch.detach(g)
75
+ x = x + self.cond(g)
76
+ x = self.conv_1(x * x_mask)
77
+ x = torch.relu(x)
78
+ x = self.norm_1(x)
79
+ x = self.drop(x)
80
+ x = self.conv_2(x * x_mask)
81
+ x = torch.relu(x)
82
+ x = self.norm_2(x)
83
+ x = self.drop(x)
84
+
85
+ output_probs = []
86
+ for dur in [dur_r, dur_hat]:
87
+ output_prob = self.forward_probability(x, x_mask, dur, g)
88
+ output_probs.append(output_prob)
89
+
90
+ return output_probs
91
+
92
+
93
+ class TransformerCouplingBlock(nn.Module):
94
+ def __init__(
95
+ self,
96
+ channels,
97
+ hidden_channels,
98
+ filter_channels,
99
+ n_heads,
100
+ n_layers,
101
+ kernel_size,
102
+ p_dropout,
103
+ n_flows=4,
104
+ gin_channels=0,
105
+ share_parameter=False,
106
+ ):
107
+ super().__init__()
108
+ self.channels = channels
109
+ self.hidden_channels = hidden_channels
110
+ self.kernel_size = kernel_size
111
+ self.n_layers = n_layers
112
+ self.n_flows = n_flows
113
+ self.gin_channels = gin_channels
114
+
115
+ self.flows = nn.ModuleList()
116
+
117
+ self.wn = (
118
+ attentions.FFT(
119
+ hidden_channels,
120
+ filter_channels,
121
+ n_heads,
122
+ n_layers,
123
+ kernel_size,
124
+ p_dropout,
125
+ isflow=True,
126
+ gin_channels=self.gin_channels,
127
+ )
128
+ if share_parameter
129
+ else None
130
+ )
131
+
132
+ for i in range(n_flows):
133
+ self.flows.append(
134
+ modules.TransformerCouplingLayer(
135
+ channels,
136
+ hidden_channels,
137
+ kernel_size,
138
+ n_layers,
139
+ n_heads,
140
+ p_dropout,
141
+ filter_channels,
142
+ mean_only=True,
143
+ wn_sharing_parameter=self.wn,
144
+ gin_channels=self.gin_channels,
145
+ )
146
+ )
147
+ self.flows.append(modules.Flip())
148
+
149
+ def forward(self, x, x_mask, g=None, reverse=False):
150
+ if not reverse:
151
+ for flow in self.flows:
152
+ x, _ = flow(x, x_mask, g=g, reverse=reverse)
153
+ else:
154
+ for flow in reversed(self.flows):
155
+ x = flow(x, x_mask, g=g, reverse=reverse)
156
+ return x
157
+
158
+
159
+ class StochasticDurationPredictor(nn.Module):
160
+ def __init__(
161
+ self,
162
+ in_channels,
163
+ filter_channels,
164
+ kernel_size,
165
+ p_dropout,
166
+ n_flows=4,
167
+ gin_channels=0,
168
+ ):
169
+ super().__init__()
170
+ filter_channels = in_channels # it needs to be removed from future version.
171
+ self.in_channels = in_channels
172
+ self.filter_channels = filter_channels
173
+ self.kernel_size = kernel_size
174
+ self.p_dropout = p_dropout
175
+ self.n_flows = n_flows
176
+ self.gin_channels = gin_channels
177
+
178
+ self.log_flow = modules.Log()
179
+ self.flows = nn.ModuleList()
180
+ self.flows.append(modules.ElementwiseAffine(2))
181
+ for i in range(n_flows):
182
+ self.flows.append(
183
+ modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
184
+ )
185
+ self.flows.append(modules.Flip())
186
+
187
+ self.post_pre = nn.Conv1d(1, filter_channels, 1)
188
+ self.post_proj = nn.Conv1d(filter_channels, filter_channels, 1)
189
+ self.post_convs = modules.DDSConv(
190
+ filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
191
+ )
192
+ self.post_flows = nn.ModuleList()
193
+ self.post_flows.append(modules.ElementwiseAffine(2))
194
+ for i in range(4):
195
+ self.post_flows.append(
196
+ modules.ConvFlow(2, filter_channels, kernel_size, n_layers=3)
197
+ )
198
+ self.post_flows.append(modules.Flip())
199
+
200
+ self.pre = nn.Conv1d(in_channels, filter_channels, 1)
201
+ self.proj = nn.Conv1d(filter_channels, filter_channels, 1)
202
+ self.convs = modules.DDSConv(
203
+ filter_channels, kernel_size, n_layers=3, p_dropout=p_dropout
204
+ )
205
+ if gin_channels != 0:
206
+ self.cond = nn.Conv1d(gin_channels, filter_channels, 1)
207
+
208
+ def forward(self, x, x_mask, w=None, g=None, reverse=False, noise_scale=1.0):
209
+ x = torch.detach(x)
210
+ x = self.pre(x)
211
+ if g is not None:
212
+ g = torch.detach(g)
213
+ x = x + self.cond(g)
214
+ x = self.convs(x, x_mask)
215
+ x = self.proj(x) * x_mask
216
+
217
+ if not reverse:
218
+ flows = self.flows
219
+ assert w is not None
220
+
221
+ logdet_tot_q = 0
222
+ h_w = self.post_pre(w)
223
+ h_w = self.post_convs(h_w, x_mask)
224
+ h_w = self.post_proj(h_w) * x_mask
225
+ e_q = (
226
+ torch.randn(w.size(0), 2, w.size(2)).to(device=x.device, dtype=x.dtype)
227
+ * x_mask
228
+ )
229
+ z_q = e_q
230
+ for flow in self.post_flows:
231
+ z_q, logdet_q = flow(z_q, x_mask, g=(x + h_w))
232
+ logdet_tot_q += logdet_q
233
+ z_u, z1 = torch.split(z_q, [1, 1], 1)
234
+ u = torch.sigmoid(z_u) * x_mask
235
+ z0 = (w - u) * x_mask
236
+ logdet_tot_q += torch.sum(
237
+ (F.logsigmoid(z_u) + F.logsigmoid(-z_u)) * x_mask, [1, 2]
238
+ )
239
+ logq = (
240
+ torch.sum(-0.5 * (math.log(2 * math.pi) + (e_q**2)) * x_mask, [1, 2])
241
+ - logdet_tot_q
242
+ )
243
+
244
+ logdet_tot = 0
245
+ z0, logdet = self.log_flow(z0, x_mask)
246
+ logdet_tot += logdet
247
+ z = torch.cat([z0, z1], 1)
248
+ for flow in flows:
249
+ z, logdet = flow(z, x_mask, g=x, reverse=reverse)
250
+ logdet_tot = logdet_tot + logdet
251
+ nll = (
252
+ torch.sum(0.5 * (math.log(2 * math.pi) + (z**2)) * x_mask, [1, 2])
253
+ - logdet_tot
254
+ )
255
+ return nll + logq # [b]
256
+ else:
257
+ flows = list(reversed(self.flows))
258
+ flows = flows[:-2] + [flows[-1]] # remove a useless vflow
259
+ z = (
260
+ torch.randn(x.size(0), 2, x.size(2)).to(device=x.device, dtype=x.dtype)
261
+ * noise_scale
262
+ )
263
+ for flow in flows:
264
+ z = flow(z, x_mask, g=x, reverse=reverse)
265
+ z0, z1 = torch.split(z, [1, 1], 1)
266
+ logw = z0
267
+ return logw
268
+
269
+
270
+ class DurationPredictor(nn.Module):
271
+ def __init__(
272
+ self, in_channels, filter_channels, kernel_size, p_dropout, gin_channels=0
273
+ ):
274
+ super().__init__()
275
+
276
+ self.in_channels = in_channels
277
+ self.filter_channels = filter_channels
278
+ self.kernel_size = kernel_size
279
+ self.p_dropout = p_dropout
280
+ self.gin_channels = gin_channels
281
+
282
+ self.drop = nn.Dropout(p_dropout)
283
+ self.conv_1 = nn.Conv1d(
284
+ in_channels, filter_channels, kernel_size, padding=kernel_size // 2
285
+ )
286
+ self.norm_1 = modules.LayerNorm(filter_channels)
287
+ self.conv_2 = nn.Conv1d(
288
+ filter_channels, filter_channels, kernel_size, padding=kernel_size // 2
289
+ )
290
+ self.norm_2 = modules.LayerNorm(filter_channels)
291
+ self.proj = nn.Conv1d(filter_channels, 1, 1)
292
+
293
+ if gin_channels != 0:
294
+ self.cond = nn.Conv1d(gin_channels, in_channels, 1)
295
+
296
+ def forward(self, x, x_mask, g=None):
297
+ x = torch.detach(x)
298
+ if g is not None:
299
+ g = torch.detach(g)
300
+ x = x + self.cond(g)
301
+ x = self.conv_1(x * x_mask)
302
+ x = torch.relu(x)
303
+ x = self.norm_1(x)
304
+ x = self.drop(x)
305
+ x = self.conv_2(x * x_mask)
306
+ x = torch.relu(x)
307
+ x = self.norm_2(x)
308
+ x = self.drop(x)
309
+ x = self.proj(x * x_mask)
310
+ return x * x_mask
311
+
312
+
313
+ class TextEncoder(nn.Module):
314
+ def __init__(
315
+ self,
316
+ n_vocab,
317
+ out_channels,
318
+ hidden_channels,
319
+ filter_channels,
320
+ n_heads,
321
+ n_layers,
322
+ kernel_size,
323
+ p_dropout,
324
+ gin_channels=0,
325
+ ):
326
+ super().__init__()
327
+ self.n_vocab = n_vocab
328
+ self.out_channels = out_channels
329
+ self.hidden_channels = hidden_channels
330
+ self.filter_channels = filter_channels
331
+ self.n_heads = n_heads
332
+ self.n_layers = n_layers
333
+ self.kernel_size = kernel_size
334
+ self.p_dropout = p_dropout
335
+ self.gin_channels = gin_channels
336
+ self.emb = nn.Embedding(len(symbols), hidden_channels)
337
+ nn.init.normal_(self.emb.weight, 0.0, hidden_channels**-0.5)
338
+ self.tone_emb = nn.Embedding(num_tones, hidden_channels)
339
+ nn.init.normal_(self.tone_emb.weight, 0.0, hidden_channels**-0.5)
340
+ self.language_emb = nn.Embedding(num_languages, hidden_channels)
341
+ nn.init.normal_(self.language_emb.weight, 0.0, hidden_channels**-0.5)
342
+ self.bert_proj = nn.Conv1d(1024, hidden_channels, 1)
343
+ self.ja_bert_proj = nn.Conv1d(768, hidden_channels, 1)
344
+
345
+ self.encoder = attentions.Encoder(
346
+ hidden_channels,
347
+ filter_channels,
348
+ n_heads,
349
+ n_layers,
350
+ kernel_size,
351
+ p_dropout,
352
+ gin_channels=self.gin_channels,
353
+ )
354
+ self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
355
+
356
+ def forward(self, x, x_lengths, tone, language, bert, ja_bert, g=None):
357
+ bert_emb = self.bert_proj(bert).transpose(1, 2)
358
+ ja_bert_emb = self.ja_bert_proj(ja_bert).transpose(1, 2)
359
+ x = (
360
+ self.emb(x)
361
+ + self.tone_emb(tone)
362
+ + self.language_emb(language)
363
+ + bert_emb
364
+ + ja_bert_emb
365
+ ) * math.sqrt(
366
+ self.hidden_channels
367
+ ) # [b, t, h]
368
+ x = torch.transpose(x, 1, -1) # [b, h, t]
369
+ x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
370
+ x.dtype
371
+ )
372
+
373
+ x = self.encoder(x * x_mask, x_mask, g=g)
374
+ stats = self.proj(x) * x_mask
375
+
376
+ m, logs = torch.split(stats, self.out_channels, dim=1)
377
+ return x, m, logs, x_mask
378
+
379
+
380
+ class ResidualCouplingBlock(nn.Module):
381
+ def __init__(
382
+ self,
383
+ channels,
384
+ hidden_channels,
385
+ kernel_size,
386
+ dilation_rate,
387
+ n_layers,
388
+ n_flows=4,
389
+ gin_channels=0,
390
+ ):
391
+ super().__init__()
392
+ self.channels = channels
393
+ self.hidden_channels = hidden_channels
394
+ self.kernel_size = kernel_size
395
+ self.dilation_rate = dilation_rate
396
+ self.n_layers = n_layers
397
+ self.n_flows = n_flows
398
+ self.gin_channels = gin_channels
399
+
400
+ self.flows = nn.ModuleList()
401
+ for i in range(n_flows):
402
+ self.flows.append(
403
+ modules.ResidualCouplingLayer(
404
+ channels,
405
+ hidden_channels,
406
+ kernel_size,
407
+ dilation_rate,
408
+ n_layers,
409
+ gin_channels=gin_channels,
410
+ mean_only=True,
411
+ )
412
+ )
413
+ self.flows.append(modules.Flip())
414
+
415
+ def forward(self, x, x_mask, g=None, reverse=False):
416
+ if not reverse:
417
+ for flow in self.flows:
418
+ x, _ = flow(x, x_mask, g=g, reverse=reverse)
419
+ else:
420
+ for flow in reversed(self.flows):
421
+ x = flow(x, x_mask, g=g, reverse=reverse)
422
+ return x
423
+
424
+
425
+ class PosteriorEncoder(nn.Module):
426
+ def __init__(
427
+ self,
428
+ in_channels,
429
+ out_channels,
430
+ hidden_channels,
431
+ kernel_size,
432
+ dilation_rate,
433
+ n_layers,
434
+ gin_channels=0,
435
+ ):
436
+ super().__init__()
437
+ self.in_channels = in_channels
438
+ self.out_channels = out_channels
439
+ self.hidden_channels = hidden_channels
440
+ self.kernel_size = kernel_size
441
+ self.dilation_rate = dilation_rate
442
+ self.n_layers = n_layers
443
+ self.gin_channels = gin_channels
444
+
445
+ self.pre = nn.Conv1d(in_channels, hidden_channels, 1)
446
+ self.enc = modules.WN(
447
+ hidden_channels,
448
+ kernel_size,
449
+ dilation_rate,
450
+ n_layers,
451
+ gin_channels=gin_channels,
452
+ )
453
+ self.proj = nn.Conv1d(hidden_channels, out_channels * 2, 1)
454
+
455
+ def forward(self, x, x_lengths, g=None):
456
+ x_mask = torch.unsqueeze(commons.sequence_mask(x_lengths, x.size(2)), 1).to(
457
+ x.dtype
458
+ )
459
+ x = self.pre(x) * x_mask
460
+ x = self.enc(x, x_mask, g=g)
461
+ stats = self.proj(x) * x_mask
462
+ m, logs = torch.split(stats, self.out_channels, dim=1)
463
+ z = (m + torch.randn_like(m) * torch.exp(logs)) * x_mask
464
+ return z, m, logs, x_mask
465
+
466
+
467
+ class Generator(torch.nn.Module):
468
+ def __init__(
469
+ self,
470
+ initial_channel,
471
+ resblock,
472
+ resblock_kernel_sizes,
473
+ resblock_dilation_sizes,
474
+ upsample_rates,
475
+ upsample_initial_channel,
476
+ upsample_kernel_sizes,
477
+ gin_channels=0,
478
+ ):
479
+ super(Generator, self).__init__()
480
+ self.num_kernels = len(resblock_kernel_sizes)
481
+ self.num_upsamples = len(upsample_rates)
482
+ self.conv_pre = Conv1d(
483
+ initial_channel, upsample_initial_channel, 7, 1, padding=3
484
+ )
485
+ resblock = modules.ResBlock1 if resblock == "1" else modules.ResBlock2
486
+
487
+ self.ups = nn.ModuleList()
488
+ for i, (u, k) in enumerate(zip(upsample_rates, upsample_kernel_sizes)):
489
+ self.ups.append(
490
+ weight_norm(
491
+ ConvTranspose1d(
492
+ upsample_initial_channel // (2**i),
493
+ upsample_initial_channel // (2 ** (i + 1)),
494
+ k,
495
+ u,
496
+ padding=(k - u) // 2,
497
+ )
498
+ )
499
+ )
500
+
501
+ self.resblocks = nn.ModuleList()
502
+ for i in range(len(self.ups)):
503
+ ch = upsample_initial_channel // (2 ** (i + 1))
504
+ for j, (k, d) in enumerate(
505
+ zip(resblock_kernel_sizes, resblock_dilation_sizes)
506
+ ):
507
+ self.resblocks.append(resblock(ch, k, d))
508
+
509
+ self.conv_post = Conv1d(ch, 1, 7, 1, padding=3, bias=False)
510
+ self.ups.apply(init_weights)
511
+
512
+ if gin_channels != 0:
513
+ self.cond = nn.Conv1d(gin_channels, upsample_initial_channel, 1)
514
+
515
+ def forward(self, x, g=None):
516
+ x = self.conv_pre(x)
517
+ if g is not None:
518
+ x = x + self.cond(g)
519
+
520
+ for i in range(self.num_upsamples):
521
+ x = F.leaky_relu(x, modules.LRELU_SLOPE)
522
+ x = self.ups[i](x)
523
+ xs = None
524
+ for j in range(self.num_kernels):
525
+ if xs is None:
526
+ xs = self.resblocks[i * self.num_kernels + j](x)
527
+ else:
528
+ xs += self.resblocks[i * self.num_kernels + j](x)
529
+ x = xs / self.num_kernels
530
+ x = F.leaky_relu(x)
531
+ x = self.conv_post(x)
532
+ x = torch.tanh(x)
533
+
534
+ return x
535
+
536
+ def remove_weight_norm(self):
537
+ print("Removing weight norm...")
538
+ for layer in self.ups:
539
+ remove_weight_norm(layer)
540
+ for layer in self.resblocks:
541
+ layer.remove_weight_norm()
542
+
543
+
544
+ class DiscriminatorP(torch.nn.Module):
545
+ def __init__(self, period, kernel_size=5, stride=3, use_spectral_norm=False):
546
+ super(DiscriminatorP, self).__init__()
547
+ self.period = period
548
+ self.use_spectral_norm = use_spectral_norm
549
+ norm_f = weight_norm if use_spectral_norm is False else spectral_norm
550
+ self.convs = nn.ModuleList(
551
+ [
552
+ norm_f(
553
+ Conv2d(
554
+ 1,
555
+ 32,
556
+ (kernel_size, 1),
557
+ (stride, 1),
558
+ padding=(get_padding(kernel_size, 1), 0),
559
+ )
560
+ ),
561
+ norm_f(
562
+ Conv2d(
563
+ 32,
564
+ 128,
565
+ (kernel_size, 1),
566
+ (stride, 1),
567
+ padding=(get_padding(kernel_size, 1), 0),
568
+ )
569
+ ),
570
+ norm_f(
571
+ Conv2d(
572
+ 128,
573
+ 512,
574
+ (kernel_size, 1),
575
+ (stride, 1),
576
+ padding=(get_padding(kernel_size, 1), 0),
577
+ )
578
+ ),
579
+ norm_f(
580
+ Conv2d(
581
+ 512,
582
+ 1024,
583
+ (kernel_size, 1),
584
+ (stride, 1),
585
+ padding=(get_padding(kernel_size, 1), 0),
586
+ )
587
+ ),
588
+ norm_f(
589
+ Conv2d(
590
+ 1024,
591
+ 1024,
592
+ (kernel_size, 1),
593
+ 1,
594
+ padding=(get_padding(kernel_size, 1), 0),
595
+ )
596
+ ),
597
+ ]
598
+ )
599
+ self.conv_post = norm_f(Conv2d(1024, 1, (3, 1), 1, padding=(1, 0)))
600
+
601
+ def forward(self, x):
602
+ fmap = []
603
+
604
+ # 1d to 2d
605
+ b, c, t = x.shape
606
+ if t % self.period != 0: # pad first
607
+ n_pad = self.period - (t % self.period)
608
+ x = F.pad(x, (0, n_pad), "reflect")
609
+ t = t + n_pad
610
+ x = x.view(b, c, t // self.period, self.period)
611
+
612
+ for layer in self.convs:
613
+ x = layer(x)
614
+ x = F.leaky_relu(x, modules.LRELU_SLOPE)
615
+ fmap.append(x)
616
+ x = self.conv_post(x)
617
+ fmap.append(x)
618
+ x = torch.flatten(x, 1, -1)
619
+
620
+ return x, fmap
621
+
622
+
623
+ class DiscriminatorS(torch.nn.Module):
624
+ def __init__(self, use_spectral_norm=False):
625
+ super(DiscriminatorS, self).__init__()
626
+ norm_f = weight_norm if use_spectral_norm is False else spectral_norm
627
+ self.convs = nn.ModuleList(
628
+ [
629
+ norm_f(Conv1d(1, 16, 15, 1, padding=7)),
630
+ norm_f(Conv1d(16, 64, 41, 4, groups=4, padding=20)),
631
+ norm_f(Conv1d(64, 256, 41, 4, groups=16, padding=20)),
632
+ norm_f(Conv1d(256, 1024, 41, 4, groups=64, padding=20)),
633
+ norm_f(Conv1d(1024, 1024, 41, 4, groups=256, padding=20)),
634
+ norm_f(Conv1d(1024, 1024, 5, 1, padding=2)),
635
+ ]
636
+ )
637
+ self.conv_post = norm_f(Conv1d(1024, 1, 3, 1, padding=1))
638
+
639
+ def forward(self, x):
640
+ fmap = []
641
+
642
+ for layer in self.convs:
643
+ x = layer(x)
644
+ x = F.leaky_relu(x, modules.LRELU_SLOPE)
645
+ fmap.append(x)
646
+ x = self.conv_post(x)
647
+ fmap.append(x)
648
+ x = torch.flatten(x, 1, -1)
649
+
650
+ return x, fmap
651
+
652
+
653
+ class MultiPeriodDiscriminator(torch.nn.Module):
654
+ def __init__(self, use_spectral_norm=False):
655
+ super(MultiPeriodDiscriminator, self).__init__()
656
+ periods = [2, 3, 5, 7, 11]
657
+
658
+ discs = [DiscriminatorS(use_spectral_norm=use_spectral_norm)]
659
+ discs = discs + [
660
+ DiscriminatorP(i, use_spectral_norm=use_spectral_norm) for i in periods
661
+ ]
662
+ self.discriminators = nn.ModuleList(discs)
663
+
664
+ def forward(self, y, y_hat):
665
+ y_d_rs = []
666
+ y_d_gs = []
667
+ fmap_rs = []
668
+ fmap_gs = []
669
+ for i, d in enumerate(self.discriminators):
670
+ y_d_r, fmap_r = d(y)
671
+ y_d_g, fmap_g = d(y_hat)
672
+ y_d_rs.append(y_d_r)
673
+ y_d_gs.append(y_d_g)
674
+ fmap_rs.append(fmap_r)
675
+ fmap_gs.append(fmap_g)
676
+
677
+ return y_d_rs, y_d_gs, fmap_rs, fmap_gs
678
+
679
+
680
+ class ReferenceEncoder(nn.Module):
681
+ """
682
+ inputs --- [N, Ty/r, n_mels*r] mels
683
+ outputs --- [N, ref_enc_gru_size]
684
+ """
685
+
686
+ def __init__(self, spec_channels, gin_channels=0):
687
+ super().__init__()
688
+ self.spec_channels = spec_channels
689
+ ref_enc_filters = [32, 32, 64, 64, 128, 128]
690
+ K = len(ref_enc_filters)
691
+ filters = [1] + ref_enc_filters
692
+ convs = [
693
+ weight_norm(
694
+ nn.Conv2d(
695
+ in_channels=filters[i],
696
+ out_channels=filters[i + 1],
697
+ kernel_size=(3, 3),
698
+ stride=(2, 2),
699
+ padding=(1, 1),
700
+ )
701
+ )
702
+ for i in range(K)
703
+ ]
704
+ self.convs = nn.ModuleList(convs)
705
+ # self.wns = nn.ModuleList([weight_norm(num_features=ref_enc_filters[i]) for i in range(K)]) # noqa: E501
706
+
707
+ out_channels = self.calculate_channels(spec_channels, 3, 2, 1, K)
708
+ self.gru = nn.GRU(
709
+ input_size=ref_enc_filters[-1] * out_channels,
710
+ hidden_size=256 // 2,
711
+ batch_first=True,
712
+ )
713
+ self.proj = nn.Linear(128, gin_channels)
714
+
715
+ def forward(self, inputs, mask=None):
716
+ N = inputs.size(0)
717
+ out = inputs.view(N, 1, -1, self.spec_channels) # [N, 1, Ty, n_freqs]
718
+ for conv in self.convs:
719
+ out = conv(out)
720
+ # out = wn(out)
721
+ out = F.relu(out) # [N, 128, Ty//2^K, n_mels//2^K]
722
+
723
+ out = out.transpose(1, 2) # [N, Ty//2^K, 128, n_mels//2^K]
724
+ T = out.size(1)
725
+ N = out.size(0)
726
+ out = out.contiguous().view(N, T, -1) # [N, Ty//2^K, 128*n_mels//2^K]
727
+
728
+ self.gru.flatten_parameters()
729
+ memory, out = self.gru(out) # out --- [1, N, 128]
730
+
731
+ return self.proj(out.squeeze(0))
732
+
733
+ def calculate_channels(self, L, kernel_size, stride, pad, n_convs):
734
+ for i in range(n_convs):
735
+ L = (L - kernel_size + 2 * pad) // stride + 1
736
+ return L
737
+
738
+
739
+ class SynthesizerTrn(nn.Module):
740
+ """
741
+ Synthesizer for Training
742
+ """
743
+
744
+ def __init__(
745
+ self,
746
+ n_vocab,
747
+ spec_channels,
748
+ segment_size,
749
+ inter_channels,
750
+ hidden_channels,
751
+ filter_channels,
752
+ n_heads,
753
+ n_layers,
754
+ kernel_size,
755
+ p_dropout,
756
+ resblock,
757
+ resblock_kernel_sizes,
758
+ resblock_dilation_sizes,
759
+ upsample_rates,
760
+ upsample_initial_channel,
761
+ upsample_kernel_sizes,
762
+ n_speakers=256,
763
+ gin_channels=256,
764
+ use_sdp=True,
765
+ n_flow_layer=4,
766
+ n_layers_trans_flow=6,
767
+ flow_share_parameter=False,
768
+ use_transformer_flow=True,
769
+ **kwargs
770
+ ):
771
+ super().__init__()
772
+ self.n_vocab = n_vocab
773
+ self.spec_channels = spec_channels
774
+ self.inter_channels = inter_channels
775
+ self.hidden_channels = hidden_channels
776
+ self.filter_channels = filter_channels
777
+ self.n_heads = n_heads
778
+ self.n_layers = n_layers
779
+ self.kernel_size = kernel_size
780
+ self.p_dropout = p_dropout
781
+ self.resblock = resblock
782
+ self.resblock_kernel_sizes = resblock_kernel_sizes
783
+ self.resblock_dilation_sizes = resblock_dilation_sizes
784
+ self.upsample_rates = upsample_rates
785
+ self.upsample_initial_channel = upsample_initial_channel
786
+ self.upsample_kernel_sizes = upsample_kernel_sizes
787
+ self.segment_size = segment_size
788
+ self.n_speakers = n_speakers
789
+ self.gin_channels = gin_channels
790
+ self.n_layers_trans_flow = n_layers_trans_flow
791
+ self.use_spk_conditioned_encoder = kwargs.get(
792
+ "use_spk_conditioned_encoder", True
793
+ )
794
+ self.use_sdp = use_sdp
795
+ self.use_noise_scaled_mas = kwargs.get("use_noise_scaled_mas", False)
796
+ self.mas_noise_scale_initial = kwargs.get("mas_noise_scale_initial", 0.01)
797
+ self.noise_scale_delta = kwargs.get("noise_scale_delta", 2e-6)
798
+ self.current_mas_noise_scale = self.mas_noise_scale_initial
799
+ if self.use_spk_conditioned_encoder and gin_channels > 0:
800
+ self.enc_gin_channels = gin_channels
801
+ self.enc_p = TextEncoder(
802
+ n_vocab,
803
+ inter_channels,
804
+ hidden_channels,
805
+ filter_channels,
806
+ n_heads,
807
+ n_layers,
808
+ kernel_size,
809
+ p_dropout,
810
+ gin_channels=self.enc_gin_channels,
811
+ )
812
+ self.dec = Generator(
813
+ inter_channels,
814
+ resblock,
815
+ resblock_kernel_sizes,
816
+ resblock_dilation_sizes,
817
+ upsample_rates,
818
+ upsample_initial_channel,
819
+ upsample_kernel_sizes,
820
+ gin_channels=gin_channels,
821
+ )
822
+ self.enc_q = PosteriorEncoder(
823
+ spec_channels,
824
+ inter_channels,
825
+ hidden_channels,
826
+ 5,
827
+ 1,
828
+ 16,
829
+ gin_channels=gin_channels,
830
+ )
831
+ if use_transformer_flow:
832
+ self.flow = TransformerCouplingBlock(
833
+ inter_channels,
834
+ hidden_channels,
835
+ filter_channels,
836
+ n_heads,
837
+ n_layers_trans_flow,
838
+ 5,
839
+ p_dropout,
840
+ n_flow_layer,
841
+ gin_channels=gin_channels,
842
+ share_parameter=flow_share_parameter,
843
+ )
844
+ else:
845
+ self.flow = ResidualCouplingBlock(
846
+ inter_channels,
847
+ hidden_channels,
848
+ 5,
849
+ 1,
850
+ n_flow_layer,
851
+ gin_channels=gin_channels,
852
+ )
853
+ self.sdp = StochasticDurationPredictor(
854
+ hidden_channels, 192, 3, 0.5, 4, gin_channels=gin_channels
855
+ )
856
+ self.dp = DurationPredictor(
857
+ hidden_channels, 256, 3, 0.5, gin_channels=gin_channels
858
+ )
859
+
860
+ if n_speakers > 1:
861
+ self.emb_g = nn.Embedding(n_speakers, gin_channels)
862
+ else:
863
+ self.ref_enc = ReferenceEncoder(spec_channels, gin_channels)
864
+
865
+ def forward(self, x, x_lengths, y, y_lengths, sid, tone, language, bert, ja_bert):
866
+ if self.n_speakers > 0:
867
+ g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
868
+ else:
869
+ g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
870
+ x, m_p, logs_p, x_mask = self.enc_p(
871
+ x, x_lengths, tone, language, bert, ja_bert, g=g
872
+ )
873
+ z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
874
+ z_p = self.flow(z, y_mask, g=g)
875
+
876
+ with torch.no_grad():
877
+ # negative cross-entropy
878
+ s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
879
+ neg_cent1 = torch.sum(
880
+ -0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True
881
+ ) # [b, 1, t_s]
882
+ neg_cent2 = torch.matmul(
883
+ -0.5 * (z_p**2).transpose(1, 2), s_p_sq_r
884
+ ) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
885
+ neg_cent3 = torch.matmul(
886
+ z_p.transpose(1, 2), (m_p * s_p_sq_r)
887
+ ) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
888
+ neg_cent4 = torch.sum(
889
+ -0.5 * (m_p**2) * s_p_sq_r, [1], keepdim=True
890
+ ) # [b, 1, t_s]
891
+ neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4
892
+ if self.use_noise_scaled_mas:
893
+ epsilon = (
894
+ torch.std(neg_cent)
895
+ * torch.randn_like(neg_cent)
896
+ * self.current_mas_noise_scale
897
+ )
898
+ neg_cent = neg_cent + epsilon
899
+
900
+ attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
901
+ attn = (
902
+ monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1))
903
+ .unsqueeze(1)
904
+ .detach()
905
+ )
906
+
907
+ w = attn.sum(2)
908
+
909
+ l_length_sdp = self.sdp(x, x_mask, w, g=g)
910
+ l_length_sdp = l_length_sdp / torch.sum(x_mask)
911
+
912
+ logw_ = torch.log(w + 1e-6) * x_mask
913
+ logw = self.dp(x, x_mask, g=g)
914
+ l_length_dp = torch.sum((logw - logw_) ** 2, [1, 2]) / torch.sum(
915
+ x_mask
916
+ ) # for averaging
917
+
918
+ l_length = l_length_dp + l_length_sdp
919
+
920
+ # expand prior
921
+ m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
922
+ logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)
923
+
924
+ z_slice, ids_slice = commons.rand_slice_segments(
925
+ z, y_lengths, self.segment_size
926
+ )
927
+ o = self.dec(z_slice, g=g)
928
+ return (
929
+ o,
930
+ l_length,
931
+ attn,
932
+ ids_slice,
933
+ x_mask,
934
+ y_mask,
935
+ (z, z_p, m_p, logs_p, m_q, logs_q),
936
+ (x, logw, logw_),
937
+ )
938
+
939
+ def infer(
940
+ self,
941
+ x,
942
+ x_lengths,
943
+ sid,
944
+ tone,
945
+ language,
946
+ bert,
947
+ ja_bert,
948
+ noise_scale=0.667,
949
+ length_scale=1,
950
+ noise_scale_w=0.8,
951
+ max_len=None,
952
+ sdp_ratio=0,
953
+ y=None,
954
+ ):
955
+ # x, m_p, logs_p, x_mask = self.enc_p(x, x_lengths, tone, language, bert)
956
+ # g = self.gst(y)
957
+ if self.n_speakers > 0:
958
+ g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
959
+ else:
960
+ g = self.ref_enc(y.transpose(1, 2)).unsqueeze(-1)
961
+ x, m_p, logs_p, x_mask = self.enc_p(
962
+ x, x_lengths, tone, language, bert, ja_bert, g=g
963
+ )
964
+ logw = self.sdp(x, x_mask, g=g, reverse=True, noise_scale=noise_scale_w) * (
965
+ sdp_ratio
966
+ ) + self.dp(x, x_mask, g=g) * (1 - sdp_ratio)
967
+ w = torch.exp(logw) * x_mask * length_scale
968
+ w_ceil = torch.ceil(w)
969
+ y_lengths = torch.clamp_min(torch.sum(w_ceil, [1, 2]), 1).long()
970
+ y_mask = torch.unsqueeze(commons.sequence_mask(y_lengths, None), 1).to(
971
+ x_mask.dtype
972
+ )
973
+ attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
974
+ attn = commons.generate_path(w_ceil, attn_mask)
975
+
976
+ m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(
977
+ 1, 2
978
+ ) # [b, t', t], [b, t, d] -> [b, d, t']
979
+ logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(
980
+ 1, 2
981
+ ) # [b, t', t], [b, t, d] -> [b, d, t']
982
+
983
+ z_p = m_p + torch.randn_like(m_p) * torch.exp(logs_p) * noise_scale
984
+ z = self.flow(z_p, y_mask, g=g, reverse=True)
985
+ o = self.dec((z * y_mask)[:, :, :max_len], g=g)
986
+ return o, attn, y_mask, (z, z_p, m_p, logs_p)
modules.py ADDED
@@ -0,0 +1,597 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import math
2
+ import torch
3
+ from torch import nn
4
+ from torch.nn import functional as F
5
+
6
+ from torch.nn import Conv1d
7
+ from torch.nn.utils import weight_norm, remove_weight_norm
8
+
9
+ import commons
10
+ from commons import init_weights, get_padding
11
+ from transforms import piecewise_rational_quadratic_transform
12
+ from attentions import Encoder
13
+
14
+ LRELU_SLOPE = 0.1
15
+
16
+
17
+ class LayerNorm(nn.Module):
18
+ def __init__(self, channels, eps=1e-5):
19
+ super().__init__()
20
+ self.channels = channels
21
+ self.eps = eps
22
+
23
+ self.gamma = nn.Parameter(torch.ones(channels))
24
+ self.beta = nn.Parameter(torch.zeros(channels))
25
+
26
+ def forward(self, x):
27
+ x = x.transpose(1, -1)
28
+ x = F.layer_norm(x, (self.channels,), self.gamma, self.beta, self.eps)
29
+ return x.transpose(1, -1)
30
+
31
+
32
+ class ConvReluNorm(nn.Module):
33
+ def __init__(
34
+ self,
35
+ in_channels,
36
+ hidden_channels,
37
+ out_channels,
38
+ kernel_size,
39
+ n_layers,
40
+ p_dropout,
41
+ ):
42
+ super().__init__()
43
+ self.in_channels = in_channels
44
+ self.hidden_channels = hidden_channels
45
+ self.out_channels = out_channels
46
+ self.kernel_size = kernel_size
47
+ self.n_layers = n_layers
48
+ self.p_dropout = p_dropout
49
+ assert n_layers > 1, "Number of layers should be larger than 0."
50
+
51
+ self.conv_layers = nn.ModuleList()
52
+ self.norm_layers = nn.ModuleList()
53
+ self.conv_layers.append(
54
+ nn.Conv1d(
55
+ in_channels, hidden_channels, kernel_size, padding=kernel_size // 2
56
+ )
57
+ )
58
+ self.norm_layers.append(LayerNorm(hidden_channels))
59
+ self.relu_drop = nn.Sequential(nn.ReLU(), nn.Dropout(p_dropout))
60
+ for _ in range(n_layers - 1):
61
+ self.conv_layers.append(
62
+ nn.Conv1d(
63
+ hidden_channels,
64
+ hidden_channels,
65
+ kernel_size,
66
+ padding=kernel_size // 2,
67
+ )
68
+ )
69
+ self.norm_layers.append(LayerNorm(hidden_channels))
70
+ self.proj = nn.Conv1d(hidden_channels, out_channels, 1)
71
+ self.proj.weight.data.zero_()
72
+ self.proj.bias.data.zero_()
73
+
74
+ def forward(self, x, x_mask):
75
+ x_org = x
76
+ for i in range(self.n_layers):
77
+ x = self.conv_layers[i](x * x_mask)
78
+ x = self.norm_layers[i](x)
79
+ x = self.relu_drop(x)
80
+ x = x_org + self.proj(x)
81
+ return x * x_mask
82
+
83
+
84
+ class DDSConv(nn.Module):
85
+ """
86
+ Dialted and Depth-Separable Convolution
87
+ """
88
+
89
+ def __init__(self, channels, kernel_size, n_layers, p_dropout=0.0):
90
+ super().__init__()
91
+ self.channels = channels
92
+ self.kernel_size = kernel_size
93
+ self.n_layers = n_layers
94
+ self.p_dropout = p_dropout
95
+
96
+ self.drop = nn.Dropout(p_dropout)
97
+ self.convs_sep = nn.ModuleList()
98
+ self.convs_1x1 = nn.ModuleList()
99
+ self.norms_1 = nn.ModuleList()
100
+ self.norms_2 = nn.ModuleList()
101
+ for i in range(n_layers):
102
+ dilation = kernel_size**i
103
+ padding = (kernel_size * dilation - dilation) // 2
104
+ self.convs_sep.append(
105
+ nn.Conv1d(
106
+ channels,
107
+ channels,
108
+ kernel_size,
109
+ groups=channels,
110
+ dilation=dilation,
111
+ padding=padding,
112
+ )
113
+ )
114
+ self.convs_1x1.append(nn.Conv1d(channels, channels, 1))
115
+ self.norms_1.append(LayerNorm(channels))
116
+ self.norms_2.append(LayerNorm(channels))
117
+
118
+ def forward(self, x, x_mask, g=None):
119
+ if g is not None:
120
+ x = x + g
121
+ for i in range(self.n_layers):
122
+ y = self.convs_sep[i](x * x_mask)
123
+ y = self.norms_1[i](y)
124
+ y = F.gelu(y)
125
+ y = self.convs_1x1[i](y)
126
+ y = self.norms_2[i](y)
127
+ y = F.gelu(y)
128
+ y = self.drop(y)
129
+ x = x + y
130
+ return x * x_mask
131
+
132
+
133
+ class WN(torch.nn.Module):
134
+ def __init__(
135
+ self,
136
+ hidden_channels,
137
+ kernel_size,
138
+ dilation_rate,
139
+ n_layers,
140
+ gin_channels=0,
141
+ p_dropout=0,
142
+ ):
143
+ super(WN, self).__init__()
144
+ assert kernel_size % 2 == 1
145
+ self.hidden_channels = hidden_channels
146
+ self.kernel_size = (kernel_size,)
147
+ self.dilation_rate = dilation_rate
148
+ self.n_layers = n_layers
149
+ self.gin_channels = gin_channels
150
+ self.p_dropout = p_dropout
151
+
152
+ self.in_layers = torch.nn.ModuleList()
153
+ self.res_skip_layers = torch.nn.ModuleList()
154
+ self.drop = nn.Dropout(p_dropout)
155
+
156
+ if gin_channels != 0:
157
+ cond_layer = torch.nn.Conv1d(
158
+ gin_channels, 2 * hidden_channels * n_layers, 1
159
+ )
160
+ self.cond_layer = torch.nn.utils.weight_norm(cond_layer, name="weight")
161
+
162
+ for i in range(n_layers):
163
+ dilation = dilation_rate**i
164
+ padding = int((kernel_size * dilation - dilation) / 2)
165
+ in_layer = torch.nn.Conv1d(
166
+ hidden_channels,
167
+ 2 * hidden_channels,
168
+ kernel_size,
169
+ dilation=dilation,
170
+ padding=padding,
171
+ )
172
+ in_layer = torch.nn.utils.weight_norm(in_layer, name="weight")
173
+ self.in_layers.append(in_layer)
174
+
175
+ # last one is not necessary
176
+ if i < n_layers - 1:
177
+ res_skip_channels = 2 * hidden_channels
178
+ else:
179
+ res_skip_channels = hidden_channels
180
+
181
+ res_skip_layer = torch.nn.Conv1d(hidden_channels, res_skip_channels, 1)
182
+ res_skip_layer = torch.nn.utils.weight_norm(res_skip_layer, name="weight")
183
+ self.res_skip_layers.append(res_skip_layer)
184
+
185
+ def forward(self, x, x_mask, g=None, **kwargs):
186
+ output = torch.zeros_like(x)
187
+ n_channels_tensor = torch.IntTensor([self.hidden_channels])
188
+
189
+ if g is not None:
190
+ g = self.cond_layer(g)
191
+
192
+ for i in range(self.n_layers):
193
+ x_in = self.in_layers[i](x)
194
+ if g is not None:
195
+ cond_offset = i * 2 * self.hidden_channels
196
+ g_l = g[:, cond_offset : cond_offset + 2 * self.hidden_channels, :]
197
+ else:
198
+ g_l = torch.zeros_like(x_in)
199
+
200
+ acts = commons.fused_add_tanh_sigmoid_multiply(x_in, g_l, n_channels_tensor)
201
+ acts = self.drop(acts)
202
+
203
+ res_skip_acts = self.res_skip_layers[i](acts)
204
+ if i < self.n_layers - 1:
205
+ res_acts = res_skip_acts[:, : self.hidden_channels, :]
206
+ x = (x + res_acts) * x_mask
207
+ output = output + res_skip_acts[:, self.hidden_channels :, :]
208
+ else:
209
+ output = output + res_skip_acts
210
+ return output * x_mask
211
+
212
+ def remove_weight_norm(self):
213
+ if self.gin_channels != 0:
214
+ torch.nn.utils.remove_weight_norm(self.cond_layer)
215
+ for l in self.in_layers:
216
+ torch.nn.utils.remove_weight_norm(l)
217
+ for l in self.res_skip_layers:
218
+ torch.nn.utils.remove_weight_norm(l)
219
+
220
+
221
+ class ResBlock1(torch.nn.Module):
222
+ def __init__(self, channels, kernel_size=3, dilation=(1, 3, 5)):
223
+ super(ResBlock1, self).__init__()
224
+ self.convs1 = nn.ModuleList(
225
+ [
226
+ weight_norm(
227
+ Conv1d(
228
+ channels,
229
+ channels,
230
+ kernel_size,
231
+ 1,
232
+ dilation=dilation[0],
233
+ padding=get_padding(kernel_size, dilation[0]),
234
+ )
235
+ ),
236
+ weight_norm(
237
+ Conv1d(
238
+ channels,
239
+ channels,
240
+ kernel_size,
241
+ 1,
242
+ dilation=dilation[1],
243
+ padding=get_padding(kernel_size, dilation[1]),
244
+ )
245
+ ),
246
+ weight_norm(
247
+ Conv1d(
248
+ channels,
249
+ channels,
250
+ kernel_size,
251
+ 1,
252
+ dilation=dilation[2],
253
+ padding=get_padding(kernel_size, dilation[2]),
254
+ )
255
+ ),
256
+ ]
257
+ )
258
+ self.convs1.apply(init_weights)
259
+
260
+ self.convs2 = nn.ModuleList(
261
+ [
262
+ weight_norm(
263
+ Conv1d(
264
+ channels,
265
+ channels,
266
+ kernel_size,
267
+ 1,
268
+ dilation=1,
269
+ padding=get_padding(kernel_size, 1),
270
+ )
271
+ ),
272
+ weight_norm(
273
+ Conv1d(
274
+ channels,
275
+ channels,
276
+ kernel_size,
277
+ 1,
278
+ dilation=1,
279
+ padding=get_padding(kernel_size, 1),
280
+ )
281
+ ),
282
+ weight_norm(
283
+ Conv1d(
284
+ channels,
285
+ channels,
286
+ kernel_size,
287
+ 1,
288
+ dilation=1,
289
+ padding=get_padding(kernel_size, 1),
290
+ )
291
+ ),
292
+ ]
293
+ )
294
+ self.convs2.apply(init_weights)
295
+
296
+ def forward(self, x, x_mask=None):
297
+ for c1, c2 in zip(self.convs1, self.convs2):
298
+ xt = F.leaky_relu(x, LRELU_SLOPE)
299
+ if x_mask is not None:
300
+ xt = xt * x_mask
301
+ xt = c1(xt)
302
+ xt = F.leaky_relu(xt, LRELU_SLOPE)
303
+ if x_mask is not None:
304
+ xt = xt * x_mask
305
+ xt = c2(xt)
306
+ x = xt + x
307
+ if x_mask is not None:
308
+ x = x * x_mask
309
+ return x
310
+
311
+ def remove_weight_norm(self):
312
+ for l in self.convs1:
313
+ remove_weight_norm(l)
314
+ for l in self.convs2:
315
+ remove_weight_norm(l)
316
+
317
+
318
+ class ResBlock2(torch.nn.Module):
319
+ def __init__(self, channels, kernel_size=3, dilation=(1, 3)):
320
+ super(ResBlock2, self).__init__()
321
+ self.convs = nn.ModuleList(
322
+ [
323
+ weight_norm(
324
+ Conv1d(
325
+ channels,
326
+ channels,
327
+ kernel_size,
328
+ 1,
329
+ dilation=dilation[0],
330
+ padding=get_padding(kernel_size, dilation[0]),
331
+ )
332
+ ),
333
+ weight_norm(
334
+ Conv1d(
335
+ channels,
336
+ channels,
337
+ kernel_size,
338
+ 1,
339
+ dilation=dilation[1],
340
+ padding=get_padding(kernel_size, dilation[1]),
341
+ )
342
+ ),
343
+ ]
344
+ )
345
+ self.convs.apply(init_weights)
346
+
347
+ def forward(self, x, x_mask=None):
348
+ for c in self.convs:
349
+ xt = F.leaky_relu(x, LRELU_SLOPE)
350
+ if x_mask is not None:
351
+ xt = xt * x_mask
352
+ xt = c(xt)
353
+ x = xt + x
354
+ if x_mask is not None:
355
+ x = x * x_mask
356
+ return x
357
+
358
+ def remove_weight_norm(self):
359
+ for l in self.convs:
360
+ remove_weight_norm(l)
361
+
362
+
363
+ class Log(nn.Module):
364
+ def forward(self, x, x_mask, reverse=False, **kwargs):
365
+ if not reverse:
366
+ y = torch.log(torch.clamp_min(x, 1e-5)) * x_mask
367
+ logdet = torch.sum(-y, [1, 2])
368
+ return y, logdet
369
+ else:
370
+ x = torch.exp(x) * x_mask
371
+ return x
372
+
373
+
374
+ class Flip(nn.Module):
375
+ def forward(self, x, *args, reverse=False, **kwargs):
376
+ x = torch.flip(x, [1])
377
+ if not reverse:
378
+ logdet = torch.zeros(x.size(0)).to(dtype=x.dtype, device=x.device)
379
+ return x, logdet
380
+ else:
381
+ return x
382
+
383
+
384
+ class ElementwiseAffine(nn.Module):
385
+ def __init__(self, channels):
386
+ super().__init__()
387
+ self.channels = channels
388
+ self.m = nn.Parameter(torch.zeros(channels, 1))
389
+ self.logs = nn.Parameter(torch.zeros(channels, 1))
390
+
391
+ def forward(self, x, x_mask, reverse=False, **kwargs):
392
+ if not reverse:
393
+ y = self.m + torch.exp(self.logs) * x
394
+ y = y * x_mask
395
+ logdet = torch.sum(self.logs * x_mask, [1, 2])
396
+ return y, logdet
397
+ else:
398
+ x = (x - self.m) * torch.exp(-self.logs) * x_mask
399
+ return x
400
+
401
+
402
+ class ResidualCouplingLayer(nn.Module):
403
+ def __init__(
404
+ self,
405
+ channels,
406
+ hidden_channels,
407
+ kernel_size,
408
+ dilation_rate,
409
+ n_layers,
410
+ p_dropout=0,
411
+ gin_channels=0,
412
+ mean_only=False,
413
+ ):
414
+ assert channels % 2 == 0, "channels should be divisible by 2"
415
+ super().__init__()
416
+ self.channels = channels
417
+ self.hidden_channels = hidden_channels
418
+ self.kernel_size = kernel_size
419
+ self.dilation_rate = dilation_rate
420
+ self.n_layers = n_layers
421
+ self.half_channels = channels // 2
422
+ self.mean_only = mean_only
423
+
424
+ self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1)
425
+ self.enc = WN(
426
+ hidden_channels,
427
+ kernel_size,
428
+ dilation_rate,
429
+ n_layers,
430
+ p_dropout=p_dropout,
431
+ gin_channels=gin_channels,
432
+ )
433
+ self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1)
434
+ self.post.weight.data.zero_()
435
+ self.post.bias.data.zero_()
436
+
437
+ def forward(self, x, x_mask, g=None, reverse=False):
438
+ x0, x1 = torch.split(x, [self.half_channels] * 2, 1)
439
+ h = self.pre(x0) * x_mask
440
+ h = self.enc(h, x_mask, g=g)
441
+ stats = self.post(h) * x_mask
442
+ if not self.mean_only:
443
+ m, logs = torch.split(stats, [self.half_channels] * 2, 1)
444
+ else:
445
+ m = stats
446
+ logs = torch.zeros_like(m)
447
+
448
+ if not reverse:
449
+ x1 = m + x1 * torch.exp(logs) * x_mask
450
+ x = torch.cat([x0, x1], 1)
451
+ logdet = torch.sum(logs, [1, 2])
452
+ return x, logdet
453
+ else:
454
+ x1 = (x1 - m) * torch.exp(-logs) * x_mask
455
+ x = torch.cat([x0, x1], 1)
456
+ return x
457
+
458
+
459
+ class ConvFlow(nn.Module):
460
+ def __init__(
461
+ self,
462
+ in_channels,
463
+ filter_channels,
464
+ kernel_size,
465
+ n_layers,
466
+ num_bins=10,
467
+ tail_bound=5.0,
468
+ ):
469
+ super().__init__()
470
+ self.in_channels = in_channels
471
+ self.filter_channels = filter_channels
472
+ self.kernel_size = kernel_size
473
+ self.n_layers = n_layers
474
+ self.num_bins = num_bins
475
+ self.tail_bound = tail_bound
476
+ self.half_channels = in_channels // 2
477
+
478
+ self.pre = nn.Conv1d(self.half_channels, filter_channels, 1)
479
+ self.convs = DDSConv(filter_channels, kernel_size, n_layers, p_dropout=0.0)
480
+ self.proj = nn.Conv1d(
481
+ filter_channels, self.half_channels * (num_bins * 3 - 1), 1
482
+ )
483
+ self.proj.weight.data.zero_()
484
+ self.proj.bias.data.zero_()
485
+
486
+ def forward(self, x, x_mask, g=None, reverse=False):
487
+ x0, x1 = torch.split(x, [self.half_channels] * 2, 1)
488
+ h = self.pre(x0)
489
+ h = self.convs(h, x_mask, g=g)
490
+ h = self.proj(h) * x_mask
491
+
492
+ b, c, t = x0.shape
493
+ h = h.reshape(b, c, -1, t).permute(0, 1, 3, 2) # [b, cx?, t] -> [b, c, t, ?]
494
+
495
+ unnormalized_widths = h[..., : self.num_bins] / math.sqrt(self.filter_channels)
496
+ unnormalized_heights = h[..., self.num_bins : 2 * self.num_bins] / math.sqrt(
497
+ self.filter_channels
498
+ )
499
+ unnormalized_derivatives = h[..., 2 * self.num_bins :]
500
+
501
+ x1, logabsdet = piecewise_rational_quadratic_transform(
502
+ x1,
503
+ unnormalized_widths,
504
+ unnormalized_heights,
505
+ unnormalized_derivatives,
506
+ inverse=reverse,
507
+ tails="linear",
508
+ tail_bound=self.tail_bound,
509
+ )
510
+
511
+ x = torch.cat([x0, x1], 1) * x_mask
512
+ logdet = torch.sum(logabsdet * x_mask, [1, 2])
513
+ if not reverse:
514
+ return x, logdet
515
+ else:
516
+ return x
517
+
518
+
519
+ class TransformerCouplingLayer(nn.Module):
520
+ def __init__(
521
+ self,
522
+ channels,
523
+ hidden_channels,
524
+ kernel_size,
525
+ n_layers,
526
+ n_heads,
527
+ p_dropout=0,
528
+ filter_channels=0,
529
+ mean_only=False,
530
+ wn_sharing_parameter=None,
531
+ gin_channels=0,
532
+ ):
533
+ assert channels % 2 == 0, "channels should be divisible by 2"
534
+ super().__init__()
535
+ self.channels = channels
536
+ self.hidden_channels = hidden_channels
537
+ self.kernel_size = kernel_size
538
+ self.n_layers = n_layers
539
+ self.half_channels = channels // 2
540
+ self.mean_only = mean_only
541
+
542
+ self.pre = nn.Conv1d(self.half_channels, hidden_channels, 1)
543
+ self.enc = (
544
+ Encoder(
545
+ hidden_channels,
546
+ filter_channels,
547
+ n_heads,
548
+ n_layers,
549
+ kernel_size,
550
+ p_dropout,
551
+ isflow=True,
552
+ gin_channels=gin_channels,
553
+ )
554
+ if wn_sharing_parameter is None
555
+ else wn_sharing_parameter
556
+ )
557
+ self.post = nn.Conv1d(hidden_channels, self.half_channels * (2 - mean_only), 1)
558
+ self.post.weight.data.zero_()
559
+ self.post.bias.data.zero_()
560
+
561
+ def forward(self, x, x_mask, g=None, reverse=False):
562
+ x0, x1 = torch.split(x, [self.half_channels] * 2, 1)
563
+ h = self.pre(x0) * x_mask
564
+ h = self.enc(h, x_mask, g=g)
565
+ stats = self.post(h) * x_mask
566
+ if not self.mean_only:
567
+ m, logs = torch.split(stats, [self.half_channels] * 2, 1)
568
+ else:
569
+ m = stats
570
+ logs = torch.zeros_like(m)
571
+
572
+ if not reverse:
573
+ x1 = m + x1 * torch.exp(logs) * x_mask
574
+ x = torch.cat([x0, x1], 1)
575
+ logdet = torch.sum(logs, [1, 2])
576
+ return x, logdet
577
+ else:
578
+ x1 = (x1 - m) * torch.exp(-logs) * x_mask
579
+ x = torch.cat([x0, x1], 1)
580
+ return x
581
+
582
+ x1, logabsdet = piecewise_rational_quadratic_transform(
583
+ x1,
584
+ unnormalized_widths,
585
+ unnormalized_heights,
586
+ unnormalized_derivatives,
587
+ inverse=reverse,
588
+ tails="linear",
589
+ tail_bound=self.tail_bound,
590
+ )
591
+
592
+ x = torch.cat([x0, x1], 1) * x_mask
593
+ logdet = torch.sum(logabsdet * x_mask, [1, 2])
594
+ if not reverse:
595
+ return x, logdet
596
+ else:
597
+ return x
monotonic_align/__init__.py ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from numpy import zeros, int32, float32
2
+ from torch import from_numpy
3
+
4
+ from .core import maximum_path_jit
5
+
6
+
7
+ def maximum_path(neg_cent, mask):
8
+ device = neg_cent.device
9
+ dtype = neg_cent.dtype
10
+ neg_cent = neg_cent.data.cpu().numpy().astype(float32)
11
+ path = zeros(neg_cent.shape, dtype=int32)
12
+
13
+ t_t_max = mask.sum(1)[:, 0].data.cpu().numpy().astype(int32)
14
+ t_s_max = mask.sum(2)[:, 0].data.cpu().numpy().astype(int32)
15
+ maximum_path_jit(path, neg_cent, t_t_max, t_s_max)
16
+ return from_numpy(path).to(device=device, dtype=dtype)
monotonic_align/core.py ADDED
@@ -0,0 +1,46 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import numba
2
+
3
+
4
+ @numba.jit(
5
+ numba.void(
6
+ numba.int32[:, :, ::1],
7
+ numba.float32[:, :, ::1],
8
+ numba.int32[::1],
9
+ numba.int32[::1],
10
+ ),
11
+ nopython=True,
12
+ nogil=True,
13
+ )
14
+ def maximum_path_jit(paths, values, t_ys, t_xs):
15
+ b = paths.shape[0]
16
+ max_neg_val = -1e9
17
+ for i in range(int(b)):
18
+ path = paths[i]
19
+ value = values[i]
20
+ t_y = t_ys[i]
21
+ t_x = t_xs[i]
22
+
23
+ v_prev = v_cur = 0.0
24
+ index = t_x - 1
25
+
26
+ for y in range(t_y):
27
+ for x in range(max(0, t_x + y - t_y), min(t_x, y + 1)):
28
+ if x == y:
29
+ v_cur = max_neg_val
30
+ else:
31
+ v_cur = value[y - 1, x]
32
+ if x == 0:
33
+ if y == 0:
34
+ v_prev = 0.0
35
+ else:
36
+ v_prev = max_neg_val
37
+ else:
38
+ v_prev = value[y - 1, x - 1]
39
+ value[y, x] += max(v_prev, v_cur)
40
+
41
+ for y in range(t_y - 1, -1, -1):
42
+ path[y, index] = 1
43
+ if index != 0 and (
44
+ index == y or value[y - 1, index] < value[y - 1, index - 1]
45
+ ):
46
+ index = index - 1
preprocess_text.py ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import json
2
+ from collections import defaultdict
3
+ from random import shuffle
4
+ from typing import Optional
5
+
6
+ from tqdm import tqdm
7
+ import click
8
+ from text.cleaner import clean_text
9
+
10
+
11
+ @click.command()
12
+ @click.option(
13
+ "--transcription-path",
14
+ default="filelists/hanser.list",
15
+ type=click.Path(exists=True, file_okay=True, dir_okay=False),
16
+ )
17
+ @click.option("--cleaned-path", default=None)
18
+ @click.option("--train-path", default="filelists/train.list")
19
+ @click.option("--val-path", default="filelists/val.list")
20
+ @click.option(
21
+ "--config-path",
22
+ default="configs/config.json",
23
+ type=click.Path(exists=True, file_okay=True, dir_okay=False),
24
+ )
25
+ @click.option("--val-per-spk", default=4)
26
+ @click.option("--max-val-total", default=8)
27
+ @click.option("--clean/--no-clean", default=True)
28
+ def main(
29
+ transcription_path: str,
30
+ cleaned_path: Optional[str],
31
+ train_path: str,
32
+ val_path: str,
33
+ config_path: str,
34
+ val_per_spk: int,
35
+ max_val_total: int,
36
+ clean: bool,
37
+ ):
38
+ if cleaned_path is None:
39
+ cleaned_path = transcription_path + ".cleaned"
40
+
41
+ if clean:
42
+ out_file = open(cleaned_path, "w", encoding="utf-8")
43
+ for line in tqdm(open(transcription_path, encoding="utf-8").readlines()):
44
+ try:
45
+ utt, spk, language, text = line.strip().split("|")
46
+ norm_text, phones, tones, word2ph = clean_text(text, language)
47
+ out_file.write(
48
+ "{}|{}|{}|{}|{}|{}|{}\n".format(
49
+ utt,
50
+ spk,
51
+ language,
52
+ norm_text,
53
+ " ".join(phones),
54
+ " ".join([str(i) for i in tones]),
55
+ " ".join([str(i) for i in word2ph]),
56
+ )
57
+ )
58
+ except Exception as error:
59
+ print("err!", line, error)
60
+
61
+ out_file.close()
62
+
63
+ transcription_path = cleaned_path
64
+
65
+ spk_utt_map = defaultdict(list)
66
+ spk_id_map = {}
67
+ current_sid = 0
68
+
69
+ with open(transcription_path, encoding="utf-8") as f:
70
+ for line in f.readlines():
71
+ utt, spk, language, text, phones, tones, word2ph = line.strip().split("|")
72
+ spk_utt_map[spk].append(line)
73
+
74
+ if spk not in spk_id_map.keys():
75
+ spk_id_map[spk] = current_sid
76
+ current_sid += 1
77
+
78
+ train_list = []
79
+ val_list = []
80
+
81
+ for spk, utts in spk_utt_map.items():
82
+ shuffle(utts)
83
+ val_list += utts[:val_per_spk]
84
+ train_list += utts[val_per_spk:]
85
+
86
+ if len(val_list) > max_val_total:
87
+ train_list += val_list[max_val_total:]
88
+ val_list = val_list[:max_val_total]
89
+
90
+ with open(train_path, "w", encoding="utf-8") as f:
91
+ for line in train_list:
92
+ f.write(line)
93
+
94
+ with open(val_path, "w", encoding="utf-8") as f:
95
+ for line in val_list:
96
+ f.write(line)
97
+
98
+ config = json.load(open(config_path, encoding="utf-8"))
99
+ config["data"]["spk2id"] = spk_id_map
100
+ with open(config_path, "w", encoding="utf-8") as f:
101
+ json.dump(config, f, indent=2, ensure_ascii=False)
102
+
103
+
104
+ if __name__ == "__main__":
105
+ main()
requirements.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ librosa==0.9.1
2
+ matplotlib
3
+ numpy
4
+ numba
5
+ phonemizer
6
+ scipy
7
+ tensorboard
8
+ torch
9
+ torchvision
10
+ Unidecode
11
+ amfm_decompy
12
+ jieba
13
+ transformers
14
+ pypinyin
15
+ cn2an
16
+ gradio
17
+ av
18
+ mecab-python3
19
+ loguru
20
+ unidic-lite
21
+ cmudict
22
+ fugashi
23
+ num2words
resample.py ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import argparse
3
+ import librosa
4
+ from multiprocessing import Pool, cpu_count
5
+
6
+ import soundfile
7
+ from tqdm import tqdm
8
+
9
+
10
+ def process(item):
11
+ spkdir, wav_name, args = item
12
+ speaker = spkdir.replace("\\", "/").split("/")[-1]
13
+ wav_path = os.path.join(args.in_dir, speaker, wav_name)
14
+ if os.path.exists(wav_path) and ".wav" in wav_path:
15
+ os.makedirs(os.path.join(args.out_dir, speaker), exist_ok=True)
16
+ wav, sr = librosa.load(wav_path, sr=args.sr)
17
+ soundfile.write(os.path.join(args.out_dir, speaker, wav_name), wav, sr)
18
+
19
+
20
+ if __name__ == "__main__":
21
+ parser = argparse.ArgumentParser()
22
+ parser.add_argument("--sr", type=int, default=44100, help="sampling rate")
23
+ parser.add_argument(
24
+ "--in_dir", type=str, default="./raw", help="path to source dir"
25
+ )
26
+ parser.add_argument(
27
+ "--out_dir", type=str, default="./dataset", help="path to target dir"
28
+ )
29
+ args = parser.parse_args()
30
+ # processes = 8
31
+ processes = cpu_count() - 2 if cpu_count() > 4 else 1
32
+ pool = Pool(processes=processes)
33
+
34
+ for speaker in os.listdir(args.in_dir):
35
+ spk_dir = os.path.join(args.in_dir, speaker)
36
+ if os.path.isdir(spk_dir):
37
+ print(spk_dir)
38
+ for _ in tqdm(
39
+ pool.imap_unordered(
40
+ process,
41
+ [
42
+ (spk_dir, i, args)
43
+ for i in os.listdir(spk_dir)
44
+ if i.endswith("wav")
45
+ ],
46
+ )
47
+ ):
48
+ pass
server.py ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from flask import Flask, request, Response
2
+ from io import BytesIO
3
+ import torch
4
+ from av import open as avopen
5
+
6
+ import commons
7
+ import utils
8
+ from models import SynthesizerTrn
9
+ from text.symbols import symbols
10
+ from text import cleaned_text_to_sequence, get_bert
11
+ from text.cleaner import clean_text
12
+ from scipy.io import wavfile
13
+
14
+ # Flask Init
15
+ app = Flask(__name__)
16
+ app.config["JSON_AS_ASCII"] = False
17
+
18
+
19
+ def get_text(text, language_str, hps):
20
+ norm_text, phone, tone, word2ph = clean_text(text, language_str)
21
+ phone, tone, language = cleaned_text_to_sequence(phone, tone, language_str)
22
+
23
+ if hps.data.add_blank:
24
+ phone = commons.intersperse(phone, 0)
25
+ tone = commons.intersperse(tone, 0)
26
+ language = commons.intersperse(language, 0)
27
+ for i in range(len(word2ph)):
28
+ word2ph[i] = word2ph[i] * 2
29
+ word2ph[0] += 1
30
+ bert = get_bert(norm_text, word2ph, language_str)
31
+ del word2ph
32
+ assert bert.shape[-1] == len(phone), phone
33
+
34
+ if language_str == "ZH":
35
+ bert = bert
36
+ ja_bert = torch.zeros(768, len(phone))
37
+ elif language_str == "JA":
38
+ ja_bert = bert
39
+ bert = torch.zeros(1024, len(phone))
40
+ else:
41
+ bert = torch.zeros(1024, len(phone))
42
+ ja_bert = torch.zeros(768, len(phone))
43
+ assert bert.shape[-1] == len(
44
+ phone
45
+ ), f"Bert seq len {bert.shape[-1]} != {len(phone)}"
46
+ phone = torch.LongTensor(phone)
47
+ tone = torch.LongTensor(tone)
48
+ language = torch.LongTensor(language)
49
+ return bert, ja_bert, phone, tone, language
50
+
51
+
52
+ def infer(text, sdp_ratio, noise_scale, noise_scale_w, length_scale, sid, language):
53
+ bert, ja_bert, phones, tones, lang_ids = get_text(text, language, hps)
54
+ with torch.no_grad():
55
+ x_tst = phones.to(dev).unsqueeze(0)
56
+ tones = tones.to(dev).unsqueeze(0)
57
+ lang_ids = lang_ids.to(dev).unsqueeze(0)
58
+ bert = bert.to(dev).unsqueeze(0)
59
+ ja_bert = ja_bert.to(device).unsqueeze(0)
60
+ x_tst_lengths = torch.LongTensor([phones.size(0)]).to(dev)
61
+ speakers = torch.LongTensor([hps.data.spk2id[sid]]).to(dev)
62
+ audio = (
63
+ net_g.infer(
64
+ x_tst,
65
+ x_tst_lengths,
66
+ speakers,
67
+ tones,
68
+ lang_ids,
69
+ bert,
70
+ ja_bert,
71
+ sdp_ratio=sdp_ratio,
72
+ noise_scale=noise_scale,
73
+ noise_scale_w=noise_scale_w,
74
+ length_scale=length_scale,
75
+ )[0][0, 0]
76
+ .data.cpu()
77
+ .float()
78
+ .numpy()
79
+ )
80
+ return audio
81
+
82
+
83
+ def replace_punctuation(text, i=2):
84
+ punctuation = ",。?!"
85
+ for char in punctuation:
86
+ text = text.replace(char, char * i)
87
+ return text
88
+
89
+
90
+ def wav2(i, o, format):
91
+ inp = avopen(i, "rb")
92
+ out = avopen(o, "wb", format=format)
93
+ if format == "ogg":
94
+ format = "libvorbis"
95
+
96
+ ostream = out.add_stream(format)
97
+
98
+ for frame in inp.decode(audio=0):
99
+ for p in ostream.encode(frame):
100
+ out.mux(p)
101
+
102
+ for p in ostream.encode(None):
103
+ out.mux(p)
104
+
105
+ out.close()
106
+ inp.close()
107
+
108
+
109
+ # Load Generator
110
+ hps = utils.get_hparams_from_file("./configs/config.json")
111
+
112
+ dev = "cuda"
113
+ net_g = SynthesizerTrn(
114
+ len(symbols),
115
+ hps.data.filter_length // 2 + 1,
116
+ hps.train.segment_size // hps.data.hop_length,
117
+ n_speakers=hps.data.n_speakers,
118
+ **hps.model,
119
+ ).to(dev)
120
+ _ = net_g.eval()
121
+
122
+ _ = utils.load_checkpoint("logs/G_649000.pth", net_g, None, skip_optimizer=True)
123
+
124
+
125
+ @app.route("/")
126
+ def main():
127
+ try:
128
+ speaker = request.args.get("speaker")
129
+ text = request.args.get("text").replace("/n", "")
130
+ sdp_ratio = float(request.args.get("sdp_ratio", 0.2))
131
+ noise = float(request.args.get("noise", 0.5))
132
+ noisew = float(request.args.get("noisew", 0.6))
133
+ length = float(request.args.get("length", 1.2))
134
+ language = request.args.get("language")
135
+ if length >= 2:
136
+ return "Too big length"
137
+ if len(text) >= 250:
138
+ return "Too long text"
139
+ fmt = request.args.get("format", "wav")
140
+ if None in (speaker, text):
141
+ return "Missing Parameter"
142
+ if fmt not in ("mp3", "wav", "ogg"):
143
+ return "Invalid Format"
144
+ if language not in ("JA", "ZH"):
145
+ return "Invalid language"
146
+ except:
147
+ return "Invalid Parameter"
148
+
149
+ with torch.no_grad():
150
+ audio = infer(
151
+ text,
152
+ sdp_ratio=sdp_ratio,
153
+ noise_scale=noise,
154
+ noise_scale_w=noisew,
155
+ length_scale=length,
156
+ sid=speaker,
157
+ language=language,
158
+ )
159
+
160
+ with BytesIO() as wav:
161
+ wavfile.write(wav, hps.data.sampling_rate, audio)
162
+ torch.cuda.empty_cache()
163
+ if fmt == "wav":
164
+ return Response(wav.getvalue(), mimetype="audio/wav")
165
+ wav.seek(0, 0)
166
+ with BytesIO() as ofp:
167
+ wav2(wav, ofp, fmt)
168
+ return Response(
169
+ ofp.getvalue(), mimetype="audio/mpeg" if fmt == "mp3" else "audio/ogg"
170
+ )
text/__init__.py ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from text.symbols import *
2
+
3
+ _symbol_to_id = {s: i for i, s in enumerate(symbols)}
4
+
5
+
6
+ def cleaned_text_to_sequence(cleaned_text, tones, language):
7
+ """Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
8
+ Args:
9
+ text: string to convert to a sequence
10
+ Returns:
11
+ List of integers corresponding to the symbols in the text
12
+ """
13
+ phones = [_symbol_to_id[symbol] for symbol in cleaned_text]
14
+ tone_start = language_tone_start_map[language]
15
+ tones = [i + tone_start for i in tones]
16
+ lang_id = language_id_map[language]
17
+ lang_ids = [lang_id for i in phones]
18
+ return phones, tones, lang_ids
19
+
20
+
21
+ def get_bert(norm_text, word2ph, language, device):
22
+ from .chinese_bert import get_bert_feature as zh_bert
23
+ from .english_bert_mock import get_bert_feature as en_bert
24
+ from .japanese_bert import get_bert_feature as jp_bert
25
+
26
+ lang_bert_func_map = {"ZH": zh_bert, "EN": en_bert, "JP": jp_bert}
27
+ bert = lang_bert_func_map[language](norm_text, word2ph, device)
28
+ return bert
text/chinese.py ADDED
@@ -0,0 +1,198 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import os
2
+ import re
3
+
4
+ import cn2an
5
+ from pypinyin import lazy_pinyin, Style
6
+
7
+ from text.symbols import punctuation
8
+ from text.tone_sandhi import ToneSandhi
9
+
10
+ current_file_path = os.path.dirname(__file__)
11
+ pinyin_to_symbol_map = {
12
+ line.split("\t")[0]: line.strip().split("\t")[1]
13
+ for line in open(os.path.join(current_file_path, "opencpop-strict.txt")).readlines()
14
+ }
15
+
16
+ import jieba.posseg as psg
17
+
18
+
19
+ rep_map = {
20
+ ":": ",",
21
+ ";": ",",
22
+ ",": ",",
23
+ "。": ".",
24
+ "!": "!",
25
+ "?": "?",
26
+ "\n": ".",
27
+ "·": ",",
28
+ "、": ",",
29
+ "...": "…",
30
+ "$": ".",
31
+ "“": "'",
32
+ "”": "'",
33
+ "‘": "'",
34
+ "’": "'",
35
+ "(": "'",
36
+ ")": "'",
37
+ "(": "'",
38
+ ")": "'",
39
+ "《": "'",
40
+ "》": "'",
41
+ "【": "'",
42
+ "】": "'",
43
+ "[": "'",
44
+ "]": "'",
45
+ "—": "-",
46
+ "~": "-",
47
+ "~": "-",
48
+ "「": "'",
49
+ "」": "'",
50
+ }
51
+
52
+ tone_modifier = ToneSandhi()
53
+
54
+
55
+ def replace_punctuation(text):
56
+ text = text.replace("嗯", "恩").replace("呣", "母")
57
+ pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
58
+
59
+ replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
60
+
61
+ replaced_text = re.sub(
62
+ r"[^\u4e00-\u9fa5" + "".join(punctuation) + r"]+", "", replaced_text
63
+ )
64
+
65
+ return replaced_text
66
+
67
+
68
+ def g2p(text):
69
+ pattern = r"(?<=[{0}])\s*".format("".join(punctuation))
70
+ sentences = [i for i in re.split(pattern, text) if i.strip() != ""]
71
+ phones, tones, word2ph = _g2p(sentences)
72
+ assert sum(word2ph) == len(phones)
73
+ assert len(word2ph) == len(text) # Sometimes it will crash,you can add a try-catch.
74
+ phones = ["_"] + phones + ["_"]
75
+ tones = [0] + tones + [0]
76
+ word2ph = [1] + word2ph + [1]
77
+ return phones, tones, word2ph
78
+
79
+
80
+ def _get_initials_finals(word):
81
+ initials = []
82
+ finals = []
83
+ orig_initials = lazy_pinyin(word, neutral_tone_with_five=True, style=Style.INITIALS)
84
+ orig_finals = lazy_pinyin(
85
+ word, neutral_tone_with_five=True, style=Style.FINALS_TONE3
86
+ )
87
+ for c, v in zip(orig_initials, orig_finals):
88
+ initials.append(c)
89
+ finals.append(v)
90
+ return initials, finals
91
+
92
+
93
+ def _g2p(segments):
94
+ phones_list = []
95
+ tones_list = []
96
+ word2ph = []
97
+ for seg in segments:
98
+ # Replace all English words in the sentence
99
+ seg = re.sub("[a-zA-Z]+", "", seg)
100
+ seg_cut = psg.lcut(seg)
101
+ initials = []
102
+ finals = []
103
+ seg_cut = tone_modifier.pre_merge_for_modify(seg_cut)
104
+ for word, pos in seg_cut:
105
+ if pos == "eng":
106
+ continue
107
+ sub_initials, sub_finals = _get_initials_finals(word)
108
+ sub_finals = tone_modifier.modified_tone(word, pos, sub_finals)
109
+ initials.append(sub_initials)
110
+ finals.append(sub_finals)
111
+
112
+ # assert len(sub_initials) == len(sub_finals) == len(word)
113
+ initials = sum(initials, [])
114
+ finals = sum(finals, [])
115
+ #
116
+ for c, v in zip(initials, finals):
117
+ raw_pinyin = c + v
118
+ # NOTE: post process for pypinyin outputs
119
+ # we discriminate i, ii and iii
120
+ if c == v:
121
+ assert c in punctuation
122
+ phone = [c]
123
+ tone = "0"
124
+ word2ph.append(1)
125
+ else:
126
+ v_without_tone = v[:-1]
127
+ tone = v[-1]
128
+
129
+ pinyin = c + v_without_tone
130
+ assert tone in "12345"
131
+
132
+ if c:
133
+ # 多音节
134
+ v_rep_map = {
135
+ "uei": "ui",
136
+ "iou": "iu",
137
+ "uen": "un",
138
+ }
139
+ if v_without_tone in v_rep_map.keys():
140
+ pinyin = c + v_rep_map[v_without_tone]
141
+ else:
142
+ # 单音节
143
+ pinyin_rep_map = {
144
+ "ing": "ying",
145
+ "i": "yi",
146
+ "in": "yin",
147
+ "u": "wu",
148
+ }
149
+ if pinyin in pinyin_rep_map.keys():
150
+ pinyin = pinyin_rep_map[pinyin]
151
+ else:
152
+ single_rep_map = {
153
+ "v": "yu",
154
+ "e": "e",
155
+ "i": "y",
156
+ "u": "w",
157
+ }
158
+ if pinyin[0] in single_rep_map.keys():
159
+ pinyin = single_rep_map[pinyin[0]] + pinyin[1:]
160
+
161
+ assert pinyin in pinyin_to_symbol_map.keys(), (pinyin, seg, raw_pinyin)
162
+ phone = pinyin_to_symbol_map[pinyin].split(" ")
163
+ word2ph.append(len(phone))
164
+
165
+ phones_list += phone
166
+ tones_list += [int(tone)] * len(phone)
167
+ return phones_list, tones_list, word2ph
168
+
169
+
170
+ def text_normalize(text):
171
+ numbers = re.findall(r"\d+(?:\.?\d+)?", text)
172
+ for number in numbers:
173
+ text = text.replace(number, cn2an.an2cn(number), 1)
174
+ text = replace_punctuation(text)
175
+ return text
176
+
177
+
178
+ def get_bert_feature(text, word2ph):
179
+ from text import chinese_bert
180
+
181
+ return chinese_bert.get_bert_feature(text, word2ph)
182
+
183
+
184
+ if __name__ == "__main__":
185
+ from text.chinese_bert import get_bert_feature
186
+
187
+ text = "啊!但是《原神》是由,米哈\游自主, [研发]的一款全.新开放世界.冒险游戏"
188
+ text = text_normalize(text)
189
+ print(text)
190
+ phones, tones, word2ph = g2p(text)
191
+ bert = get_bert_feature(text, word2ph)
192
+
193
+ print(phones, tones, word2ph, bert.shape)
194
+
195
+
196
+ # # 示例用法
197
+ # text = "这是一个示例文本:,你好!这是一个测试...."
198
+ # print(g2p_paddle(text)) # 输出: 这是一个示例文本你好这是一个测试
text/chinese_bert.py ADDED
@@ -0,0 +1,100 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ import sys
3
+ from transformers import AutoTokenizer, AutoModelForMaskedLM
4
+
5
+ tokenizer = AutoTokenizer.from_pretrained("./bert/chinese-roberta-wwm-ext-large")
6
+
7
+ models = dict()
8
+
9
+
10
+ def get_bert_feature(text, word2ph, device=None):
11
+ if (
12
+ sys.platform == "darwin"
13
+ and torch.backends.mps.is_available()
14
+ and device == "cpu"
15
+ ):
16
+ device = "mps"
17
+ if not device:
18
+ device = "cuda"
19
+ if device not in models.keys():
20
+ models[device] = AutoModelForMaskedLM.from_pretrained(
21
+ "./bert/chinese-roberta-wwm-ext-large"
22
+ ).to(device)
23
+ with torch.no_grad():
24
+ inputs = tokenizer(text, return_tensors="pt")
25
+ for i in inputs:
26
+ inputs[i] = inputs[i].to(device)
27
+ res = models[device](**inputs, output_hidden_states=True)
28
+ res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
29
+
30
+ assert len(word2ph) == len(text) + 2
31
+ word2phone = word2ph
32
+ phone_level_feature = []
33
+ for i in range(len(word2phone)):
34
+ repeat_feature = res[i].repeat(word2phone[i], 1)
35
+ phone_level_feature.append(repeat_feature)
36
+
37
+ phone_level_feature = torch.cat(phone_level_feature, dim=0)
38
+
39
+ return phone_level_feature.T
40
+
41
+
42
+ if __name__ == "__main__":
43
+ import torch
44
+
45
+ word_level_feature = torch.rand(38, 1024) # 12个词,每个词1024维特征
46
+ word2phone = [
47
+ 1,
48
+ 2,
49
+ 1,
50
+ 2,
51
+ 2,
52
+ 1,
53
+ 2,
54
+ 2,
55
+ 1,
56
+ 2,
57
+ 2,
58
+ 1,
59
+ 2,
60
+ 2,
61
+ 2,
62
+ 2,
63
+ 2,
64
+ 1,
65
+ 1,
66
+ 2,
67
+ 2,
68
+ 1,
69
+ 2,
70
+ 2,
71
+ 2,
72
+ 2,
73
+ 1,
74
+ 2,
75
+ 2,
76
+ 2,
77
+ 2,
78
+ 2,
79
+ 1,
80
+ 2,
81
+ 2,
82
+ 2,
83
+ 2,
84
+ 1,
85
+ ]
86
+
87
+ # 计算总帧数
88
+ total_frames = sum(word2phone)
89
+ print(word_level_feature.shape)
90
+ print(word2phone)
91
+ phone_level_feature = []
92
+ for i in range(len(word2phone)):
93
+ print(word_level_feature[i].shape)
94
+
95
+ # 对每个词重复word2phone[i]次
96
+ repeat_feature = word_level_feature[i].repeat(word2phone[i], 1)
97
+ phone_level_feature.append(repeat_feature)
98
+
99
+ phone_level_feature = torch.cat(phone_level_feature, dim=0)
100
+ print(phone_level_feature.shape) # torch.Size([36, 1024])
text/cleaner.py ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from text import chinese, japanese, cleaned_text_to_sequence
2
+
3
+
4
+ language_module_map = {"ZH": chinese, "JP": japanese}
5
+
6
+
7
+ def clean_text(text, language):
8
+ language_module = language_module_map[language]
9
+ norm_text = language_module.text_normalize(text)
10
+ phones, tones, word2ph = language_module.g2p(norm_text)
11
+ return norm_text, phones, tones, word2ph
12
+
13
+
14
+ def clean_text_bert(text, language):
15
+ language_module = language_module_map[language]
16
+ norm_text = language_module.text_normalize(text)
17
+ phones, tones, word2ph = language_module.g2p(norm_text)
18
+ bert = language_module.get_bert_feature(norm_text, word2ph)
19
+ return phones, tones, bert
20
+
21
+
22
+ def text_to_sequence(text, language):
23
+ norm_text, phones, tones, word2ph = clean_text(text, language)
24
+ return cleaned_text_to_sequence(phones, tones, language)
25
+
26
+
27
+ if __name__ == "__main__":
28
+ pass
text/cmudict.rep ADDED
The diff for this file is too large to render. See raw diff
 
text/cmudict_cache.pickle ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b9b21b20325471934ba92f2e4a5976989e7d920caa32e7a286eacb027d197949
3
+ size 6212655
text/english.py ADDED
@@ -0,0 +1,214 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import pickle
2
+ import os
3
+ import re
4
+ from g2p_en import G2p
5
+
6
+ from text import symbols
7
+
8
+ current_file_path = os.path.dirname(__file__)
9
+ CMU_DICT_PATH = os.path.join(current_file_path, "cmudict.rep")
10
+ CACHE_PATH = os.path.join(current_file_path, "cmudict_cache.pickle")
11
+ _g2p = G2p()
12
+
13
+ arpa = {
14
+ "AH0",
15
+ "S",
16
+ "AH1",
17
+ "EY2",
18
+ "AE2",
19
+ "EH0",
20
+ "OW2",
21
+ "UH0",
22
+ "NG",
23
+ "B",
24
+ "G",
25
+ "AY0",
26
+ "M",
27
+ "AA0",
28
+ "F",
29
+ "AO0",
30
+ "ER2",
31
+ "UH1",
32
+ "IY1",
33
+ "AH2",
34
+ "DH",
35
+ "IY0",
36
+ "EY1",
37
+ "IH0",
38
+ "K",
39
+ "N",
40
+ "W",
41
+ "IY2",
42
+ "T",
43
+ "AA1",
44
+ "ER1",
45
+ "EH2",
46
+ "OY0",
47
+ "UH2",
48
+ "UW1",
49
+ "Z",
50
+ "AW2",
51
+ "AW1",
52
+ "V",
53
+ "UW2",
54
+ "AA2",
55
+ "ER",
56
+ "AW0",
57
+ "UW0",
58
+ "R",
59
+ "OW1",
60
+ "EH1",
61
+ "ZH",
62
+ "AE0",
63
+ "IH2",
64
+ "IH",
65
+ "Y",
66
+ "JH",
67
+ "P",
68
+ "AY1",
69
+ "EY0",
70
+ "OY2",
71
+ "TH",
72
+ "HH",
73
+ "D",
74
+ "ER0",
75
+ "CH",
76
+ "AO1",
77
+ "AE1",
78
+ "AO2",
79
+ "OY1",
80
+ "AY2",
81
+ "IH1",
82
+ "OW0",
83
+ "L",
84
+ "SH",
85
+ }
86
+
87
+
88
+ def post_replace_ph(ph):
89
+ rep_map = {
90
+ ":": ",",
91
+ ";": ",",
92
+ ",": ",",
93
+ "。": ".",
94
+ "!": "!",
95
+ "?": "?",
96
+ "\n": ".",
97
+ "·": ",",
98
+ "、": ",",
99
+ "...": "…",
100
+ "v": "V",
101
+ }
102
+ if ph in rep_map.keys():
103
+ ph = rep_map[ph]
104
+ if ph in symbols:
105
+ return ph
106
+ if ph not in symbols:
107
+ ph = "UNK"
108
+ return ph
109
+
110
+
111
+ def read_dict():
112
+ g2p_dict = {}
113
+ start_line = 49
114
+ with open(CMU_DICT_PATH) as f:
115
+ line = f.readline()
116
+ line_index = 1
117
+ while line:
118
+ if line_index >= start_line:
119
+ line = line.strip()
120
+ word_split = line.split(" ")
121
+ word = word_split[0]
122
+
123
+ syllable_split = word_split[1].split(" - ")
124
+ g2p_dict[word] = []
125
+ for syllable in syllable_split:
126
+ phone_split = syllable.split(" ")
127
+ g2p_dict[word].append(phone_split)
128
+
129
+ line_index = line_index + 1
130
+ line = f.readline()
131
+
132
+ return g2p_dict
133
+
134
+
135
+ def cache_dict(g2p_dict, file_path):
136
+ with open(file_path, "wb") as pickle_file:
137
+ pickle.dump(g2p_dict, pickle_file)
138
+
139
+
140
+ def get_dict():
141
+ if os.path.exists(CACHE_PATH):
142
+ with open(CACHE_PATH, "rb") as pickle_file:
143
+ g2p_dict = pickle.load(pickle_file)
144
+ else:
145
+ g2p_dict = read_dict()
146
+ cache_dict(g2p_dict, CACHE_PATH)
147
+
148
+ return g2p_dict
149
+
150
+
151
+ eng_dict = get_dict()
152
+
153
+
154
+ def refine_ph(phn):
155
+ tone = 0
156
+ if re.search(r"\d$", phn):
157
+ tone = int(phn[-1]) + 1
158
+ phn = phn[:-1]
159
+ return phn.lower(), tone
160
+
161
+
162
+ def refine_syllables(syllables):
163
+ tones = []
164
+ phonemes = []
165
+ for phn_list in syllables:
166
+ for i in range(len(phn_list)):
167
+ phn = phn_list[i]
168
+ phn, tone = refine_ph(phn)
169
+ phonemes.append(phn)
170
+ tones.append(tone)
171
+ return phonemes, tones
172
+
173
+
174
+ def text_normalize(text):
175
+ # todo: eng text normalize
176
+ return text
177
+
178
+
179
+ def g2p(text):
180
+ phones = []
181
+ tones = []
182
+ words = re.split(r"([,;.\-\?\!\s+])", text)
183
+ for w in words:
184
+ if w.upper() in eng_dict:
185
+ phns, tns = refine_syllables(eng_dict[w.upper()])
186
+ phones += phns
187
+ tones += tns
188
+ else:
189
+ phone_list = list(filter(lambda p: p != " ", _g2p(w)))
190
+ for ph in phone_list:
191
+ if ph in arpa:
192
+ ph, tn = refine_ph(ph)
193
+ phones.append(ph)
194
+ tones.append(tn)
195
+ else:
196
+ phones.append(ph)
197
+ tones.append(0)
198
+ # todo: implement word2ph
199
+ word2ph = [1 for i in phones]
200
+
201
+ phones = [post_replace_ph(i) for i in phones]
202
+ return phones, tones, word2ph
203
+
204
+
205
+ if __name__ == "__main__":
206
+ # print(get_dict())
207
+ # print(eng_word_to_phoneme("hello"))
208
+ print(g2p("In this paper, we propose 1 DSPGAN, a GAN-based universal vocoder."))
209
+ # all_phones = set()
210
+ # for k, syllables in eng_dict.items():
211
+ # for group in syllables:
212
+ # for ph in group:
213
+ # all_phones.add(ph)
214
+ # print(all_phones)
text/english_bert_mock.py ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ import torch
2
+
3
+
4
+ def get_bert_feature(norm_text, word2ph):
5
+ return torch.zeros(1024, sum(word2ph))
text/japanese.py ADDED
@@ -0,0 +1,586 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Convert Japanese text to phonemes which is
2
+ # compatible with Julius https://github.com/julius-speech/segmentation-kit
3
+ import re
4
+ import unicodedata
5
+
6
+ from transformers import AutoTokenizer
7
+
8
+ from text import punctuation, symbols
9
+
10
+ try:
11
+ import MeCab
12
+ except ImportError as e:
13
+ raise ImportError("Japanese requires mecab-python3 and unidic-lite.") from e
14
+ from num2words import num2words
15
+
16
+ _CONVRULES = [
17
+ # Conversion of 2 letters
18
+ "アァ/ a a",
19
+ "イィ/ i i",
20
+ "イェ/ i e",
21
+ "イャ/ y a",
22
+ "ウゥ/ u:",
23
+ "エェ/ e e",
24
+ "オォ/ o:",
25
+ "カァ/ k a:",
26
+ "キィ/ k i:",
27
+ "クゥ/ k u:",
28
+ "クャ/ ky a",
29
+ "クュ/ ky u",
30
+ "クョ/ ky o",
31
+ "ケェ/ k e:",
32
+ "コォ/ k o:",
33
+ "ガァ/ g a:",
34
+ "ギィ/ g i:",
35
+ "グゥ/ g u:",
36
+ "グャ/ gy a",
37
+ "グュ/ gy u",
38
+ "グョ/ gy o",
39
+ "ゲェ/ g e:",
40
+ "ゴォ/ g o:",
41
+ "サァ/ s a:",
42
+ "シィ/ sh i:",
43
+ "スゥ/ s u:",
44
+ "スャ/ sh a",
45
+ "スュ/ sh u",
46
+ "スョ/ sh o",
47
+ "セェ/ s e:",
48
+ "ソォ/ s o:",
49
+ "ザァ/ z a:",
50
+ "ジィ/ j i:",
51
+ "ズゥ/ z u:",
52
+ "ズャ/ zy a",
53
+ "ズュ/ zy u",
54
+ "ズョ/ zy o",
55
+ "ゼェ/ z e:",
56
+ "ゾォ/ z o:",
57
+ "タァ/ t a:",
58
+ "チィ/ ch i:",
59
+ "ツァ/ ts a",
60
+ "ツィ/ ts i",
61
+ "ツゥ/ ts u:",
62
+ "ツャ/ ch a",
63
+ "ツュ/ ch u",
64
+ "ツョ/ ch o",
65
+ "ツェ/ ts e",
66
+ "ツォ/ ts o",
67
+ "テェ/ t e:",
68
+ "トォ/ t o:",
69
+ "ダァ/ d a:",
70
+ "ヂィ/ j i:",
71
+ "ヅゥ/ d u:",
72
+ "ヅャ/ zy a",
73
+ "ヅュ/ zy u",
74
+ "ヅョ/ zy o",
75
+ "デェ/ d e:",
76
+ "ドォ/ d o:",
77
+ "ナァ/ n a:",
78
+ "ニィ/ n i:",
79
+ "ヌゥ/ n u:",
80
+ "ヌャ/ ny a",
81
+ "ヌュ/ ny u",
82
+ "ヌョ/ ny o",
83
+ "ネェ/ n e:",
84
+ "ノォ/ n o:",
85
+ "ハァ/ h a:",
86
+ "ヒィ/ h i:",
87
+ "フゥ/ f u:",
88
+ "フャ/ hy a",
89
+ "フュ/ hy u",
90
+ "フョ/ hy o",
91
+ "ヘェ/ h e:",
92
+ "ホォ/ h o:",
93
+ "バァ/ b a:",
94
+ "ビィ/ b i:",
95
+ "ブゥ/ b u:",
96
+ "フャ/ hy a",
97
+ "ブュ/ by u",
98
+ "フョ/ hy o",
99
+ "ベェ/ b e:",
100
+ "ボォ/ b o:",
101
+ "パァ/ p a:",
102
+ "ピィ/ p i:",
103
+ "プゥ/ p u:",
104
+ "プャ/ py a",
105
+ "プュ/ py u",
106
+ "プョ/ py o",
107
+ "ペェ/ p e:",
108
+ "ポォ/ p o:",
109
+ "マァ/ m a:",
110
+ "ミィ/ m i:",
111
+ "ムゥ/ m u:",
112
+ "ムャ/ my a",
113
+ "ムュ/ my u",
114
+ "ムョ/ my o",
115
+ "メェ/ m e:",
116
+ "モォ/ m o:",
117
+ "ヤァ/ y a:",
118
+ "ユゥ/ y u:",
119
+ "ユャ/ y a:",
120
+ "ユュ/ y u:",
121
+ "ユョ/ y o:",
122
+ "ヨォ/ y o:",
123
+ "ラァ/ r a:",
124
+ "リィ/ r i:",
125
+ "ルゥ/ r u:",
126
+ "ルャ/ ry a",
127
+ "ルュ/ ry u",
128
+ "ルョ/ ry o",
129
+ "レェ/ r e:",
130
+ "ロォ/ r o:",
131
+ "ワァ/ w a:",
132
+ "ヲォ/ o:",
133
+ "ディ/ d i",
134
+ "デェ/ d e:",
135
+ "デャ/ dy a",
136
+ "デュ/ dy u",
137
+ "デョ/ dy o",
138
+ "ティ/ t i",
139
+ "テェ/ t e:",
140
+ "テャ/ ty a",
141
+ "テュ/ ty u",
142
+ "テョ/ ty o",
143
+ "スィ/ s i",
144
+ "ズァ/ z u a",
145
+ "ズィ/ z i",
146
+ "ズゥ/ z u",
147
+ "ズャ/ zy a",
148
+ "ズュ/ zy u",
149
+ "ズョ/ zy o",
150
+ "ズェ/ z e",
151
+ "ズォ/ z o",
152
+ "キャ/ ky a",
153
+ "キュ/ ky u",
154
+ "キョ/ ky o",
155
+ "シャ/ sh a",
156
+ "シュ/ sh u",
157
+ "シェ/ sh e",
158
+ "ショ/ sh o",
159
+ "チャ/ ch a",
160
+ "チュ/ ch u",
161
+ "チェ/ ch e",
162
+ "チョ/ ch o",
163
+ "トゥ/ t u",
164
+ "トャ/ ty a",
165
+ "トュ/ ty u",
166
+ "トョ/ ty o",
167
+ "ドァ/ d o a",
168
+ "ドゥ/ d u",
169
+ "ドャ/ dy a",
170
+ "ドュ/ dy u",
171
+ "ドョ/ dy o",
172
+ "ドォ/ d o:",
173
+ "ニャ/ ny a",
174
+ "ニュ/ ny u",
175
+ "ニョ/ ny o",
176
+ "ヒャ/ hy a",
177
+ "ヒュ/ hy u",
178
+ "ヒョ/ hy o",
179
+ "ミャ/ my a",
180
+ "ミュ/ my u",
181
+ "ミョ/ my o",
182
+ "リャ/ ry a",
183
+ "リュ/ ry u",
184
+ "リョ/ ry o",
185
+ "ギャ/ gy a",
186
+ "ギュ/ gy u",
187
+ "ギョ/ gy o",
188
+ "ヂェ/ j e",
189
+ "ヂャ/ j a",
190
+ "ヂュ/ j u",
191
+ "ヂョ/ j o",
192
+ "ジェ/ j e",
193
+ "ジャ/ j a",
194
+ "ジュ/ j u",
195
+ "ジョ/ j o",
196
+ "ビャ/ by a",
197
+ "ビュ/ by u",
198
+ "ビョ/ by o",
199
+ "ピャ/ py a",
200
+ "ピュ/ py u",
201
+ "ピョ/ py o",
202
+ "ウァ/ u a",
203
+ "ウィ/ w i",
204
+ "ウェ/ w e",
205
+ "ウォ/ w o",
206
+ "ファ/ f a",
207
+ "フィ/ f i",
208
+ "フゥ/ f u",
209
+ "フャ/ hy a",
210
+ "フュ/ hy u",
211
+ "フョ/ hy o",
212
+ "フェ/ f e",
213
+ "フォ/ f o",
214
+ "ヴァ/ b a",
215
+ "ヴィ/ b i",
216
+ "ヴェ/ b e",
217
+ "ヴォ/ b o",
218
+ "ヴュ/ by u",
219
+ # Conversion of 1 letter
220
+ "ア/ a",
221
+ "イ/ i",
222
+ "ウ/ u",
223
+ "エ/ e",
224
+ "オ/ o",
225
+ "カ/ k a",
226
+ "キ/ k i",
227
+ "ク/ k u",
228
+ "ケ/ k e",
229
+ "コ/ k o",
230
+ "サ/ s a",
231
+ "シ/ sh i",
232
+ "ス/ s u",
233
+ "セ/ s e",
234
+ "ソ/ s o",
235
+ "タ/ t a",
236
+ "チ/ ch i",
237
+ "ツ/ ts u",
238
+ "テ/ t e",
239
+ "ト/ t o",
240
+ "ナ/ n a",
241
+ "ニ/ n i",
242
+ "ヌ/ n u",
243
+ "ネ/ n e",
244
+ "ノ/ n o",
245
+ "ハ/ h a",
246
+ "ヒ/ h i",
247
+ "フ/ f u",
248
+ "ヘ/ h e",
249
+ "ホ/ h o",
250
+ "マ/ m a",
251
+ "ミ/ m i",
252
+ "ム/ m u",
253
+ "メ/ m e",
254
+ "モ/ m o",
255
+ "ラ/ r a",
256
+ "リ/ r i",
257
+ "ル/ r u",
258
+ "レ/ r e",
259
+ "ロ/ r o",
260
+ "ガ/ g a",
261
+ "ギ/ g i",
262
+ "グ/ g u",
263
+ "ゲ/ g e",
264
+ "ゴ/ g o",
265
+ "ザ/ z a",
266
+ "ジ/ j i",
267
+ "ズ/ z u",
268
+ "ゼ/ z e",
269
+ "ゾ/ z o",
270
+ "ダ/ d a",
271
+ "ヂ/ j i",
272
+ "ヅ/ z u",
273
+ "デ/ d e",
274
+ "ド/ d o",
275
+ "バ/ b a",
276
+ "ビ/ b i",
277
+ "ブ/ b u",
278
+ "ベ/ b e",
279
+ "ボ/ b o",
280
+ "パ/ p a",
281
+ "ピ/ p i",
282
+ "プ/ p u",
283
+ "ペ/ p e",
284
+ "ポ/ p o",
285
+ "ヤ/ y a",
286
+ "ユ/ y u",
287
+ "ヨ/ y o",
288
+ "ワ/ w a",
289
+ "ヰ/ i",
290
+ "ヱ/ e",
291
+ "ヲ/ o",
292
+ "ン/ N",
293
+ "ッ/ q",
294
+ "ヴ/ b u",
295
+ "ー/:",
296
+ # Try converting broken text
297
+ "ァ/ a",
298
+ "ィ/ i",
299
+ "ゥ/ u",
300
+ "ェ/ e",
301
+ "ォ/ o",
302
+ "ヮ/ w a",
303
+ "ォ/ o",
304
+ # Symbols
305
+ "、/ ,",
306
+ "。/ .",
307
+ "!/ !",
308
+ "?/ ?",
309
+ "・/ ,",
310
+ ]
311
+
312
+ _COLON_RX = re.compile(":+")
313
+ _REJECT_RX = re.compile("[^ a-zA-Z:,.?]")
314
+
315
+
316
+ def _makerulemap():
317
+ l = [tuple(x.split("/")) for x in _CONVRULES]
318
+ return tuple({k: v for k, v in l if len(k) == i} for i in (1, 2))
319
+
320
+
321
+ _RULEMAP1, _RULEMAP2 = _makerulemap()
322
+
323
+
324
+ def kata2phoneme(text: str) -> str:
325
+ """Convert katakana text to phonemes."""
326
+ text = text.strip()
327
+ res = []
328
+ while text:
329
+ if len(text) >= 2:
330
+ x = _RULEMAP2.get(text[:2])
331
+ if x is not None:
332
+ text = text[2:]
333
+ res += x.split(" ")[1:]
334
+ continue
335
+ x = _RULEMAP1.get(text[0])
336
+ if x is not None:
337
+ text = text[1:]
338
+ res += x.split(" ")[1:]
339
+ continue
340
+ res.append(text[0])
341
+ text = text[1:]
342
+ # res = _COLON_RX.sub(":", res)
343
+ return res
344
+
345
+
346
+ _KATAKANA = "".join(chr(ch) for ch in range(ord("ァ"), ord("ン") + 1))
347
+ _HIRAGANA = "".join(chr(ch) for ch in range(ord("ぁ"), ord("ん") + 1))
348
+ _HIRA2KATATRANS = str.maketrans(_HIRAGANA, _KATAKANA)
349
+
350
+
351
+ def hira2kata(text: str) -> str:
352
+ text = text.translate(_HIRA2KATATRANS)
353
+ return text.replace("う゛", "ヴ")
354
+
355
+
356
+ _SYMBOL_TOKENS = set(list("・、。?!"))
357
+ _NO_YOMI_TOKENS = set(list("「」『』―()[][]"))
358
+ _TAGGER = MeCab.Tagger()
359
+
360
+
361
+ def text2kata(text: str) -> str:
362
+ parsed = _TAGGER.parse(text)
363
+ res = []
364
+ for line in parsed.split("\n"):
365
+ if line == "EOS":
366
+ break
367
+ parts = line.split("\t")
368
+
369
+ word, yomi = parts[0], parts[1]
370
+ if yomi:
371
+ res.append(yomi)
372
+ else:
373
+ if word in _SYMBOL_TOKENS:
374
+ res.append(word)
375
+ elif word in ("っ", "ッ"):
376
+ res.append("ッ")
377
+ elif word in _NO_YOMI_TOKENS:
378
+ pass
379
+ else:
380
+ res.append(word)
381
+ return hira2kata("".join(res))
382
+
383
+
384
+ _ALPHASYMBOL_YOMI = {
385
+ "#": "シャープ",
386
+ "%": "パーセント",
387
+ "&": "アンド",
388
+ "+": "プラス",
389
+ "-": "マイナス",
390
+ ":": "コロン",
391
+ ";": "セミコロン",
392
+ "<": "小なり",
393
+ "=": "イコール",
394
+ ">": "大なり",
395
+ "@": "アット",
396
+ "a": "エー",
397
+ "b": "ビー",
398
+ "c": "シー",
399
+ "d": "ディー",
400
+ "e": "イー",
401
+ "f": "エフ",
402
+ "g": "ジー",
403
+ "h": "エイチ",
404
+ "i": "アイ",
405
+ "j": "ジェー",
406
+ "k": "ケー",
407
+ "l": "エル",
408
+ "m": "エム",
409
+ "n": "エヌ",
410
+ "o": "オー",
411
+ "p": "ピー",
412
+ "q": "キュー",
413
+ "r": "アール",
414
+ "s": "エス",
415
+ "t": "ティー",
416
+ "u": "ユー",
417
+ "v": "ブイ",
418
+ "w": "ダブリュー",
419
+ "x": "エックス",
420
+ "y": "ワイ",
421
+ "z": "ゼット",
422
+ "α": "アルファ",
423
+ "β": "ベータ",
424
+ "γ": "ガンマ",
425
+ "δ": "デルタ",
426
+ "ε": "イプシロン",
427
+ "ζ": "ゼータ",
428
+ "η": "イータ",
429
+ "θ": "シータ",
430
+ "ι": "イオタ",
431
+ "κ": "カッパ",
432
+ "λ": "ラムダ",
433
+ "μ": "ミュー",
434
+ "ν": "ニュー",
435
+ "ξ": "クサイ",
436
+ "ο": "オミクロン",
437
+ "π": "パイ",
438
+ "ρ": "ロー",
439
+ "σ": "シグマ",
440
+ "τ": "タウ",
441
+ "υ": "ウプシロン",
442
+ "φ": "ファイ",
443
+ "χ": "カイ",
444
+ "ψ": "プサイ",
445
+ "ω": "オメガ",
446
+ }
447
+
448
+
449
+ _NUMBER_WITH_SEPARATOR_RX = re.compile("[0-9]{1,3}(,[0-9]{3})+")
450
+ _CURRENCY_MAP = {"$": "ドル", "¥": "円", "£": "ポンド", "€": "ユーロ"}
451
+ _CURRENCY_RX = re.compile(r"([$¥£€])([0-9.]*[0-9])")
452
+ _NUMBER_RX = re.compile(r"[0-9]+(\.[0-9]+)?")
453
+
454
+
455
+ def japanese_convert_numbers_to_words(text: str) -> str:
456
+ res = _NUMBER_WITH_SEPARATOR_RX.sub(lambda m: m[0].replace(",", ""), text)
457
+ res = _CURRENCY_RX.sub(lambda m: m[2] + _CURRENCY_MAP.get(m[1], m[1]), res)
458
+ res = _NUMBER_RX.sub(lambda m: num2words(m[0], lang="ja"), res)
459
+ return res
460
+
461
+
462
+ def japanese_convert_alpha_symbols_to_words(text: str) -> str:
463
+ return "".join([_ALPHASYMBOL_YOMI.get(ch, ch) for ch in text.lower()])
464
+
465
+
466
+ def japanese_text_to_phonemes(text: str) -> str:
467
+ """Convert Japanese text to phonemes."""
468
+ res = unicodedata.normalize("NFKC", text)
469
+ res = japanese_convert_numbers_to_words(res)
470
+ # res = japanese_convert_alpha_symbols_to_words(res)
471
+ res = text2kata(res)
472
+ res = kata2phoneme(res)
473
+ return res
474
+
475
+
476
+ def is_japanese_character(char):
477
+ # 定义日语文字系统的 Unicode 范围
478
+ japanese_ranges = [
479
+ (0x3040, 0x309F), # 平假名
480
+ (0x30A0, 0x30FF), # 片假名
481
+ (0x4E00, 0x9FFF), # 汉字 (CJK Unified Ideographs)
482
+ (0x3400, 0x4DBF), # 汉字扩展 A
483
+ (0x20000, 0x2A6DF), # 汉字扩展 B
484
+ # 可以根据需要添加其他汉字扩展范围
485
+ ]
486
+
487
+ # 将字符的 Unicode 编码转换为整数
488
+ char_code = ord(char)
489
+
490
+ # 检查字符是否在任何一个日语范围内
491
+ for start, end in japanese_ranges:
492
+ if start <= char_code <= end:
493
+ return True
494
+
495
+ return False
496
+
497
+
498
+ rep_map = {
499
+ ":": ",",
500
+ ";": ",",
501
+ ",": ",",
502
+ "。": ".",
503
+ "!": "!",
504
+ "?": "?",
505
+ "\n": ".",
506
+ "·": ",",
507
+ "、": ",",
508
+ "...": "…",
509
+ }
510
+
511
+
512
+ def replace_punctuation(text):
513
+ pattern = re.compile("|".join(re.escape(p) for p in rep_map.keys()))
514
+
515
+ replaced_text = pattern.sub(lambda x: rep_map[x.group()], text)
516
+
517
+ replaced_text = re.sub(
518
+ r"[^\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF\u3400-\u4DBF"
519
+ + "".join(punctuation)
520
+ + r"]+",
521
+ "",
522
+ replaced_text,
523
+ )
524
+
525
+ return replaced_text
526
+
527
+
528
+ def text_normalize(text):
529
+ res = unicodedata.normalize("NFKC", text)
530
+ res = japanese_convert_numbers_to_words(res)
531
+ # res = "".join([i for i in res if is_japanese_character(i)])
532
+ res = replace_punctuation(res)
533
+ return res
534
+
535
+
536
+ def distribute_phone(n_phone, n_word):
537
+ phones_per_word = [0] * n_word
538
+ for task in range(n_phone):
539
+ min_tasks = min(phones_per_word)
540
+ min_index = phones_per_word.index(min_tasks)
541
+ phones_per_word[min_index] += 1
542
+ return phones_per_word
543
+
544
+
545
+ tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
546
+
547
+
548
+ def g2p(norm_text):
549
+ tokenized = tokenizer.tokenize(norm_text)
550
+ phs = []
551
+ ph_groups = []
552
+ for t in tokenized:
553
+ if not t.startswith("#"):
554
+ ph_groups.append([t])
555
+ else:
556
+ ph_groups[-1].append(t.replace("#", ""))
557
+ word2ph = []
558
+ for group in ph_groups:
559
+ phonemes = kata2phoneme(text2kata("".join(group)))
560
+ # phonemes = [i for i in phonemes if i in symbols]
561
+ for i in phonemes:
562
+ assert i in symbols, (group, norm_text, tokenized)
563
+ phone_len = len(phonemes)
564
+ word_len = len(group)
565
+
566
+ aaa = distribute_phone(phone_len, word_len)
567
+ word2ph += aaa
568
+
569
+ phs += phonemes
570
+ phones = ["_"] + phs + ["_"]
571
+ tones = [0 for i in phones]
572
+ word2ph = [1] + word2ph + [1]
573
+ return phones, tones, word2ph
574
+
575
+
576
+ if __name__ == "__main__":
577
+ tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
578
+ text = "hello,こんにちは、世界!……"
579
+ from text.japanese_bert import get_bert_feature
580
+
581
+ text = text_normalize(text)
582
+ print(text)
583
+ phones, tones, word2ph = g2p(text)
584
+ bert = get_bert_feature(text, word2ph)
585
+
586
+ print(phones, tones, word2ph, bert.shape)
text/japanese_bert.py ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import torch
2
+ from transformers import AutoTokenizer, AutoModelForMaskedLM
3
+ import sys
4
+
5
+ tokenizer = AutoTokenizer.from_pretrained("./bert/bert-base-japanese-v3")
6
+
7
+ models = dict()
8
+
9
+
10
+ def get_bert_feature(text, word2ph, device=None):
11
+ if (
12
+ sys.platform == "darwin"
13
+ and torch.backends.mps.is_available()
14
+ and device == "cpu"
15
+ ):
16
+ device = "mps"
17
+ if not device:
18
+ device = "cuda"
19
+ if device not in models.keys():
20
+ models[device] = AutoModelForMaskedLM.from_pretrained(
21
+ "./bert/bert-base-japanese-v3"
22
+ ).to(device)
23
+ with torch.no_grad():
24
+ inputs = tokenizer(text, return_tensors="pt")
25
+ for i in inputs:
26
+ inputs[i] = inputs[i].to(device)
27
+ res = models[device](**inputs, output_hidden_states=True)
28
+ res = torch.cat(res["hidden_states"][-3:-2], -1)[0].cpu()
29
+ assert inputs["input_ids"].shape[-1] == len(word2ph)
30
+ word2phone = word2ph
31
+ phone_level_feature = []
32
+ for i in range(len(word2phone)):
33
+ repeat_feature = res[i].repeat(word2phone[i], 1)
34
+ phone_level_feature.append(repeat_feature)
35
+
36
+ phone_level_feature = torch.cat(phone_level_feature, dim=0)
37
+
38
+ return phone_level_feature.T
text/opencpop-strict.txt ADDED
@@ -0,0 +1,429 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ a AA a
2
+ ai AA ai
3
+ an AA an
4
+ ang AA ang
5
+ ao AA ao
6
+ ba b a
7
+ bai b ai
8
+ ban b an
9
+ bang b ang
10
+ bao b ao
11
+ bei b ei
12
+ ben b en
13
+ beng b eng
14
+ bi b i
15
+ bian b ian
16
+ biao b iao
17
+ bie b ie
18
+ bin b in
19
+ bing b ing
20
+ bo b o
21
+ bu b u
22
+ ca c a
23
+ cai c ai
24
+ can c an
25
+ cang c ang
26
+ cao c ao
27
+ ce c e
28
+ cei c ei
29
+ cen c en
30
+ ceng c eng
31
+ cha ch a
32
+ chai ch ai
33
+ chan ch an
34
+ chang ch ang
35
+ chao ch ao
36
+ che ch e
37
+ chen ch en
38
+ cheng ch eng
39
+ chi ch ir
40
+ chong ch ong
41
+ chou ch ou
42
+ chu ch u
43
+ chua ch ua
44
+ chuai ch uai
45
+ chuan ch uan
46
+ chuang ch uang
47
+ chui ch ui
48
+ chun ch un
49
+ chuo ch uo
50
+ ci c i0
51
+ cong c ong
52
+ cou c ou
53
+ cu c u
54
+ cuan c uan
55
+ cui c ui
56
+ cun c un
57
+ cuo c uo
58
+ da d a
59
+ dai d ai
60
+ dan d an
61
+ dang d ang
62
+ dao d ao
63
+ de d e
64
+ dei d ei
65
+ den d en
66
+ deng d eng
67
+ di d i
68
+ dia d ia
69
+ dian d ian
70
+ diao d iao
71
+ die d ie
72
+ ding d ing
73
+ diu d iu
74
+ dong d ong
75
+ dou d ou
76
+ du d u
77
+ duan d uan
78
+ dui d ui
79
+ dun d un
80
+ duo d uo
81
+ e EE e
82
+ ei EE ei
83
+ en EE en
84
+ eng EE eng
85
+ er EE er
86
+ fa f a
87
+ fan f an
88
+ fang f ang
89
+ fei f ei
90
+ fen f en
91
+ feng f eng
92
+ fo f o
93
+ fou f ou
94
+ fu f u
95
+ ga g a
96
+ gai g ai
97
+ gan g an
98
+ gang g ang
99
+ gao g ao
100
+ ge g e
101
+ gei g ei
102
+ gen g en
103
+ geng g eng
104
+ gong g ong
105
+ gou g ou
106
+ gu g u
107
+ gua g ua
108
+ guai g uai
109
+ guan g uan
110
+ guang g uang
111
+ gui g ui
112
+ gun g un
113
+ guo g uo
114
+ ha h a
115
+ hai h ai
116
+ han h an
117
+ hang h ang
118
+ hao h ao
119
+ he h e
120
+ hei h ei
121
+ hen h en
122
+ heng h eng
123
+ hong h ong
124
+ hou h ou
125
+ hu h u
126
+ hua h ua
127
+ huai h uai
128
+ huan h uan
129
+ huang h uang
130
+ hui h ui
131
+ hun h un
132
+ huo h uo
133
+ ji j i
134
+ jia j ia
135
+ jian j ian
136
+ jiang j iang
137
+ jiao j iao
138
+ jie j ie
139
+ jin j in
140
+ jing j ing
141
+ jiong j iong
142
+ jiu j iu
143
+ ju j v
144
+ jv j v
145
+ juan j van
146
+ jvan j van
147
+ jue j ve
148
+ jve j ve
149
+ jun j vn
150
+ jvn j vn
151
+ ka k a
152
+ kai k ai
153
+ kan k an
154
+ kang k ang
155
+ kao k ao
156
+ ke k e
157
+ kei k ei
158
+ ken k en
159
+ keng k eng
160
+ kong k ong
161
+ kou k ou
162
+ ku k u
163
+ kua k ua
164
+ kuai k uai
165
+ kuan k uan
166
+ kuang k uang
167
+ kui k ui
168
+ kun k un
169
+ kuo k uo
170
+ la l a
171
+ lai l ai
172
+ lan l an
173
+ lang l ang
174
+ lao l ao
175
+ le l e
176
+ lei l ei
177
+ leng l eng
178
+ li l i
179
+ lia l ia
180
+ lian l ian
181
+ liang l iang
182
+ liao l iao
183
+ lie l ie
184
+ lin l in
185
+ ling l ing
186
+ liu l iu
187
+ lo l o
188
+ long l ong
189
+ lou l ou
190
+ lu l u
191
+ luan l uan
192
+ lun l un
193
+ luo l uo
194
+ lv l v
195
+ lve l ve
196
+ ma m a
197
+ mai m ai
198
+ man m an
199
+ mang m ang
200
+ mao m ao
201
+ me m e
202
+ mei m ei
203
+ men m en
204
+ meng m eng
205
+ mi m i
206
+ mian m ian
207
+ miao m iao
208
+ mie m ie
209
+ min m in
210
+ ming m ing
211
+ miu m iu
212
+ mo m o
213
+ mou m ou
214
+ mu m u
215
+ na n a
216
+ nai n ai
217
+ nan n an
218
+ nang n ang
219
+ nao n ao
220
+ ne n e
221
+ nei n ei
222
+ nen n en
223
+ neng n eng
224
+ ni n i
225
+ nian n ian
226
+ niang n iang
227
+ niao n iao
228
+ nie n ie
229
+ nin n in
230
+ ning n ing
231
+ niu n iu
232
+ nong n ong
233
+ nou n ou
234
+ nu n u
235
+ nuan n uan
236
+ nun n un
237
+ nuo n uo
238
+ nv n v
239
+ nve n ve
240
+ o OO o
241
+ ou OO ou
242
+ pa p a
243
+ pai p ai
244
+ pan p an
245
+ pang p ang
246
+ pao p ao
247
+ pei p ei
248
+ pen p en
249
+ peng p eng
250
+ pi p i
251
+ pian p ian
252
+ piao p iao
253
+ pie p ie
254
+ pin p in
255
+ ping p ing
256
+ po p o
257
+ pou p ou
258
+ pu p u
259
+ qi q i
260
+ qia q ia
261
+ qian q ian
262
+ qiang q iang
263
+ qiao q iao
264
+ qie q ie
265
+ qin q in
266
+ qing q ing
267
+ qiong q iong
268
+ qiu q iu
269
+ qu q v
270
+ qv q v
271
+ quan q van
272
+ qvan q van
273
+ que q ve
274
+ qve q ve
275
+ qun q vn
276
+ qvn q vn
277
+ ran r an
278
+ rang r ang
279
+ rao r ao
280
+ re r e
281
+ ren r en
282
+ reng r eng
283
+ ri r ir
284
+ rong r ong
285
+ rou r ou
286
+ ru r u
287
+ rua r ua
288
+ ruan r uan
289
+ rui r ui
290
+ run r un
291
+ ruo r uo
292
+ sa s a
293
+ sai s ai
294
+ san s an
295
+ sang s ang
296
+ sao s ao
297
+ se s e
298
+ sen s en
299
+ seng s eng
300
+ sha sh a
301
+ shai sh ai
302
+ shan sh an
303
+ shang sh ang
304
+ shao sh ao
305
+ she sh e
306
+ shei sh ei
307
+ shen sh en
308
+ sheng sh eng
309
+ shi sh ir
310
+ shou sh ou
311
+ shu sh u
312
+ shua sh ua
313
+ shuai sh uai
314
+ shuan sh uan
315
+ shuang sh uang
316
+ shui sh ui
317
+ shun sh un
318
+ shuo sh uo
319
+ si s i0
320
+ song s ong
321
+ sou s ou
322
+ su s u
323
+ suan s uan
324
+ sui s ui
325
+ sun s un
326
+ suo s uo
327
+ ta t a
328
+ tai t ai
329
+ tan t an
330
+ tang t ang
331
+ tao t ao
332
+ te t e
333
+ tei t ei
334
+ teng t eng
335
+ ti t i
336
+ tian t ian
337
+ tiao t iao
338
+ tie t ie
339
+ ting t ing
340
+ tong t ong
341
+ tou t ou
342
+ tu t u
343
+ tuan t uan
344
+ tui t ui
345
+ tun t un
346
+ tuo t uo
347
+ wa w a
348
+ wai w ai
349
+ wan w an
350
+ wang w ang
351
+ wei w ei
352
+ wen w en
353
+ weng w eng
354
+ wo w o
355
+ wu w u
356
+ xi x i
357
+ xia x ia
358
+ xian x ian
359
+ xiang x iang
360
+ xiao x iao
361
+ xie x ie
362
+ xin x in
363
+ xing x ing
364
+ xiong x iong
365
+ xiu x iu
366
+ xu x v
367
+ xv x v
368
+ xuan x van
369
+ xvan x van
370
+ xue x ve
371
+ xve x ve
372
+ xun x vn
373
+ xvn x vn
374
+ ya y a
375
+ yan y En
376
+ yang y ang
377
+ yao y ao
378
+ ye y E
379
+ yi y i
380
+ yin y in
381
+ ying y ing
382
+ yo y o
383
+ yong y ong
384
+ you y ou
385
+ yu y v
386
+ yv y v
387
+ yuan y van
388
+ yvan y van
389
+ yue y ve
390
+ yve y ve
391
+ yun y vn
392
+ yvn y vn
393
+ za z a
394
+ zai z ai
395
+ zan z an
396
+ zang z ang
397
+ zao z ao
398
+ ze z e
399
+ zei z ei
400
+ zen z en
401
+ zeng z eng
402
+ zha zh a
403
+ zhai zh ai
404
+ zhan zh an
405
+ zhang zh ang
406
+ zhao zh ao
407
+ zhe zh e
408
+ zhei zh ei
409
+ zhen zh en
410
+ zheng zh eng
411
+ zhi zh ir
412
+ zhong zh ong
413
+ zhou zh ou
414
+ zhu zh u
415
+ zhua zh ua
416
+ zhuai zh uai
417
+ zhuan zh uan
418
+ zhuang zh uang
419
+ zhui zh ui
420
+ zhun zh un
421
+ zhuo zh uo
422
+ zi z i0
423
+ zong z ong
424
+ zou z ou
425
+ zu z u
426
+ zuan z uan
427
+ zui z ui
428
+ zun z un
429
+ zuo z uo