Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
Anonymous Author(s)
AffiliationAddress email
Abstract
Polyphone disambiguation aims to capture accurate pronunciation knowledgefrom natural text sequences for reliable Text-to-speech (TTS) systems.
However,previous approaches require substantial annotated training data and additionalefforts from language experts, making it difficult to extend high-quality neural TTSsystems to out-of-domain daily conversations and countless languages worldwide.
This paper tackles the polyphone disambiguation problem from a concise and novelperspective: we propose Dict-TTS, a semantic-aware generative text-to-speechmodel with an online website dictionary (the existing prior information in thenatural language).
Specifically, we design a semantics-to-pronunciation attention(S2PA) module to match the semantic patterns between the input text sequence andthe prior semantics in the dictionary and obtain the corresponding pronunciations;
The S2PA module can be easily trained with the end-to-end TTS model withoutany annotated phoneme labels.
Experimental results in three languages show thatour model outperforms several strong baseline models in terms of pronunciationaccuracy and improves the prosody modeling of TTS systems.
Further extensiveanalyses with different linguistic encoders demonstrate that each design in DictTTS is effective.
Audio samples are available at https://dicttts.github.io/
DictTTS-Demo/ .
Introduction
Capturing the pronunciations from raw texts is challenging for end-to-end text-to-speech (TTS)systems [2, 28, 38, 41, 51, 24, 14, 37], since there are full of words that are not covered by generalpronunciation rules [4, 21, 46].
Therefore, polyphone 1 disambiguation (one of the biggest challengesin converting texts into phonemes [33, 56, 42]) plays an important role in the construction of highquality neural TTS systems [18, 34].
However, since the exact pronunciation of a polyphone must beinferred based on its semantic contexts, current solutions still face several challenges: 1) the rulebased approaches [60, 19] with limited linguistic knowledge or the neural models [56, 34, 42] trainedon limited data suffer from significant performance degradation on out-of-domain text datasets; 2)the neural network-based models [6, 5, 34] learn the grapheme-to-phoneme (G2P) mapping in anend-to-end manner without explicit semantics modeling, which hinders their pronunciation accuracyin real-life applications. 3) based on the above two points, a reliable polyphone disambiguationmodule is usually based on a combination of hand-crafted rules, structured G2P-oriented lexicons,and neural models [16], which requires substantial phonemes labels and external knowledge fromlanguage experts.
Unlike the previous rule-based or neural network-based approaches, we address the above challengeswith the existing prior information worldwide.
As shown in Figure 1, an arbitrary dictionary
物の端の方。物事の終わりの方。 末： < マツ > < バツ >
Pronunciation
Definitions すえ。終わり。主要でない。
Usages 末端 ・ 末尾 ・ 末葉 / 巻末 ・ 期末 ・ 結末 ・ 月末 ・ 毫末 ( ごうまつ )
Character 末子 ・ 末孫 ・ 末弟 ・ 末席 …… used in daily life can be viewed as a prior knowledge database.
Intuitively, it contains valuable prior knowledge pronunciations in conversations.
When one is confused about the acousticpronunciation of a specific polyphone, he or she will resort to the dictionary website to infer itsexact reading the
We imitate this scenario in our architecture designand propose Dict
TTS an unsupervised polyphone disambiguation framework, which explicitly the online to identify the and
• To explicitly learn the semantics-to-pronunciation mapping, we adopt a semantic encoder
to obtain the semantic contexts of the input text and utilize a semantics-to-pronunciation
attention (S2PA) module to search the matched semantic patterns in the dictionary so as to
find the correct pronunciations.
We also use the retrieved semantic information as auxiliary
information for prosody modeling of the TTS model.
• To perform polyphone disambiguation without phoneme labels, we combine our S2PA
module into end-to-end TTS systems’ training and inference processes.
Different from
current neural polyphone disambiguation models, our module can be trained with the
guidance of mel-spectrogram reconstruction loss in a fully end-to-end manner, which
significantly reduces the cost of building such a system.
To demonstrate the generalization ability of our Dict-TTS, we perform experiments on three datasets,including a standard Mandarin dataset [3], a Japanese corpus [43], and a Cantonese dataset [1].
Experiments on these datasets show that Dict-TTS outperforms other state-of-the-art polyphonedisambiguation models in pronunciation accuracy and improves the prosody modeling of TTS systemsin terms of both subjective and objective evaluation metrics.
The pronunciation accuracy of Dict-TTSis further improved by being pre-trained on a large-scaled automatic speech recognition (ASR) dataset.
The main contributions of this work are summarized as follows:
• We incorporate the online dictionary into TTS systems and propose a semantic-aware methodfor polyphone disambiguation, which improves the pronunciation accuracy and robustness ofend-to-end TTS systems.
Moreover, the idea of introducing the prior knowledge worldwidecan also inspire other tasks like neural language modeling [23] and sequence labeling
[30].• We propose a novel and general framework for unsupervised polyphone disambiguation inTTS systems, which further enables the efficient pre-training on large-scaled ASR datasetsand improves the generalization capacity significantly.•
We also find that the retrieved semantics in the dictionary knowledge can be used asauxiliary information to improve prosody modeling and help the TTS system to generatemore expressive speech.• We further analyze the characteristics of the linguistic encoder based on phoneme andcharacter and provide valuable interpretations about our semantics-to-pronunciation attentionmodule.
Background
This section describes the background of TTS, grapheme-to-phoneme (G2P) pipeline, and theirrelations with polyphone disambiguation.
We also review the existing works that aim at semanticaware polyphone disambiguation and analyze their advantages and disadvantages.
Text-to-Speech Text-to-speech (TTS) models [51, 2, 27, 28, 38, 24, 37] first generate melspectrogram from text and then synthesize speech waveform from the generated mel-spectrogramusing a separately pre-trained vocoder [48, 26], or directly generate waveform from text in an end-toend manner [36, 13, 25].
The frontend model of end-to-end TTS system should tackle one importanttask, i.e., polyphone disambiguation [61].
Properly mapping the grapheme sequence into phonemesequence requires the linguistic encoder to capture the empirical pronunciation rules in daily conversations.
However, it is extremely difficult for the linguistic encoder to learn all the pronunciation rulesnecessary to produce speech in an end-to-end manner, which results in inevitable mispronunciationsin the generated speech.
To alleviate this problem, robust TTS models usually convert the textsequence into the phoneme sequence with open-source grapheme-to-phoneme pipelines and predictthe mel-spectrogram from the phoneme sequence.
However, the rule-based or neural network-basedgrapheme-to-phoneme pipelines suffer from significant performance degradation on out-of-domaindatasets since it is extremely difficult and costly for them to cover all linguistic knowledge.
Grapheme-to-Phoneme The grapheme-to-phoneme models map grapheme sequence to phonemesequence to reduce pronunciation errors in modern TTS systems.
For logographic languages like
Chinese, Japanese and Korean, although the lexicon can cover nearly all the characters, there are fullof polyphones that can only be decided according to the semantic context of a character [46].
Thus,polyphone disambiguation is the most important challenge in grapheme-to-phoneme conversions forthis kind of languages.
Moreover, many alphabetic languages including English and French also havepolyphones in daily conversations.
Current polyphone disambiguation approaches can be categorizedinto the rule-based approach [60, 19] and the data-driven approach [39, 6, 5, 34].
The rule-based
G2P method is based on a combination of hand-crafted rules and structured G2P-oriented lexicons,which requires a substantial amount of linguistic knowledge.
The data-driven G2P algorithm adoptsstatistical methods [29] or neural encoder-decoder architecture [54, 5, 44, 56, 34, 42, 50].
However,building a data-driven G2P model requires a large amount of carefully labeled data and substantiallinguistic knowledge from language experts, which is extremely costly and laborious.
Semantic-Aware Polyphone Disambiguation
Polyphone disambiguation is the core issue for G2Pconversion in various languages.
The pronunciation of a polyphone is defined by the semantic contextinformation of neighbouring characters [46].
In order to comprehend the semantic meaning in thegiven sentence for polyphone disambiguation, previous methods [11, 53, 45, 18, 8] have adoptedthe pre-trained language model [12] to extract semantic features from raw character sequences andpredict the pronunciation of polyphones with neural classifiers according to the semantic features.
Among them, PnG BERT and Mixed-Phoneme BERT [22, 58] take both phoneme and grapheme asinput to train an augmented BERT and use the pre-trained augmented BERT as the TTS encoder.
However, these methods still require annotated data to train and can not be incorporated into the TTStraining in an end-to-end manner.
Although NLR [16] directly injects BERT-derived knowledge intothe TTS systems without phoneme labels and successfully reduces pronunciation errors, their methodconfounds the acoustic and semantic space, which significantly affects the pronunciation accuracy.
Method
To exploit the prior linguistic knowledge in the online dictionary for TTS systems, we propose
Dict-TTS, which explicitly captures the semantic relevance between the input sentences and thedictionary entries for polyphone disambiguation.
In this section, we firstly introduce the overallarchitecture design of Dict-TTS based on PortaSpeech [37].
Then after the comparison between thephoneme-based and character-based TTS systems in the acoustic and semantic space, we design anovel semantics-to-pronunciation attention (S2PA) module to learn grapheme-to-phoneme mappingsbased on the semantic context; In general, Dict-TTS exploits the prior pronunciation knowledge in theonline dictionary with the following steps: Firstly, the character sequence is fed into the self-attentionbased semantic encoder to obtain the semantic representations of the input character sequence, andwe utilize a pre-trained cross-lingual language model [10] to extract the semantic context informationin the dictionary entries;
Secondly, we calculate the most relevant dictionary entries of the inputgraphemes and obtain the corresponding pronunciation sequence contained in the dictionary entries;
Finally, the extracted semantic context information and the pronunciations are fed into the linguisticencoder for feature fusion.
We describe these designs in detail in the following subsections.
Input Character Sequence
Semantic Encoder
Attention 0.
Semantic Embeddings
PronunciationEmbeddings
Linguistic Encoder
Variational Generator
ML-Discriminator
External Vocoder
Dictionary Entries
Pronunciations
Semantics-to-Pronunciation Attention 听乐队 ...
（ listen to the band ... ）
快乐 音乐 欣赏 地名 乐观 乐 0.
取乐 ...
乐曲 声乐 ...
文言 文 ...
用字 乐陵 ...
l e y ue y ao
l ao
Weights x ...
Semantic Space
Acoustic Space
SemanticContexts
3.1 Model Overview
The overall model architecture of Dict-TTS is shown in Figure 2.
Dict-TTS keeps the main structuresof PortaSpeech: a Transformer-based linguistic encoder; a VAE-based variational generator withflow-based prior to generate diverse mel-spectrogram.
The flow-based post-net in PortaSpeech isreplaced with a multi-length discriminator [7] based on random windows of different lengths, whichhas been proved to improve the naturalness of word pronunciations [55].
However, since there are nophoneme inputs in our scenarios, we replace the linguistic encoder that combines hard word-levelalignment and soft phoneme-level alignment with: 1) a semantic encoder that extracts the semanticrepresentations in the grapheme sequence; 2) a semantics-to-pronunciation attention module thatmatches the semantic patterns between the dictionary entries and the grapheme representations andobtains the corresponding semantic embedding and pronunciation embedding; 3) a linguistic encoderthat fuses the semantic embedding and pronunciation embedding.
[Figure]
3.2 Comparison between the phoneme-based and character-based TTS systems
[Figure]
In this subsection, we make preliminary analyses about thelinguistic encoder of phoneme-based and character-based TTSsystems.
For simplicity, we describe the following conceptsaccording to the logographic writing system, where a singlewritten character represents a complete grammatical word ormorpheme.
These concepts can be extended to alphabeticlanguages like English by replacing “character” with “word”.
Phoneme-based TTS systems As is shown in Figure 3, thelinguistic encoder of the phoneme-based TTS system takesphoneme sequence p generated by the G2P module as inputs.
The main challenge of the phoneme-based linguistic encoderis to comprehend the semantic and syntactic representation sfrom p and deduce the pitch trajectory, speaking duration, andother acoustic features from s and p to generate the expressive and natural pronunciation hidden state g .
Since the phoneme sequence p is a combination of the smallest units of sound in speech,it may be ambiguous in terms of semantic meaning, which brings difficulties for the deduction ofthe representation s .
For example, “AE1, T, F, ER1, S, T” can be easily classified as “At first”, but“W, EH1, DH, ER0” can be classified as “Whether” or “Weather”.
Homophones like “to”, “too”,and “two” can be converted to the same phoneme sequence “T, UW1”, but their local speakingduration and pitch are different.
Moreover, the tree-structured syntactic information contained in theword-based input sequence is missing.
Since semantic information and syntactic information possessrich intonational features such as pitch accent and phrasing of the input text [55], the ambiguity of shurts the prosody modeling in phoneme-based TTS systems.
Character-based
TTS systems The first challenge for a character-based TTS system is to predictthe correct phoneme sequence p .
Unlike the phoneme-based TTS system, the character-based TTSsystem does not know the phoneme sequence p when characters arrive.
Thus, it does not know how topronounce the words accurately at first.
Then the mel-spectrogram reconstruction loss in TTS trainingwould drag the character representation c to the acoustic space.
For example, Chinese characters “ 火 ”(“fire” in English) and “ 伙 ” (“partner” in English) share the same pronunciation (“H UO3”) and havedifferent semantic meanings.
However, with the guidance of the mel-spectrogram reconstruction loss,their representations distribute according to the acoustic pronunciation, which hinders the semanticcomprehension for polyphone disambiguation and prosody modeling.
From the above analyses, it can be seen that the character representations c should locate in thesemantic space so that we can easily capture s based on the context, deduce p based on the dictionaryand s , and finally obtain the natural pronunciation hidden state g .
The following subsection mainlydescribes how we achieve the above goals with our semantics-to-pronunciation attention module.
3.3 Semantics-to-Pronunciation Attention
As shown in Figure 2, the semantics-to-pronunciation attention(S2PA) module is designed for explicit semantics comprehension and polyphone disambiguation.
Dictionary Definition Assuming that we have an dictionary
D which contains a sequence of characters
C =
[ c 1 , c 2 , ..., c n ] ,where n is the size of characters set in a language 2 .
Inthe dictionary, each character c i has a sequence of possible pronunciations p
i
=
[ p i, 1 , p i, 2 , ..., p i,m ] and each pronunciation p i,j has its corresponding dictionary entry
e i,j =
[ e i,j, 1 , e i,j, 2 , ..., e i,j,u ] ∈ E (definitions, usages, and translations are merged together as a single characters sequence),where m is the number of the possible pronunciation of c i andu is the number of characters in the corresponding entry.
Notethat for polyphones m > 1 and for characters that have onlyone pronunciation m = 1 .
Semantics Matching
The goal of our S2PA is to obtain the pronunciation sequence p by measuringthe semantic similarity between the input character sequence t =
[ t 1 , ..., t l ] and the correspondinggloss items in the dictionary, where l is the sequence length.
As shown in Figure 4, we firstlyextract the semantic context information k of every entry e with a pretrained cross-lingual languagemodel [10] and store k as the prior dictionary knowledge.
Then we use a semantic encoder to obtainthe semantic contexts z from the input character sequence t .
For each character token t i , its semanticfeature vector z i is used as the query vector to a semantics-based attention module.
Here, attention isused to learn a similarity measure between the semantic feature vector z i and k i,j,k :
[ a i, 1 , 1 , ..., a i,m,u ]
[Equation]
where d is the scaling factor and [ a i, 1 , 1 , ..., a i,m,u ] denotes the semantic similarity between t i andeach item in [ e i, 1 , 1 , e i,m,u ] .
The retrieved semantic embeddings s ′ i can be extracted by s ′ i
=
softmax ([ a i, 1 , 1 , ..., a i,m,u ]) · [ k i, 1 , 1 , ..., k i,m,u ] .
The rich linguistic information in s ′ i can be usedas auxiliary information to improve the naturalness and expressiveness of the generated speeches.
Polyphone Disambiguation
The aggregated attention weight w i,j = (cid:80) uk =1 a i,j,k can be seen asthe probability of the pronunciation p i,j .
However, since a polyphone in a specific sentence has onlyone correct pronunciation, here we use the Gumbel-Softmax function [20] to sample a differentiableapproximation of the most possible pronunciation p ′
i :
[Equation] [Equation]
where g i, 1 , ..., i,m are i.i.d samples drawn from Gumbel(0,1) distribution and τ is the softmax temperature.
Then the retrieved pronunciation embeddings p ′
i and semantic embeddings s ′
i are then fedinto the rest of the linguistic encoder for feature fusion and syntax prediction.
Intuitively, our S2PAmodule can be thought of as an end-to-end method for decomposing the character representation,pronunciation, and semantics with dictionary following the preliminary analyses in Subsection 3.2.
Through our S2PA module, the character representation is successfully distributed in the semanticspace so that the model can easily deduce the correct pronunciation and semantics based on thelexicon knowledge like human brain.
3.4 Training and Pre-training
In training, the S2PA module weights (including character token embeddings and the pronunciationtoken embeddings) are jointly trained by the reconstruction loss from the TTS decoder.
Thus, our
Dict-TTS does not require any explicit phoneme labels.
In inference, the pronunciations can bespecified by feeding the text sequence into the S2PA module to get the predicted pronunciations.
Besides, our method is compatible with the predefined rules from language experts by directly addingspecific rules to pronunciation weight w i,j .
Although the semantics-to-pronunciation mappings can be explicitly learned by the S2PA module, itcould be not accurate enough, due to the following reason: the text training data is not large enough(about 10k sentences) in the TTS dataset, leading to relatively inaccurate context comprehension.
To improve the semantic comprehension and the generalization capacity for our S2PA module,we propose a pre-training method using low-quality text-speech pairs from large-scaled automaticspeech recognition (ASR) datasets.
Since our S2PA module can be trained without hand-craftedphoneme labels, it can be easily pre-trained and effectively finetuned to various domains to improvethe pronunciation accuracy of the TTS systems.
Experiments
4.1 Experimental Setup
Datasets We evaluate Dict-TTS on three datasets of different sizes, including: 1) Biaobei [3], a
Chinese speech corpus consisting of 10,000 sentences (about 12 hours) from a Chinese speaker;2)
JSUT [43], a Japanese speech corpus containing reading-style speeches from a Japanese femalespeaker (we use the basic5000 subset that contains 5,000 daily-use sentences).
3) Common Voice(HK) [1], a Cantonese speech corpus that contains 125 hours of speeches from 2,869 speakers (Weuse the 102 hours of the validated speeches).
For each of the three datasets, we randomly sample 400samples for validation and 400 samples for testing.
We randomly choose 50 samples in the test setfor subjective audio quality and prosody evaluation and use all testing samples for other evaluations.
The ground truth mel-spectrograms are generated from the raw waveform with the frame size 1024and the hop size 256.
For computational efficiency, we firstly use the pre-trained XLM-R [10] modelto extract the semantic representations from the raw text of the whole dictionary and record them inthe disk.
We then load the mini-batch along with the pre-constructed dictionary representation duringtraining and testing.
[Table]
Implementation Details Our Dict-TTS consists of a S2PA module, an encoder, a variationalgenerator and a post-net.
The encoder consists of multiple feed-forward Transformer blocks [38] withrelative position encoding [40] following Glow-TTS [24].
The encoder and decoder in variationalgenerator are 2D-convolution networks following PortaSpeech [37].
We replace the post-net in
PortaSpeech with a multi-length discriminator [7], which has been proved to improve the naturalnessof word pronunciations [55].
We add more detailed model configurations in Appendix B.1, B.2.
Wetrain Dict-TTS on 1 NVIDIA 3080Ti GPU, with batch size of 40 sentences on each GPU.
We use the
Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
98 , ϵ = 10 − 9 and follow the same learning rate schedulein [49].
The softmax temperature τ is initialized and annealed using the schedule in [20].
It takes320k steps for training until convergence.
The predicted mel-spectrograms are transformed into audiosamples using pre-trained HiFi-GAN [26] 3 .
4.2 Results of Pronunciation Accuracy
[Table]
We compare the pronunciation accuracyof our Dict-TTS with other systems, including 1) character-based systems, wherewe directly feed character into the linguistic encoder; 2) Phoneme-based systems,where we convert the text sequence tothe phoneme sequence [51, 41, 24] withmost popular open-source grapheme-tophoneme tools 4 .
We measure objectivephoneme error rate (PER-O), subjectivephoneme error rate (PER-S), and subjective sentence error rate (SER-S) in theevaluations.
The phoneme labels in theobjective PER evaluation are from the corresponding dataset (since the Common
Voice (HK) dataset does not have phoneme labels, we only evaluate the subjective metrics).
Inthe subjective evaluations, each audio in the test set is listened by at least 4 language experts.
Weask them to write down the mispronounced phonemes and discuss them with each other until aconclusion is reached.
Note that for SER-S, the error rate is calculated in sentence level (e.g., asentence with multiple errors will be counted only once).
More details about these evaluationscan be found in Appendix B.3.
The results are shown in Table 1.
It can be seen that Dict-TTSgreatly surpasses the character-based baseline in three languages.
Moreover, Dict-TTS achieves thecomparable objective and subjective PER with the most popular grapheme-to-phoneme tools on tworelatively larger datasets (Biaobei and Common Voice (HK)) and maintain a good pronunciationaccuracy on a relatively small dataset (JSUT), which demonstrates the superiority of the explicitsemantics matching in our S2PA module.
We also compare the pronunciation accuracy of our Dict-TTS with various types of systems, including:1) a character-based system; 2) a BERT embedding based system [15], where the BERT derived embeddings are concatenated with the character embeddings; 3) NLR [16], a TTS system that
[Figure]
directly injects BERT derived knowledge into the linguistic encoder without phoneme labels; 4)
Phoneme (G2PM [34]), PortaSpeech with phoneme labels derived from G2PM (a powerful neural
G2P system); 5) Phoneme (pypinyin), PortaSpeech with phoneme labels derived from pypinyin (oneof the most popular Chinese G2P system).
As shown in Table 2, Dict-TTS greatly surpasses thesystems that implicitly model the semantic representations for character-to-pronunciation mappinglike NLR [16] and shows comparable performance with phoneme-based systems.
Since our Dict-TTSdoes not require any phoneme labels for training, we can pre-train Dict-TTS on a large-scaled ASRdataset [57] with a small amount of effort to improve its generalization capacity.
It can be seenthat the pronunciation accuracy of Dict-TTS on the Biaobei dataset is significantly improved bypre-training, which demonstrates the effectiveness of our unsupervised polyphone disambiguationframework.
4.3 Results of Audio Quality and Prosody
[Figure]
[Table]
We compare the audio quality, audioprosody, and pitch accuracy 5 of Dict-TTSwith the systems evaluated in Subsection 4.2.
GT (the ground truth audio) and
GT (voc.)
(the ground truth audio thatis firstly converted into mel-spectrogramand converted back to audio waveform using Hifi-GAN [26]) are also included inthis experiment.
We keep the text content consistent among different models toexclude other interference factors, onlyexamining the audio quality or prosody.
Each audio is listened by at least 20 testers.
For audio quality and prosody, we conduct the mean opinion score (MOS) evaluation via Amazon Mechanical Turk.
Weanalyze the MOS in two aspects: MOSP (Prosody: naturalness of pitch, energy,and duration) and MOS-Q (Quality: {+clar
 We (DTW) [32]+}
[Table]
ity, high-frequency, and original timbre reconstruction).
We tell the tester to focus on one corresponding aspect and ignore the other aspect when scoring.
We put more information about thesubjective evaluation in Appendix B.3.
As shown in Table 3, for audio quality, Dict-TTS significantlyoutperforms those TTS systems without phoneme labels and achieves a comparable performancewith phoneme-based systems.
And for audio prosody and pitch accuracy, Dict-TTS even surpassesthe phoneme-based systems, which demonstrates the effectiveness of the extracted semantics fromprior dictionary knowledge.
We put more analyses on the naturalness of prosody in Appendix E.
We then visualize the mel-spectrograms generated by the above systems in Figure 5.
We can see that
Dict-TTS can generate mel-spectrograms with comparable details in harmonics, unvoiced frames,and high-frequency parts with the phoneme-based system, which results in similar natural sounds.
Moreover, our Dict-TTS can capture more accurate local changes in pitch and speaking duration,indicating the effectiveness of introducing semantic representations in the dictionary.
4.4 Ablation Studies
We conduct ablation studies to demonstrate the effectiveness of designs in Dict-TTS, including theauxiliary semantic information and the Gumbel-Softmax sample strategy.
We conduct pronunciationaccuracy and CMOS (comparative mean opinion score) evaluations for these ablation studies.
Theresults are shown in Table 4.
We can see that CMOS-P drops when we remove the introduced semanticembeddings in Dict-TTS, indicating that the semantic information extracted from the dictionarycan improve the audio prosody.
Besides, to demonstrate the effectiveness of the Gumbel-Softmaxsample strategy, we also compare the weighted sum of the pronunciation embeddings with the
Gumbel-Softmax sample strategy.
Since measuring PER-O requires one-hot vectors, we do notcalculate the PER-O score for the weight-sum version of Dict-TTS.
The results are shown in row 3in Table 4.
It can be seen that PER-S and SER-S increase when we use the weighted sum method.
In the experiments, the weights of different pronunciations for some characters might be close toeach other, which results in relatively worse performance in the subjective results.
For example, thetwo pronunciations “ZH ANG3” and “CH ANG2” of the character “ 长 ” might be ambiguous whentheir weights are close to each other (e.g., 0 . 6 and 0 . 4 ).
Therefore, to accurately model the subjectivepronunciations, we utilize the Gumbel-Softmax function to sample the most likely pronunciation inboth training and inference stages.
Conclusion
In this paper, we proposed Dict-TTS, an unsupervised framework for polyphone disambiguation inend-to-end text-to-speech systems.
Dict-TTS uses a semantics-to-pronunciation attention (S2PA)module to explicitly extract the corresponding pronunciations and semantic information from priordictionary knowledge.
The S2PA module can be trained with the end-to-end TTS model with theguidance of mel-spectrogram reconstruction loss without phoneme labels, which significantly reducesthe cost of building a polyphone disambiguation system and further enables the efficient pre-trainingon large-scaled ASR datasets.
Our experimental results in three languages show that Dict-TTSoutperforms several strong G2P baseline models in terms of pronunciation accuracy and improves theprosody modeling of the baseline TTS system.
Further comprehensive ablation studies demonstratethat our S2PA module successfully decomposes the character representation, pronunciation, andsemantics for character-based TTS systems.
However, the dictionary knowledge does not containthe tree-structured syntactic information of the input text sequence, which also affects the prosodymodeling.
Moreover, since we crawl the dictionary from online websites and do not make any specificchanges, the performance of Dict-TTS can be further improved by a well-designed dictionary.
In thefuture, we will try to inject syntactic information into Dict-TTS and extend it to more languages. Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeAppendix H in the supplementary materials.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemen-
tary materials.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Subsection 4.1.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We report confidence intervals of subjective metric results
and describe the random seed settings in Appendix in the supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Subsection 4.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Subsection 4.1.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] See Appendix B.3 in the supplementary materials.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] See Appendix B.3 in the supplementary
materials.