Dict-TTS: Learning to Pronounce with Prior Dictionary Knowledge for Text-to-Speech
Abstract
Polyphone disambiguation aims to capture accurate pronunciation knowledge from natural text sequences for reliable Text-to-speech (TTS) systems.
However, previous approaches require substantial annotated training data and additional efforts from language experts, making it difficult to extend high-quality neural TTS systems to out-of-domain daily conversations and countless languages worldwide.
This paper tackles the polyphone disambiguation problem from a concise and novel perspective: we propose Dict-TTS, a semantic-aware generative text-to-speech model with an online website dictionary (the existing prior information in the natural language).
Specifically, we design a semantics-to-pronunciation attention (S2PA) module to match the semantic patterns between the input text sequence and the prior semantics in the dictionary and obtain the corresponding pronunciations;
The S2PA module can be easily trained with the end-to-end TTS model without any annotated phoneme labels.
Experimental results in three languages show that our model outperforms several strong baseline models in terms of pronunciation accuracy and improves the prosody modeling of TTS systems.
Further extensive analyses demonstrate that each design in Dict-TTS is effective.
The code is available at https://github.com/Zain-Jiang/Dict-TTS .
Introduction
Capturing the pronunciations from raw texts is challenging for end-to-end text-to-speech (TTS) systems [2, 31, 42, 45, 55, 27, 14, 41, 22, 21], since there are full of words that are not covered by general pronunciation rules [4, 24, 50].
Therefore, polyphone 3 disambiguation (one of the biggest challenges in converting texts into phonemes [37, 60, 46]) plays an important role in the construction of high-quality neural TTS systems [18, 38].
However, since the exact pronunciation of a polyphone must be inferred based on its semantic contexts, current solutions still face several challenges: 1) the rule-based approaches [64, 19] with limited linguistic knowledge or the neural models [60, 38, 46]
欢喜，快活；使人快乐的事情 ...
乐： <l è> <y uè> <y ào> <l ào>
Pronunciation
Definitions 声音，成调的声音。或姓氏 ...
Usages 快乐。乐融融。其乐无穷。乐观。乐天。取乐。逗乐。快乐 ...
Character 音乐。声乐。乐池。乐音。乐曲（①音乐与歌曲；②伴奏 ... 喜好、欣赏。用于文言文 ...
知者乐水，仁者乐山。 地名用字。 河北省乐亭、山东省乐陵
。 trained on limited data suffer from significant performance degradation on out-of-domain text datasets; 2) the neural network-based models [6, 5, 38] learn the grapheme-to-phoneme (G2P) mapping in an end-to-end manner without explicit semantics modeling, which hinders their pronunciation accuracy in real-life applications. 3) based on the above two points, a reliable polyphone disambiguation module is usually based on a combination of hand-crafted rules, structured G2P-oriented lexicons, and neural models [16], which requires substantial phonemes labels and external knowledge from language experts.
Unlike the previous rule-based or neural network-based approaches, we address the above challenges with the existing prior information worldwide.
As shown in Figure 1, an arbitrary dictionary used in daily life can be viewed as a prior knowledge database.
Intuitively, it contains valuable prior knowledge for pronunciations in conversations.
When one is confused about the acoustic pronunciation of a specific polyphone, he or she will resort to the dictionary website to infer its exact reading based on the semantic context.
We imitate this scenario in our architecture design and propose Dict-TTS, an unsupervised polyphone disambiguation framework, which explicitly
consults
the online dictionary to
identify the correct
semantic meanings and acoustic
pronunciations
of polyphones.
Specifically
, • To explicitly learn the semantics-to-pronunciation mapping, we adopt a semantic encoder to obtain the semantic contexts of the input text and utilize a semantics-to-pronunciation attention (S2PA) module to search the matched semantic patterns in the dictionary so as to find the correct pronunciations.
We also use the retrieved semantic information as auxiliary information for prosody modeling of the TTS model.
• To perform polyphone disambiguation without phoneme labels, we combine our S2PA module into end-to-end TTS systems’ training and inference processes.
Different from current neural polyphone disambiguation models, our module can be trained with the guidance of mel-spectrogram reconstruction loss in a fully end-to-end manner, which significantly reduces the cost of building such a system.
To demonstrate the generalization ability of our Dict-TTS, we perform experiments on three datasets, including a standard Mandarin dataset [3], a Japanese corpus [47], and a Cantonese dataset [1].
Experiments on these datasets show that Dict-TTS outperforms other state-of-the-art polyphone disambiguation models in pronunciation accuracy and improves the prosody modeling of TTS systems in terms of both subjective and objective evaluation metrics.
The pronunciation accuracy of Dict-TTS is further improved by being pre-trained on a large-scaled automatic speech recognition (ASR) dataset.
The main contributions of this work are summarized as follows:
• We incorporate the online dictionary into TTS systems and propose a semantic-aware method for polyphone disambiguation, which improves the pronunciation accuracy and robustness of end-to-end TTS systems.
Moreover, the idea of introducing the prior knowledge worldwide can also inspire other tasks like neural language modeling [26] and sequence labeling [34].
• We propose a novel and general framework for unsupervised polyphone disambiguation in TTS systems, which further enables the efficient pre-training on large-scaled ASR datasets and improves the generalization capacity significantly.
• We also find that the retrieved semantics in the dictionary knowledge can be used as auxiliary information
to improve prosody modeling and help the TTS system to generate more expressive speech.
• We further analyze the characteristics of the linguistic encoder based on phoneme and character and provide valuable interpretations about our semantics-to-pronunciation attention module.
Background
This section describes the background of TTS, grapheme-to-phoneme (G2P) pipeline, and their relations with polyphone disambiguation.
We also review the existing works that aim at semanticaware polyphone disambiguation and analyze their advantages and disadvantages.
Text-to-Speech Text-to-speech (TTS) models [55, 2, 30, 31, 42, 27, 41, 32] first generate melspectrogram from text and then synthesize speech waveform from the generated mel-spectrogram using a separately pre-trained vocoder [52, 29, 20], or directly generate waveform from text in an end-to-end manner [40, 13, 28].
The frontend model of end-to-end TTS system should tackle one important task, i.e., polyphone disambiguation [65].
Properly mapping the grapheme sequence into phoneme sequence requires the linguistic encoder to capture the empirical pronunciation rules in daily conversations.
However, it is extremely difficult for the linguistic encoder to learn all the pronunciation rules necessary to produce speech in an end-to-end manner, which results in inevitable mispronunciations in the generated speech.
To alleviate this problem, robust TTS models usually convert the text sequence into the phoneme sequence with open-source grapheme-to-phoneme pipelines and predict the mel-spectrogram from the phoneme sequence.
However, the rule-based or neural network-based grapheme-to-phoneme pipelines suffer from significant performance degradation on out-of-domain datasets since it is extremely difficult and costly for them to cover all linguistic knowledge.
Grapheme-to-Phoneme The grapheme-to-phoneme models map grapheme sequence to phoneme sequence to reduce pronunciation errors in modern TTS systems.
For logographic languages like
Chinese, Japanese and Korean, although the lexicon can cover nearly all the characters, there are full of polyphones that can only be decided according to the semantic context of a character [50].
Thus, polyphone disambiguation is the most important challenge in grapheme-to-phoneme conversions for this kind of languages.
Moreover, many alphabetic languages including English and French also have polyphones in daily conversations.
Current polyphone disambiguation approaches can be categorized into the rule-based approach [64, 19] and the data-driven approach [43, 6, 5, 38].
The rule-based
G2P method is based on a combination of hand-crafted rules and structured G2P-oriented lexicons, which requires a substantial amount of linguistic knowledge.
The data-driven G2P algorithm adopts statistical methods [33] or neural encoder-decoder architecture [58, 5, 48, 60, 38, 46, 54].
However, building a data-driven G2P model requires a large amount of carefully labeled data and substantial linguistic knowledge from language experts, which is extremely costly and laborious.
Semantic-Aware Polyphone Disambiguation
Polyphone disambiguation is the core issue for G2P conversion in various languages.
The pronunciation of a polyphone is defined by the semantic context information of neighbouring characters [50].
In order to comprehend the semantic meaning in the given sentence for polyphone disambiguation, previous methods [11, 57, 49, 18, 8] have adopted the pre-trained language model [12] to extract semantic features from raw character sequences and predict the pronunciation of polyphones with neural classifiers according to the semantic features.
Among them, PnG BERT and Mixed-Phoneme BERT [25, 62] take both phoneme and grapheme as input to train an augmented BERT and use the pre-trained augmented BERT as the TTS encoder.
However, these methods still require annotated data to train and can not be incorporated into the TTS training in an end-to-end manner.
Although NLR [16] directly injects BERT-derived knowledge into the TTS systems without phoneme labels and successfully reduces pronunciation errors, their method confounds the acoustic and semantic space, which significantly affects the pronunciation accuracy.
Method
To exploit the prior linguistic knowledge in the online dictionary for TTS systems, we propose
Dict-TTS, which explicitly captures the semantic relevance between the input sentences and the dictionary entries for polyphone disambiguation.
In this section, we firstly introduce the overall architecture design of Dict-TTS based on PortaSpeech [41].
Then after the comparison between the
Input Character Sequence
Semantic Encoder
Attention 0.
Semantic Embeddings
PronunciationEmbeddings
Linguistic Encoder
Variational Generator
ML-Discriminator
External Vocoder
Dictionary Entries
Pronunciations
Semantics-to-Pronunciation Attention 听乐队 ...
（ listen to the band ... ）
快乐 音乐 欣赏 地名 乐观 乐 0.
取乐 ...
乐曲 声乐 ...
文言 文 ...
用字 乐陵 ...
l e y ue y ao
l ao
Weights x ...
Semantic Space
Acoustic Space
SemanticContexts phoneme-based and character-based TTS systems in the acoustic and semantic space, we design a novel semantics-to-pronunciation attention (S2PA) module to learn grapheme-to-phoneme mappings based on the semantic context; In general, Dict-TTS exploits the prior pronunciation knowledge in the online dictionary with the following steps: Firstly, the character sequence is fed into the self-attention based semantic encoder to obtain the semantic representations of the input character sequence, and we utilize a pre-trained cross-lingual language model [10] to extract the semantic context information in the dictionary entries;
Secondly, we calculate the most relevant dictionary entries of the input graphemes and obtain the corresponding pronunciation sequence contained in the dictionary entries;
Finally, the extracted semantic context information and the pronunciations are fed into the linguistic encoder for feature fusion.
We describe these designs in detail in the following subsections.
3.1 Model Overview
The overall model architecture of Dict-TTS is shown in Figure 2.
Dict-TTS keeps the main structures of PortaSpeech: a Transformer-based linguistic encoder; a VAE-based variational generator with flow-based prior to generate diverse mel-spectrogram.
The flow-based post-net in PortaSpeech is replaced with a multi-length discriminator [7] based on random windows of different lengths, which has been proved to improve the naturalness of word pronunciations [59].
However, since there are no phoneme inputs in our scenarios, we replace the linguistic encoder that combines hard word-level alignment and soft phoneme-level alignment with: 1) a semantic encoder that extracts the semantic representations in the grapheme sequence; 2) a semantics-to-pronunciation attention module that matches the semantic patterns between the dictionary entries and the grapheme representations and obtains the corresponding semantic embedding and pronunciation embedding; 3) a linguistic encoder that fuses the semantic embedding and pronunciation embedding.
3.2 Comparison between the phoneme-based and character-based TTS systems
In this subsection, we make preliminary analyses about the linguistic encoder of phoneme-based and character-based TTS systems.
For simplicity, we describe the following concepts according to the logographic writing system, where a single written character represents a complete grammatical word or morpheme.
These concepts can be extended to alphabetic languages like English by replacing “character” with “word”.
[Figure]
Phoneme-based TTS systems As is shown in Figure 3, the linguistic encoder of the phoneme-based TTS system takes phoneme sequence p generated by the G2P module as inputs.
The main challenge of the phoneme-based linguistic encoder is to comprehend the semantic and syntactic representation s from p and deduce the pitch trajectory, speaking duration, and other acoustic features from s and p to generate the expressive and natural pronunciation hidden state g .
Since the phoneme sequence p is a combination of the smallest units of sound in speech, it may be ambiguous in terms of semantic meaning, which brings difficulties for the deduction of the representation s .
For example, “AE1, T, F, ER1, S, T” can be easily classified as “At first”, but “W, EH1, DH, ER0” can be classified as “Whether” or “Weather”.
Homophones like “to”, “too”, and “two” can be converted to the same phoneme sequence “T, UW1”, but their local speaking duration and pitch are different.
Moreover, the tree-structured syntactic information contained in the wordbased input sequence is missing.
Since semantic information and syntactic information possess rich intonational features such as pitch accent and phrasing of the input text [59], the ambiguity of s hurts the prosody modeling in phoneme-based TTS systems.
Character-based TTS systems
The first challenge for a character-based TTS system is to predict the correct phoneme sequence p .
Unlike the phoneme-based TTS system, the character-based TTS system does not know the phoneme sequence p when characters arrive.
Thus, it does not know how to pronounce the words accurately at first.
Then the mel-spectrogram reconstruction loss in TTS training would drag the character representation c to the acoustic space.
For example, Chinese characters “ 火 ” (“fire” in English) and “ 伙 ” (“partner” in English) share the same pronunciation (“H UO3”) and have different semantic meanings.
However, with the guidance of the mel-spectrogram reconstruction loss, their representations distribute according to the acoustic pronunciation, which hinders the semantic comprehension for polyphone disambiguation and prosody modeling.
From the above analyses, it can be seen that the character representations c should locate in the semantic space so that we can easily capture s based on the context, deduce p based on the dictionary and s , and finally obtain the natural pronunciation hidden state g .
The following subsection mainly describes how we achieve the above goals with our semantics-to-pronunciation attention module.
3.3 Semantics-to-Pronunciation Attention
As shown in Figure 2, the semantics-to-pronunciation attention (S2PA) module is designed for explicit semantics comprehension and polyphone disambiguation.
Dictionary Definition Assuming that we have an dictionary
D which contains a sequence of characters
C =
[ c 1 , c 2 , ..., c n ] , where n is the size of characters set in a language 4 .
In the dictionary, each character c i has a sequence of possible pronunciations p
i
=
[ p i, 1 , p i, 2 , ..., p i,m ] and each pronunciation p i,j has its corresponding dictionary entry
e i,j =
[ e i,j, 1 , e i,j, 2 , ..., e i,j,u ] ∈ E (definitions, usages, and translations are merged together as a single characters sequence), where m is the number of the possible pronunciation of c
i and u is the number of characters in the corresponding entry.
Note that for polyphones m > 1 and for characters that have only one pronunciation m = 1 .
Semantics Matching
The goal of our S2PA is to obtain the pronunciation sequence p by measuring the semantic similarity between the input character sequence t =
[ t 1 , ..., t l ] and the corresponding gloss items in the dictionary, where l is the sequence length.
As shown in Figure 4, we firstly extract the semantic context information k of every entry e with a pretrained cross-lingual language model [10] and store k as the prior dictionary knowledge.
Then we use a semantic encoder to obtain the semantic contexts z from the input character sequence t .
For each character token t i , its semantic feature vector z i is used as the query vector to a semantics-based attention module.
Here, attention is used to learn a similarity measure between the semantic feature vector z i and k i,j,k :
[Equation]
where d is the scaling factor and [ a i, 1 , 1 , ..., a i,m,u ] denotes the semantic similarity between t i and each item in [ e i, 1 , 1 , e i,m,u ] .
The retrieved semantic embeddings s ′ i can be extracted by s ′ i
=
softmax ([ a i, 1 , 1 , ..., a i,m,u ]) · [ k i, 1 , 1 , ..., k i,m,u ] .
The rich linguistic information in s ′ i can be used as auxiliary information to improve the naturalness and expressiveness of the generated speeches.
Polyphone Disambiguation
The aggregated attention weight w i,j = (cid:80) uk =1 a i,j,k can be seen as the probability of the pronunciation p i,j .
However, since a polyphone in a specific sentence has only one correct pronunciation, here we use the Gumbel-Softmax function [23] to sample a differentiable approximation of the most possible pronunciation p ′
i :
[Equation]
where g i, 1 , ..., i,m are i.i.d samples drawn from Gumbel(0,1) distribution and τ is the softmax temperature.
Then the retrieved pronunciation embeddings p ′
i and semantic embeddings s ′
i are then fed into the rest of the linguistic encoder for feature fusion and syntax prediction.
Intuitively, our S2PA module can be thought of as an end-to-end method for decomposing the character representation, pronunciation, and semantics with dictionary following the preliminary analyses in Subsection 3.2.
Through our S2PA module, the character representation is successfully distributed in the semantic space so that the model can easily deduce the correct pronunciation and semantics based on the lexicon knowledge like human brain.
3.4 Training and Pre-training
In training, the S2PA module weights (including character token embeddings and the pronunciation token embeddings) are jointly trained by the reconstruction loss from the TTS decoder.
Thus, our
Dict-TTS does not require any explicit phoneme labels.
In inference, the pronunciations can be specified by feeding the text sequence into the S2PA module to get the predicted pronunciations.
Besides, our method is compatible with the predefined rules from language experts by directly adding specific rules to pronunciation weight w i,j .
Although the semantics-to-pronunciation mappings can be explicitly learned by the S2PA module, it could be not accurate enough, due to the following reason: the text training data is not large enough (about 10k sentences) in the TTS dataset, leading to relatively inaccurate context comprehension.
To improve the semantic comprehension and the generalization capacity for our S2PA module, we propose a pre-training method using low-quality text-speech pairs from large-scaled automatic speech recognition (ASR) datasets.
Since our S2PA module can be trained without hand-crafted phoneme labels, it can be easily pre-trained and effectively finetuned to various domains to improve the pronunciation accuracy of the TTS systems.
Experiments 4.1 Experimental Setup
Datasets We evaluate Dict-TTS on three datasets of different sizes, including: 1) Biaobei [3], a
Chinese speech corpus consisting of 10,000 sentences (about 12 hours) from a Chinese speaker;
[Table]
2) JSUT [47], a Japanese speech corpus containing reading-style speeches from a Japanese female speaker (we use the basic5000 subset that contains 5,000 daily-use sentences).
Common Voice (HK) [1], a Cantonese speech corpus that contains 125 hours of speeches from 2,869 speakers (We use the 102 hours of the validated speeches).
For each of the three datasets, we randomly samplesamples for validation and 400 samples for testing.
We randomly choose 50 samples in the test set for subjective audio quality and prosody evaluation and use all testing samples for other evaluations.
The ground truth mel-spectrograms are generated from the raw waveform with the frame sizeand the hop size 256.
For computational efficiency, we firstly use the pre-trained XLM-R [10] model to extract the semantic representations from the raw text of the whole dictionary and record them in the disk.
We then load the mini-batch along with the pre-constructed dictionary representation during training and testing.
Implementation Details Our Dict-TTS consists of a S2PA module, an encoder, a variational generator and a post-net.
The encoder consists of multiple feed-forward Transformer blocks [42] with relative position encoding [44] following Glow-TTS [27].
The encoder and decoder in variational generator are 2D-convolution networks following PortaSpeech [41].
We replace the post-net in
PortaSpeech with a multi-length discriminator [7], which has been proved to improve the naturalness of word pronunciations [59].
We add more detailed model configurations in Appendix B.1, B.2.
We train Dict-TTS on 1 NVIDIA 3080Ti GPU, with batch size of 40 sentences on each GPU.
We use the
Adam optimizer with β 1 = 0 .
9 , β 2 = 0 .
98 , ϵ = 10 − 9 and follow the same learning rate schedule in [53].
The softmax temperature τ is initialized and annealed using the schedule in [23].
It takes 320k steps for training until convergence.
The predicted mel-spectrograms are transformed into audio samples using pre-trained HiFi-GAN [29] 5 .
4.2 Results of Pronunciation Accuracy
[Table]
We compare the pronunciation accuracy of our Dict-TTS with other systems, including 1) character-based systems, where we directly feed character into the linguistic encoder; 2) Phoneme-based systems, where we convert the text sequence to the phoneme sequence [55, 45, 27] with most popular open-source grapheme-tophoneme tools 6 .
We measure objective phoneme error rate (PER-O), subjective phoneme error rate (PER-S), and subjective sentence error rate (SER-S) in the evaluations.
The phoneme labels in the objective PER evaluation are from the corresponding dataset (since the Common
Voice (HK) dataset does not have phoneme labels, we only evaluate the subjective metrics).
In the subjective evaluations, each audio in the test set is listened by at least 4 language experts.
We ask them to write down the mispronounced phonemes and discuss them with each other until a https
[Figure]
conclusion is reached.
Note that for SER-S, the error rate is calculated in sentence level (e.g., a sentence with multiple errors will be counted only once).
More details about these evaluations can be found in Appendix B.3.
The results are shown in Table 1.
It can be seen that Dict-TTS greatly surpasses the character-based baseline in three languages.
Moreover, Dict-TTS achieves the comparable objective and subjective PER with the most popular grapheme-to-phoneme tools on two relatively larger datasets (Biaobei and Common Voice (HK)) and maintain a good pronunciation accuracy on a relatively small dataset (JSUT), which demonstrates the superiority of the explicit semantics matching in our S2PA module.
We also compare the pronunciation accuracy of our Dict-TTS with various types of systems, including: 1) a character-based system; 2) a BERT embedding based system [15], where the BERT derived embeddings are concatenated with the character embeddings; 3) NLR [16], a TTS system that directly injects BERT derived knowledge into the linguistic encoder without phoneme labels; 4)
Phoneme (G2PM [38]), PortaSpeech with phoneme labels derived from G2PM (a powerful neural
G2P system); 5) Phoneme (pypinyin), PortaSpeech with phoneme labels derived from pypinyin (one of the most popular Chinese G2P system).
As shown in Table 2, Dict-TTS greatly surpasses the systems that implicitly model the semantic representations for character-to-pronunciation mapping like NLR [16] and shows comparable performance with phoneme-based systems.
Since our Dict-TTS does not require any phoneme labels for training, we can pre-train Dict-TTS on a large-scaled ASR dataset [61] with a small amount of effort to improve its generalization capacity.
It can be seen that the pronunciation accuracy of Dict-TTS on the Biaobei dataset is significantly improved by pre-training, which demonstrates the effectiveness of our unsupervised polyphone disambiguation framework.
4.3 Results of Audio Quality and Prosody
We compare the audio quality, audio prosody, and pitch accuracy 7 of Dict-TTS with the systems evaluated in Subsection 4.2.
GT (the ground truth audio) and GT (voc.)
(the ground truth audio that is firstly converted into mel-spectrogram and converted back to audio waveform using Hifi-GAN [29]) are also included in this experiment.
We keep the text content consistent among different models to exclude other interference factors, only examining the audio quality or prosody.
Each audio is listened by at least 20 testers.
For audio quality and prosody, we conduct the mean opinion score (MOS) evaluation via Amazon Mechanical Turk.
[Table] [Table]
We analyze the MOS in two aspects:
MOS-P (Prosody: naturalness of pitch, energy, and duration) and MOS-Q (Quality: clarity,
high-frequency
[Table]
and original timbre reconstruction).
We tell the tester to focus on one corresponding aspect and ignore the other aspect when scoring.
We put more information about the subjective evaluation in Appendix B.3.
As shown in
Table 3, for audio quality, Dict-TTS significantly outperforms those TTS systems without phoneme labels and achieves a comparable performance with phonemebased systems.
And for audio prosody and pitch accuracy, Dict-TTS even surpasses the phoneme-based systems, which demonstrates the effectiveness of the extracted semantics from prior dictionary knowledge.
We put more analyses on the naturalness of prosody in Appendix E.
We then visualize the mel-spectrograms generated by the above systems in Figure 5.
We can see that
Dict-TTS can generate mel-spectrograms with comparable details in harmonics, unvoiced frames, and high-frequency parts with the phoneme-based system, which results in similar natural sounds.
Moreover, our Dict-TTS can capture more accurate local changes in pitch and speaking duration, indicating the effectiveness of introducing semantic representations in the dictionary.
4.4 Ablation Studies
We conduct ablation studies to demonstrate the effectiveness of designs in Dict-TTS, including the auxiliary semantic information and the Gumbel-Softmax sample strategy.
We conduct pronunciation accuracy and CMOS (comparative mean opinion score) evaluations for these ablation studies.
The results are shown in Table 4.
We can see that CMOS-P drops when we remove the introduced semantic embeddings in Dict-TTS, indicating that the semantic information extracted from the dictionary can improve the audio prosody.
Besides, to demonstrate the effectiveness of the Gumbel-Softmax sample strategy, we also compare the weighted sum of the pronunciation embeddings with the
Gumbel-Softmax sample strategy.
Since measuring PER-O requires one-hot vectors, we do not calculate the PER-O score for the weight-sum version of Dict-TTS.
The results are shown in rowin Table 4.
It can be seen that PER-S and SER-S increase when we use the weighted sum method.
In the experiments, the weights of different pronunciations for some characters might be close to each other, which results in relatively worse performance in the subjective results.
For example, the two pronunciations “ZH ANG3” and “CH ANG2” of the character “ 长 ” might be ambiguous when their weights are close to each other (e.g., 0 . 6 and 0 . 4 ).
Therefore, to accurately model the subjective pronunciations, we utilize the Gumbel-Softmax function to sample the most likely pronunciation in both training and inference stages.
Conclusion
In this paper, we proposed Dict-TTS, an unsupervised framework for polyphone disambiguation in end-to-end text-to-speech systems.
Dict-TTS uses a semantics-to-pronunciation attention (S2PA) module to explicitly extract the corresponding pronunciations and semantic information from prior dictionary knowledge.
The S2PA module can be trained with the end-to-end TTS model with the guidance of mel-spectrogram reconstruction loss without phoneme labels, which significantly reduces the cost of building a polyphone disambiguation system and further enables the efficient pre-training on large-scaled ASR datasets.
Our experimental results in three languages show that Dict-TTS outperforms several strong G2P baseline models in terms of pronunciation accuracy and improves the prosody modeling of the baseline TTS system.
Further comprehensive ablation studies verify that each component in Dict-TTS is effective.
However, the dictionary knowledge does not contain the tree-structured syntactic information of the input text sequence, which also affects the prosody modeling.
Moreover, since we crawl the dictionary from online websites and do not make any specific changes, the performance of Dict-TTS can be further improved by a well-designed dictionary.
In the future, we will try to inject syntactic information into Dict-TTS and extend it to more languages.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (Grant
No.62072397 and No.61836002), Zhejiang Natural Science Foundation (LR19F020006), Yiwise, and National Key R&D Program of China (Grant No.2020YFC0832505).
We appreciate the support from Mindspore, which is a new deep learning computing framework.
This work was also supported by Alibaba Group through Alibaba Innovative Research Program.
A Details of Models
In this section, we describe details about the dictionary construction, the architecture of the semantic encoder and linguistic encoder, and multi-length discriminator.
For further information about the variational generator, one could refer to PortaSpeech [41].
We also describe our modifications to model architecture to support the pre-training task of Dict-TTS with large-scaled ASR datasets.
A.1 Dictionary Construction
The Chinese dictionary used in the experiments is obtained from https://github.com/yihui/ zdict , which is crawled from https://www.zdic.net/zd/zb/ty/ .
The Japanese dictionary is crawled from https://dictionary.goo.ne.jp and the Cantonese dictionary is crawled from https://humanum.arts.cuhk.edu.hk/Lexis/lexi-can .
The crawled dictionaries are processed into the format described in SubSection 3.3.
We use a pre-trained cross-lingual language model [10] to extract the semantic context information k of every gloss entry e and store them in the disk for computational efficiency 8 .
Note that our
Dict-TTS does not need the heavy BERT model in the inference stage.
As is pointed out by the previous works, lower layers of BERT are found to perform broad attention across all pairs of words or encode local syntax [9, 39] and middle layers are found to mostly capture transferable syntactic and semantic knowledge [17, 51].
But the upper layers are specifically tuned towards the pre-training tasks of BERT [56].
Thus, we use the average of the first-layer word embeddings and the later 8-layer contextual BERT representations as the extracted knowledge.
A.2 Architecture of the Semantic Encoder and Linguistic Encoder
[Figure]
~
[Figure]
As shown in Figure 6, the semantic encoder and linguistic encoder of our Dict-TTS are both stacks of feed-forward Transformer layers with relative position encoding [44] following
PortaSpeech [41].
And there are four layers of the semantic encoder and four layers of the linguistic encoder in our Dict-TTS.
A.3 Modifications for Pre-training Task
The vanilla architecture of PortaSpeech is tested on LJSpeech (a single-speaker dataset).
By introducing a group of learnable speaker embeddings to represent the speakers’ timbre, pronunciation habits, and other features, it can be trained on multispeaker datasets.
However, since large-scaled ASR datasets may not have explicit speaker information, the following modifications should be made: we extract the speaker embeddings from audio samples using resemblyzer 9 and feed them into the variational generator and duration predictor.
We pre-train
our
DictTTS on the WenetSpeech dataset [61], which is a multidomain corpus for speech recognition.
It takes 600k steps for pre-training until convergence.
As is shown in Table 2, the semantic comprehension and the generalization capacity of our Dict-TTS are significantly improved by our pre-training process.
B Detailed Experimental Settings
In this section, we describe more model configurations and details in subjective evaluation.
B.1 Dict-TTS Model Configurations
We list the model hyper-parameters of Dict-TTS in Table
[Table] [Table]
B.2 Baseline Model Configurations
The baseline systems in our experiments can be divided into: 1) phoneme-based systems; 2) character-based systems.
For phoneme-based systems, we use the mixture alignment proposed in PortaSpeech [41]
[Table]
which takes phoneme sequence as inputs but utilize both soft phoneme-level and hard word-level duration for mixture alignment.
We use a 4-layer phoneme encoder and a 4-layer word encoder following the implementation of PortaSpeech [41].
For character-based systems, we use an 8-layer character encoder for fair comparisons.
The encoders above are stacks of feed-forward
Transformer layers with relative position encoding [44] following PortaSpeech [41].
Other parts of the architecture are kept the same as Dict-TTS to exclude other interference factors.
B.3 Details in Subjective Evaluation
Audio Quality and Prosody
We perform the audio quality and prosody evaluation on Amazon
Mechanical Turk (MTurk).
For each dataset, we randomly select 50 texts from the test set and use the
TTS systems to generate the audio samples.
Each audio has been listened to by at least 20 listeners.
For MOS, each tester is asked to evaluate the subjective naturalness of a sentence on a 1-5 Likert scale.
For CMOS, listeners are asked to compare pairs of audio generated by systems A and B, indicate which of the two audio they prefer, and choose one of the following scores: 0 indicating no difference, 1 indicating a slight difference, 2 indicating a significant difference and 3 indicating a very large difference.
For audio quality evaluation (MOS-Q and CMOS-Q), we tell listeners to “ focus on examining the naturalness of audio quality (e.g., noise, timbre, sound clarity, and high-frequency details), and ignore the differences of prosody and rhythm (e.g., pitch, energy, and duration) ”.
For prosody evaluations (MOS-P and CMOS-P), we tell listeners to “ focus on examining the naturalness of prosody and rhythm (e.g., pitch, energy, and duration), and ignore the differences in audio quality (e.g., noise, timbre, sound clarity, and high-frequency details) ”.
The screenshots of instructions for
[Figure]
testers are shown in Figure 7.
We paid $8 to participants hourly and totally spent about $500 on participant compensation.
Pronunciation Accuracy We perform the pronunciation accuracy evaluation on MolarData 10 .
For each dataset, we use all texts in the test set to generate the audio samples.
Each audio has been listened to by at least 4 language experts.
Each tester is asked to carefully listen to the audio multiple times, write down the mispronounced phonemes, and discuss with each other until a conclusion is reached.
The screenshots of instructions for testers are shown in Figure 8.
We paid $15 to participants hourly and totally spent about $800 on participant compensation.
A small subset of speech samples used in the test is available at https://dicttts.github.io/DictTTS-Demo/ .
B.4 Details of the G2P tools used in the experiments
We use pypinyin (0.46.0) 11 and G2PM 12 in Biaobei dataset.
Until 20 May, 2022, the latest version of pypinyin is 0.46.0.
And it is worth noting that the pronunciation accuracy is greatly improved when we update the version from 0.36.0 to 0.46.0.
In JSUT dataset, we use pyopenjtalk (0.2.0) 13 .
And in
Common Voice (HK) dataset, we use pycantonese (3.4.0) 14 .
B.5 Details of Datasets
In this subsection, we mainly describe the distribution characteristics of polyphones in the datasets used in our experiments, including: 1) Biaobei [3]; 2) JSUT [47] and 3) Common Voice (HK) [1].
As shown in Figure 9, we calculate the number of polyphones in every sentence.
The average sentence lengths of the Biaobei, JSUT, and Common Voice (HK) datasets are 18, 29, and 10.
Among these datasets, the most frequent number of polyphones appearing in a sentence is three to five and some sentences even include eight (or more) polyphones, indicating the importance of polyphone disambiguation.
B.6 Error Bars and Random Seeds
For the experiments of the audio quality and prosody, we report confidence intervals of subjective metric results in Table 3.
For the experiments of the pronunciation accuracy, we ran the experimentstimes with 10 different random seeds ( [1234 , 1111 , 2222 , 3333 , 4444 , 5555 , 6666 , 7777 , 8888 , 9999] ) and obtained the averaged results.
C Visualization of Attention Weights
We put some semantics attention visualizations in Figure 10.
We can see that Dict-TTS can create reasonable text-to-dictionary alignments, which improves the performance of polyphone disambiguation and helps the semantics comprehension for end-to-end TTS systems.
[Figure]
D The Definition of Polyphone and Heteronym
Polyphones are letters or characters having more than one phonetic value, while heteronyms are words that have two (or more) different possible pronunciations that are associated with two (or more) different meanings [35].
According to the previous research [63], the difficulty of polyphone disambiguation in logographic languages mainly lies in heteronyms.
Thus, the previous works [11, 49, 38] usually use “polyphone disambiguation” to refer to “heteronym disambiguation” for those languages.
In this work, we use “polyphone disambiguation” to refer to “heteronym disambiguation” following the previous works.
E Analyses on Prosodic Realization
The DTW measure would be impacted by both lexical tone and prosodic realization.
In order to further analyse the naturalness of prosody and rhythm for Dict-TTS and baseline systems, we evaluate the duration errors and character-level average pitch errors.
For duration errors, we calculate the MSE of character-level durations.
For character-level average pitch errors, we firstly calculate the mean pitch for each character’s region in the mel spectrogram according to the Montreal Forced Aligner (MFA) to remove the influence of lexical tone, and then we calculate the MSE of the mean pitch sequences.
The results on the Biaobei dataset are shown in Table 6.
It can be seen that the duration error and the average pitch error of Dict-TTS are significantly lower than those of the baseline systems, demonstrating the effectiveness of the extracted semantics from prior dictionary knowledge.
[Table]
F Adding Rules to the Pronunciation Weights
There are some pronunciation rules (like “sandhi rules”) that can not be learned from the dictionary.
For example, in Mandarin, " 一 " before tone4 should be “Y I2” (e.g., “ 一 段 ”) and when “ 一 ” is an ordinal word, it should be “Y I1”
(e.g., “ 一 四 九五 年 ”).
According to these pronunciation rules, we can obtain the correct pronunciation labels for some specific characters based on the part-of-speech (POS) tags of the input character sequence.
After we obtain the correct pronunciation labels for these specific characters, we can directly force the pronunciation weights of these characters to be the ground truth values.
G Polyphone Disambiguation for Various Languages
The polyphone disambiguation problem is critical in logographic languages such as Chinese, but is less problematic in phonograms like English.
For logographic languages like Chinese, although the lexicon can cover nearly all the characters, there are a lot of polyphones that can only be decided according to the context of a character.
Thus,
G2P conversion in this kind of languages is mainly responsible for polyphone disambiguation, which decides the appropriate pronunciation based on the current word context.
Therefore, polyphone disambiguation is crucial in these languages and our method is an effective solution for the polyphone disambiguation problem in these languages.
For alphabetic languages like English, lexicon cannot cover the pronunciations of all the words.
Thus, the G2P conversion for English is mainly responsible for generating the pronunciations of out-of-vocabulary words [50].
Although the polyphone disambiguation is less problematic in these languages, our methods can still be used as the modules to retrieve the correct pronunciation for polyphones and heteronyms in their G2P process (e.g., the
Algorithm step 2 in https://github.com/Kyubyong/g2p ).
In our experiments, JSUT dataset is a mixture of phonograms and logograms, which is different from Biaobei and Common Voice (HK).
Japanese writing system consists of two types of characters: the kanji ( 漢 字 ) and the syllabic kana – hiragana ( 平 仮 名 ) and katakana ( 片 仮 名 ).
In our analysis, 32.42% of the characters in JSUT dataset are kanji.
The pronunciations of a part of the kanji can not only be specified by the semantic information and should be specified by empirical pronunciation rules.
For example, most kanji ( 漢 字 ) can be pronounced multiple ways: on-yomi ( 音 読 み ) and kun-yomi ( 訓 読 み ).
Although the compound kanji usually uses on-yomi and one kanji probably uses kunyomi, the different readings are largely just chosen empirically in practice.
Our Dict-TTS has the potential to work only for the kanji whose pronunciation should be specified based on the semantic meaning.
Due to the characteristics of Japanese writing systems, in Table 1, although Dict-TTS surpasses the character-based system, it does not show comparable performance with the open source
G2P module in Japanese.
But as shown in Section 3.4, our method is compatible with the predefined rules from language experts by directly adding specific rules to pronunciation weight.
We are sure that the performance of our method can be further improved by introducing the pronunciation rules in Japanese (e.g., the rules in the rule-based G2P baseline "pyopenjtalk").
H Potential Negative Societal Impacts
Dict-TTS improves the pronunciation accuracy and prosody of the synthesized speech voice and lowers the requirements for G2P conversion, which may cause unemployment for people with related occupations.
Besides, the production of fake speeches may cause voice security issues.
Further efforts in automatic speaker verification should be made to improve voice security.