A DA S PEECH : A DAPTIVE T EXT TO S PEECH FOR C USTOM V OICE
A BSTRACT
Custom voice, a speciﬁc text to speech (TTS) service in commercial speech platforms, aims to adapt a source TTS model to synthesize personal voice for a target speaker using few speech from her/him.
Custom voice presents two unique challenges for TTS adaptation: 1) to support diverse customers, the adaptation model needs to handle diverse acoustic conditions which could be very different from source speech data, and 2) to support a large number of customers, the adaptation parameters need to be small enough for each target speaker to reduce memory usage while maintaining high voice quality.
In this work, we propose AdaSpeech, an adaptive TTS system for high-quality and efﬁcient customization of new voices.
We design several techniques in AdaSpeech to address the two challenges in custom voice: 1) To handle different acoustic conditions, we model the acoustic information in both utterance and phoneme level.
Speciﬁcally, we use one acoustic encoder to extract an utterance-level vector and another one to extract a sequence of phoneme-level vectors from the target speech during pre-training and ﬁne-tuning; in inference, we extract the utterance-level vector from a reference speech and use an acoustic predictor to predict the phonemelevel vectors.
2) To better trade off the adaptation parameters and voice quality, we introduce conditional layer normalization in the mel-spectrogram decoder of AdaSpeech, and ﬁne-tune this part in addition to speaker embedding for adaptation.
We pre-train the source TTS model on LibriTTS datasets and ﬁne-tune it on VCTK and LJSpeech datasets (with different acoustic conditions from LibriTTS) with few adaptation data, e.g., 20 sentences, about 1 minute speech.
Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K speciﬁc parameters for each speaker, which demonstrates its effectiveness for custom voice.
The audio samples are available at https://speechresearch.github.io/adaspeech/ .
I NTRODUCTION
Text to speech (TTS) aims to synthesize natural and intelligible voice from text, and attracts a lot of interests in machine learning community (Arik et al., 2017;
Wang et al., 2017; Gibiansky et al., 2017;
Ping et al., 2018; Shen et al., 2018; Ren et al., 2019).
TTS models can synthesize natural human voice when training with a large amount of high-quality and single-speaker recordings (Ito, 2017), and has been extended to multi-speaker scenarios (Gibiansky et al., 2017; Ping et al., 2018;
Zen et al., 2019; Chen et al., 2020) using multi-speaker corpora (Panayotov et al., 2015; Veaux et al., 2016; Zen et al., 2019).
However, these corpora contain a ﬁxed set of speakers where each speaker still has a certain amount of speech data.
Nowadays, custom voice has attracted increasing interests in different application scenarios such as personal assistant, news broadcast and audio navigation, and has been widely supported in commercial speech platforms (some custom voice services include Microsoft Azure, Amazon AWS and Google
Cloud). In custom voice, a source TTS model is usually adapted on personalized voices with few adaptation data, since the users of custom voice prefer to record as few adaptation data as possible (several minutes or seconds) for convenient purpose.
Few adaptation data presents great challenges on the naturalness and similarity of adapted voice.
Furthermore, there are also several distinctive challenges in custom voice: 1)
The recordings of the custom users are usually of different acoustic conditions from the source speech data (the data to train the source TTS model).
For example, the adaptation data is usually recorded with diverse speaking prosodies, styles, emotions, accents and recording environments.
The mismatch in these acoustic conditions makes the source model difﬁcult to generalize and leads to poor adaptation quality.
When adapting the source TTS model to a new voice, there is a trade-off between the ﬁne-tuning parameters and voice quality.
Generally speaking, more adaptation parameters will usually result in better voice quality, which, as a result, increases the memory storage and serving cost 1 .
While previous works in TTS adaptation have well considered the few adaptation data setting in custom voice, they have not fully addressed the above challenges.
They ﬁne-tune the whole model (Chen et al., 2018;
Kons et al., 2019) or decoder part (Moss et al., 2020; Zhang et al., 2020), achieving good quality but causing too many adaptation parameters.
Reducing the amount of adaptation parameters is necessary for the deployment of commercialized custom voice.
Otherwise, the memory storage would explode as the increase of users.
Some works only ﬁne-tune the speaker embedding (Arik et al., 2018; Chen et al., 2018), or train a speaker encoder module (Arik et al., 2018;
Jia et al., 2018; Cooper et al., 2020; Li et al., 2017; Wan et al., 2018) that does not need ﬁne-tuning during adaptation.
While these approaches lead a light-weight and efﬁcient adaptation, they result in poor adaptation quality.
Moreover, most previous works assume the source speech data and adaptation data are in the same domain and do not consider the setting with different acoustic conditions, which is not practical in custom voice scenarios.
In this paper, we propose AdaSpeech, an adaptive TTS model for high-quality and efﬁcient customization of new voice.
AdaSpeech employ a three-stage pipeline for custom voice: 1) pre-training; 2) ﬁne-tuning; 3) inference.
During the pre-training stage, the TTS model is trained on large-scale multi-speaker datasets, which can ensure the TTS model to cover diverse text and speaking voices that is helpful for adaptation.
During the ﬁne-tuning stage, the source TTS model is adapted on a new voice by ﬁne-tuning (a part of) the model parameters on the limited adaptation data with diverse acoustic conditions.
During the inference stage, both the unadapted part (parameters shared by all custom voices) and the adapted part (each custom voice has speciﬁc adapted parameters) of the TTS model are used for the inference request.
We build AdaSpeech based on the popular non-autoregressive TTS models (Ren et al., 2019; Peng et al., 2020;
Kim et al., 2020; Ren et al., 2021) and further design several techniques to address the challenges in custom voice:
• Acoustic condition modeling.
In order to handle different acoustic conditions for adaptation, we model the acoustic conditions in both utterance and phoneme level in pre-training and ﬁne-tuning.
Speciﬁcally, we use two acoustic encoders to extract an utterance-level vector and a sequence of phoneme-level vectors from the target speech, which are taken as the input of the mel-spectrogram decoder to represent the global and local acoustic conditions respectively.
In this way, the decoder can predict speech in different acoustic conditions based on these acoustic information.
Otherwise, the model would memorize the acoustic conditions and cannot generalize well.
In inference, we extract the utterance-level vector from a reference speech and use another acoustic predictor that is built upon the phoneme encoder to predict the phoneme-level vectors.
• Conditional layer normalization.
To ﬁne-tune as small amount of parameters as possible while ensuring the adaptation quality, we modify the layer normalization (Ba et al., 2016) in the melspectrogram decoder in pre-training, by using speaker embedding as the conditional information to generate the scale and bias vector in layer normalization.
In ﬁne-tuning, we only adapt the parameters related to the conditional layer normalization.
In this way, we can greatly reduce adaptation parameters and thus memory storage 2 compared with ﬁne-tuning the whole model, but maintain high-quality adaptation voice thanks to the ﬂexibility of conditional layer normalization.
To evaluate the effectiveness of our proposed AdaSpeech for custom voice, we conduct experiments to train the TTS model on LibriTTS datasets and adapt the model on VCTK and LJSpeech datasets with different adaptation settings.
Experiment results show that AdaSpeech achieves better adaptation quality in terms of MOS (mean opinion score) and SMOS (similarity MOS) than baseline methods, with only about 5K speciﬁc parameters for each speaker, demonstrating its effectiveness for custom voice.
Audio samples are available at https://speechresearch.github.io/adaspeech/ .
A DA S PEECH
[Figure]
Phoneme Encoder
Phoneme Embedding
Phoneme
[Figure]
Mel Decoder
[Figure] In this section, we ﬁrst describe the overall design of our proposed
AdaSpeech, and then introduce the key techniques to address the challenges in custom voice.
At last, we list the pre-training, ﬁnetuning and inference pipeline of AdaSpeech for custom voice.
The model structure of AdaSpeech is shown in Figure 1.
We adopt
FastSpeech 2 (Ren et al., 2021) as the model backbone considering the FastSpeech (Ren et al., 2019; 2021) series are one of the most popular models in non-autoregressive TTS.
The basic model backbone consists of a phoneme encoder, a mel-spectrogram decoder, and a variance adaptor which provides variance information including duration, pitch and energy into the phoneme hidden sequence following Ren et al.
As shown in Figure 1, we design two additional components to address the distinctive challenges in custom voice: 1) to support diverse customers, we use acoustic condition modeling to capture the diverse acoustic conditions of adaptation speech in different granularities; 2) to support a large number of customers with affordable memory storage, we use conditional layer normalization in decoder for efﬁcient adaptation with few parameters while high voice quality.
In the next subsections, we introduce the details of these components respectively.
2. A COUSTIC C ONDITION M ODELING
In custom voice, the adaptation data can be spoken with diverse prosodies, styles, accents, and can be recorded under various environments, which can make the acoustic conditions far different from that in source speech data.
This presents great challenges to adapt the source TTS model, since the source speech cannot cover all the acoustic conditions in custom voice.
A practical way to alleviate this issue is to improve the adaptability (generalizability) of source TTS model.
In text to speech, since the input text lacks enough acoustic conditions (such as speaker timbre, prosody and recording environments) to predict the target speech, the model tends to memorize and overﬁt on the training data (Ren et al., 2021), and has poor generalization during adaptation.
A natural way to solve such problem is to provide corresponding acoustic conditions as input to make the model learn reasonable text-to-speech mapping towards better generalization instead of memorizing.
To better model the acoustic conditions with different granularities, we categorize the acoustic conditions in different levels as shown in Figure 2a: 1) speaker level, the coarse-grained acoustic conditions to capture the overall characteristics of a speaker; 2) utterance level, the ﬁne-grained acoustic conditions in each utterance of a speaker; 3) phoneme level, the more ﬁne-grained acoustic conditions in each phoneme of an utterance, such as accents on speciﬁc phonemes, pitches, prosodies and temporal environment noises 3 .
Since speaker ID (embedding) is widely used to capture speakerlevel acoustic conditions in multi-speaker scenario (Chen et al., 2020), speaker embedding is used by default.
We describe the utterance-level and phoneme-level acoustic condition modeling as follows
• Utterance Level.
We use an acoustic encoder to extract a vector from a reference speech, similar to Arik et al.
; Jia et al.
; Cooper et al.
(2020), and then expand and add it to the phoneme hidden sequence to provide the utterance-level acoustic conditions.
As shown in Figure 2b, the acoustic encoder consists of several convolutional layers and a mean pooling layer to get a single vector.
The reference speech is the target speech during training, while a randomly chosen speech of this speaker during inference.
• Phoneme Level.
We use another acoustic encoder (shown in Figure 2c) to extract a sequence of phoneme-level vectors from the target speech and add it to the phoneme hidden sequence to
exist, but have marginal beneﬁts considering their prediction difﬁculty.
Similarly, more coarse-grained language level conditions also exist, but we do not consider multilingual setting in this work and leave it for future work.
[Figure]
Mean Pooling
Conv1D
[Figure]
[Figure]
Mel
Utterance-Level Vector
[Figure]
Conv1D
[Figure]
[Figure]
Linear Layer
Conv1D
[Figure]
[Figure]
Phoneme
[Figure]
Phoneme
[Figure] Level Vectors [Figure] [Figure]
Linear Layer
Conv1D (
[Figure]
ReLU
[Figure] LN [Figure] Phoneme
[Figure]
Predicted Phoneme-Level Vectors
[Figure]
As shown in Figure 2a, the phoneme-level vectors are directly added element-wisely into the hidden sequence utterance level speaker level then added the hidden sequence.
provide the phoneme-level acoustic conditions 4 .
In order to extract phoneme-level information from speech, we ﬁrst average the speech frames corresponding to the same phoneme according to alignment between phoneme and mel-spectrogram sequence (shown in Figure 2a), to convert to length of speech frame sequence into the length of phoneme sequence, similar to Sun et al.
Zeng et al.
During inference, we use another phoneme-level acoustic predictor (shown in
Figure 2d) which is built upon the original phoneme encoder to predict the phoneme-level vectors.
Using speech encoders to extract a single vector or a sequence of vectors to represent the characteristics of a speech sequence has been adopted in previous works (Arik et al., 2018;
Jia et al., 2018;
Cooper et al., 2020; Sun et al., 2020; Zeng et al., 2020).
They usually leverage them to improve the speaker timbre or prosody of the TTS model, or improve the controllability of the model.
The key contribution in our acoustic condition modeling in this work is the novel perspective to model the diverse acoustic conditions in different granularities to make the source model more adaptable to different adaptation data.
As analyzed in Section 4.2, utterance-level and phoneme-level acoustic modeling can indeed help the learning of acoustic conditions and is critical to ensure the adaptation quality.
[Figure]
Achieving high adaptation quality while using small adaptation parameters is challenging.
Previous works use zero-shot adaptation with speaker encoder (Arik et al., 2018;
Jia et al., 2018; Cooper et al., 2020) or only ﬁne-tune the speaker embedding cannot achieve satisﬁed quality.
Can we greatly increase the voice quality at the cost of slightly more but negligible parameters?
To this end, we analyze the model parameters of
FastSpeech 2 (Ren et al., 2021), which is basically built upon the structure of Transformer (Vaswani et al., 2017), with a self-attention network and a feed-forward network in each Transformer block.
Both the matrix multiplications in the query, key, value and output of self-attention and two-layer feed-forward networks are parameter-intensive, which is not efﬁcient to adapt.
We ﬁnd that layer normalization (Ba et al., 2016) is adopted in each self-attention and feed-forward network in decoder, which can greatly inﬂuence the hidden activation and ﬁnal prediction with a light-weight learnable scale vector γ and bias vector β : LN ( x ) =
γ x − µσ + β , where µ and σ are the mean and variance of hidden vector x .
If we can determine the scale and bias vector in layer normalization with the corresponding speaker characteristics using a small conditional network, then we can ﬁne-tune this conditional network when adapting to a new voice, and greatly reduce the adaptation parameters while ensuring the adaptation quality.
As shown in Figure 3, the conditional network consists of two simple linear layers W γc and
W βc that take speaker embedding E s as input and output the scale and bias vector respectively:
[Equation]
where s denotes the speaker ID, and c ∈ [ C ] denotes there are C conditional layer normalizations in the decoder (the number of decoder layer is ( C − 1) / 2 since each layer has two conditional layer normalizations corresponding to self-attention and feed-forward network in Transformer, and there is an additional layer normalization at the ﬁnal output) and each uses different conditional matrices.
2. P IPELINE OF A DA S PEECH
We list the pre-training, ﬁne-tuning and inference pipeline of AdaSpeech in Algorithm 1.
During ﬁne-tuning, we only ﬁne-tune the two matrices W γc and W βc in each conditional layer normalization in decoder and the speaker embedding E s , ﬁxing other model parameters including the utterance-level and phoneme-level acoustic encoders and phoneme-level acoustic predictor as described in Section 2.1.
During inference, we do not directly use the two matrices W γc and W βc in each conditional layer normalization since they still have large parameters.
Instead we use the two matrices to calculate each scale and bias vector γ sc and β sc from speaker embedding E s according to Equation 1 considering E s is ﬁxed in inference.
In this way, we can save a lot of memory storage 5 .
1: Pre-training : Train the AdaSpeech model θ with source training data D . 2: Fine-tuning : Fine-tune W γc and W βc in each conditional layer normalization c ∈
[ C ] and speaker embedding E s with the adaptation data D s for each custom
speaker/voice s .
3: Inference : Deployment : 1) Calculate γ sc , β sc in each conditional layer normalization c ∈
[ C ] , and get the parameters θ s = {{ γ sc , β sc } Cc =1 , E s } for speaker s . 2)
Deploy the shared model parameters ˜
θ (not ﬁne-tuned in θ during adaptation)
and speaker speciﬁc parameters θ s for s .
Inference : Use ˜ θ
and θ s to synthesize custom voice for speaker s .
E XPERIMENTAL S ETUP
Datasets We train the AdaSpeech source model on LibriTTS (Zen et al., 2019) dataset, which is a multi-speaker corpus (2456 speakers) derived from LibriSpeech (Panayotov et al., 2015) and contains 586 hours speech data.
In order to evaluate AdaSpeech in custom voice scenario, we adapt the source model to the voices in other datasets including VCTK (Veaux et al., 2016) (a multi-speaker datasets with 108 speakers and 44 hours speech data) and LJSpeech (Ito, 2017) (a single-speaker high-quality dataset with 24 hours speech data), which have different acoustic conditions from LibriTTS.
As a comparison, we also adapt the source model to the voices in the same LibriTTS dataset.
We randomly choose several speakers (including both male and female) from the training set of
LibriTTS and VCTK and the only single speaker from the training set of LJSpeech for adaptation.
For each chosen speaker, we randomly choose K = 20 sentences for adaptation and also study the effects of smaller K in experiment part.
We use all the speakers in the training set of LibriTTS (exclude those chosen for adaptation) to train the source AdaSpeech model, and use the original test sets in these datasets corresponding to the adaptation speakers to evaluate the adaptation voice quality.
We conduct the following preprocessing on the speech and text data in these corpora: 1) convert the sampling rate of all speech data to 16kHz; 2) extract the mel-spectrogram with 12.5ms hop size and 50ms window size following the common practice in Shen et al.
; Ren et al.
(2019); 3) convert
Model Conﬁgurations
The model of AdaSpeech follows the basic structure in FastSpeech 2 (Ren et al., 2021), which consists of 4 feed-forward Transformer blocks for the phoneme encoder and melspectrogram decoder.
The hidden dimension (including the phoneme embedding, speaker embedding, the hidden in self-attention, and the input and output hidden of feed-forward network) is set to 256.
The number of attention heads, the feed-forward ﬁlter size and kernel size are set to 2, 1024 and 9 respectively.
The output linear layer converts the 256-dimensional hidden into 80-dimensional mel-spectrogram.
Other model conﬁgurations follow Ren et al.
(2021) unless otherwise stated.
The phoneme-level acoustic encoder (Figure 2c) and predictor (Figure 2d) share the same structure, which consists of 2 convolutional layers with ﬁlter size and kernel size of 256 and 3 respectively, and a linear layer to compress the hidden to a dimension of 4 (we choose the dimension of 4 according to our preliminary study and is also consistent with previous works (Sun et al., 2020; Zeng et al., 2020)).
We use MFA (McAuliffe et al., 2017) to extract the alignment between the phoneme and mel-spectrogram sequence, which is used to prepare the input of the phoneme-level acoustic encoder.
We also tried to leverage VQ-VAE (Sun et al., 2020) into the phoneme-level acoustic encoder but found no obvious gains.
The utterance-level acoustic encoder consists of 2 convolutional layers with ﬁlter size, kernel size and stride size of 256, 5 and 3, and a pooling layer to obtain a single vector.
Training, Adaptation and Inference In the source model training process, we ﬁrst train
AdaSpeech for 60,000 steps, and all the model parameters are optimized except the parameters of phoneme-level acoustic predictor.
Then we train AdaSpeech and the phoneme-level acoustic predictor jointly for the remaining 40,000 steps, where the output hidden of the phoneme-level acoustic encoder is used as the label (the gradient is stopped to prevent ﬂowing back to the phoneme-level acoustic encoder) to train the phoneme-level acoustic predictor with mean square error (MSE) loss.
We train AdaSpeech on 4 NVIDIA P40 GPUs and each GPU has a batch size of about 12,500 speech frames.
Adam optimizer is used with β 1 = 0.9, β 2 = 0.98, (cid:15) =
10 − 9 .
In the adaptation process, we ﬁne-tune AdaSpeech on 1 NVIDIA P40 GPU for 2000 steps, where only the parameters of speaker embedding and conditional layer-normalization are optimized.
In the inference process, the utterance-level acoustic conditions are extracted from another reference speech of the speaker, and the phoneme-level acoustic conditions are predicted from phoneme-level acoustic predictor.
We use MelGAN (Kumar et al., 2019) as the vocoder to synthesize waveform from the generated mel-spectrogram.
R ESULTS
In this section, we ﬁrst evaluate the quality of the adaptation voices of AdaSpeech, and conduct ablation study to verify the effectiveness of each component in AdaSpeech, and ﬁnally we show some analyses of our method.
4. T HE Q UALITY OF A DAPTATION V OICE
We evaluate the quality of adaption voices in terms of naturalness (how the synthesized voices sound natural like human) and similarity (how the synthesized voices sound similar to this speaker).
Therefore, we conduct human evaluations with MOS (mean opinion score) for naturalness and SMOS (similarity MOS) for similarity.
Each sentence is listened by 20 judgers.
For VCTK and LibriTTS, we average the MOS and SMOS scores of multiple adapted speakers as the ﬁnal scores.
We compare
AdaSpeech with several settings: 1) GT, the ground-truth recordings; 2) GT mel + Vocoder, using ground-truth mel-spectrogram to synthesize waveform with MelGAN vocoder; 3) Baseline (spk emb), a baseline system based on FastSpeech2 which only ﬁne-tunes the speaker embedding during adaptation, and can be regarded as our lower bound; 4) Baseline (decoder), another baseline system based on FastSpeech2 which ﬁne-tunes the whole decoder during adaptation, and can be regarded as a strong comparable system since it uses more parameters during adaptation; 5) AdaSpeech, our proposed AdaSpeech system with utterance-/phoneme-level acoustic condition modeling and conditional layer normalization during adaptation 6 .
[Table]
The MOS and SMOS results are shown in Table 1.
We have several observations: 1) Adapting the model (trained on LibriTTS) to the cross-domain datasets (LJSpeech and VCTK) is more difﬁcult than adapting to the in-domain datasets (LibriTTS), since the MOS and SMOS gap between the adaptation models (two baselines and AdaSpeech) and the ground-truth mel + vocoder setting is bigger on cross-domain datasets 7 .
This also conﬁrms the challenges of modeling different acoustic conditions in custom voice scenarios.
Compared with only ﬁne-tuning speaker embedding, i.e.,
Baseline (spk emb) , AdaSpeech achieves signiﬁcant improvements in terms of both MOS and SMOS in the three adaptation datasets, by only leveraging slightly more parameters in conditional layer normalization.
We also analyze in next subsection (Table 3) that even if we increase the adaptation parameters of baseline to match or surpass that in AdaSpeech, it still performs much worse than
AdaSpeech.
Compared with ﬁne-tuning the whole decoder, i.e., Baseline (decoder) , AdaSpeech achieves slightly better quality in both MOS and SMOS and importantly with much smaller adaptation parameters, which demonstrates the effectiveness and efﬁciency of our proposed acoustic condition modeling and conditional layer normalization.
Note that ﬁne-tuning the whole decoder causes too much adaptation parameters that cannot satisfy the custom voice scenario.
4. M ETHOD A NALYSIS
Setting
CMOS
[Table]
[Table]
In this section, we ﬁrst conduct ablation studies to verify the effectiveness of each component in
AdaSpeech, including utterance-level and phonemelevel acoustic condition modeling, and conditional layer normalization, and then conduct more detailed analyses on our proposed AdaSpeech.
Ablation Study We compare the CMOS (comparison MOS) of the adaptation voice quality when removing each component in AdaSpeech on VCTK testset (each sentence is listened by 20 judgers).
Speciﬁcally, when removing conditional layer normalization, we only ﬁne-tune the speaker embedding.
From Table 2, we can see that removing utterance-level and phoneme-level acoustic modeling, and conditional layer normalization all result in performance drop in voice quality, demonstrating the effectiveness of each component in AdaSpeech.
Analyses on Acoustic Condition Modeling We analyze the vectors extracted from the utterancelevel acoustic encoder for several speakers on LibriTTS datasets.
We use t-SNE (Maaten & Hinton,
[Figure]
[Figure]
[Figure] [Figure]
S e c o n d p r i n c i p a l c o m p o n e n t [Figure] 2008) to illustrate them in Figure 4a, where each point represents an utterance-level vector and each color belongs to the same speaker.
It can be seen that different utterances of the same speaker are clustered together but have difference in acoustic conditions.
There are some exceptions, such as the two pink points one blue point in the brown solid circle.
According to our investigation on the corresponding speech data, these points correspond to the utterances with short and emotional voice, and thus are close to each other although belonging to different speakers.
Setting
CMOS
CLN
[Table]
− 0 .
[Table]
− 0 . Analyses on Conditional Layer Normalization We further compare conditional layer normalization (CLN) with other two settings: 1) LN + ﬁne-tune scale/bias: removing the condition on speaker embedding, and only ﬁne-tuning scale/bias in layer normalization and speaker embedding; 2) LN + ﬁne-tuning others: removing the condition on speaker embedding, and instead ﬁne-tuning other (similar or even larger amount of) parameters in the decoder 8 .
The CMOS evaluations are shown in Table 3.
It can be seen that both settings result in worse quality compared with conditional layer normalization, which veriﬁes its effectiveness.
Varying Adaptation Data We study the voice quality with different amount of adaptation data (fewer than the default setting) on VCTK and LJSpeech, and conduct MOS evaluation as shown in
Figure 4b.
It can be seen that the voice quality continue drops when adaptation data decreases, and drops quickly when the adaptation data is fewer than 10 sentences.
C ONCLUSIONS
In this paper, we have developed AdaSpeech, an adaptive TTS system to support the distinctive requirements in custom voice.
We propose acoustic condition modeling to make the source TTS model more adaptable for custom voice with various acoustic conditions.
We further design conditional layer normalization to improve the adaptation efﬁciency: ﬁne-tuning few model parameters to achieve high voice quality.
We ﬁnally present the pipeline of pre-training, ﬁne-tuning and inference in AdaSpeech for custom voice.
Experiment results demonstrate that AdaSpeech can support custom voice with different acoustic conditions with few memory storage and at the same time with high voice quality.
For future work, we will further improve the modeling of acoustic conditions in the source TTS model and study more diverse acoustic conditions such as noisy speech in custom voice.
We will also investigate the adaptation setting with untranscribed data (Yan et al., 2021) and further compress the model size (Luo et al., 2021) to support more custom voices.