GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech
Anonymous Author(s)
AffiliationAddress email
Abstract
Style transfer for out-of-domain (OOD) speech synthesis aims to generate speechsamples with unseen style (e.g., speaker identity, emotion, and prosody) derivedfrom an acoustic reference, while facing the following challenges: 1) The highlydynamic style features in expressive voice are difficult to model and transfer; and2) the TTS models should be robust enough to handle diverse
OOD
conditionsthat differ from the source data.
This paper proposes GenerSpeech, a text-tospeech model towards high-fidelity zero-shot style transfer of OOD custom voice.
GenerSpeech decomposes the speech variation into the style-agnostic and stylespecific parts by introducing two components: 1) a multi-level style adaptor toefficiently model a large range of style conditions, including global speaker andemotion characteristics, and the local (utterance, phoneme, and word-level) finegrained prosodic representations; and 2) a generalizable content adaptor with
Mix-Style Layer Normalization to eliminate style information in the linguisticcontent representation and thus improve model generalization.
Our evaluations onzero-shot style transfer demonstrate that GenerSpeech surpasses the state-of-the-artmodels in terms of audio quality and style similarity.
The extension studies toadaptive style transfer further show that GenerSpeech performs robustly in thefew-shot data setting.
Introduction
Text-to-speech [29, 27, 13] aims to generate almost human-like audios using text, which attractsbroad interest in the machine learning community.
These TTS models have been extended to morecomplex scenarios, including multiple speakers, emotions and styles for expressive and diverse voicegeneration [5, 19].
A growing number of applications, such as voice assistant services and long-formreading, have been actively developed and deployed to real-world speech platforms.
Increasing demand for personalized speech generation challenges TTS models especially in unseenscenarios regarding domain shifts.
Unlike typically controllable speech synthesis, style transfer forgeneralizable out-of-domain (OOD) text-to-speech aims to generate high-quality speech sampleswith unseen styles (e.g., timbre, emotion, and prosody) derived from an acoustic reference (i.e.,custom voice), which is hampered by two major challenges: 1) style modeling and transferring: thehigh dynamic range in expressive voice is difficult to control and transfer.
Many TTS models onlylearn an averaged distribution over input data and lack the ability to fine-grained control style inspeech sample; and 2) model generalization: when the distributions of style attributes in customvoice differ from training data, the quality and similarity of synthesized speech often deteriorate due
to distribution gaps.
While current TTS literature [38, 34, 19, 5]
has
considered improving modelcapability towards OOD text-to-speech, they fail to address
the above challenges fully.
Specifically,• Style modeling and transferring .
Researchers propose several methods to model and transfer thestyle attributes: 1) Global style token.
Frequently mentioned style transfer work [38] introduces theidea of global style token (GST) and its derivative [34] further manages to memorize and reproducethe global scale style features of speech.
2) Fine-grained latent variables.
Sun et al.
[32, 31]adopt VAE to represent the fine-grained prosody variable, naturally enabling sampling of differentprosody features for each phoneme.
Li et al.
[19] utilizes both the global utterance-level and thelocal quasi-phoneme-level style features of the target speech.
However, these methods are prettylimited in capturing differential style characteristics and fail to simultaneously reflect the correctspeaker identity, emotion, and prosody ranges.• Model generalization .
Researchers investigate paths to improve model generalization towardsOOD custom voice: 1) Data-driven method.
A popular approach [11, 24] to improve the robustnessof the TTS model is to pre-train on a larger dataset consisting of various speeches to expanddata distribution.
Unfortunately, this data-hungry approach requires many audio samples andcorresponding transcripts, which is often costly or even impossible.
2) Style adaptation.
Chen et al.[5] adapts new voice by finetuning on the limited adaptation data with diverse acoustic conditions.
Several works [22, 10] adopt meta-learning to adapt to new speakers that have not been seen duringtraining.
However, style adaptation relies on a strong assumption that target voice is accessiblefor model adaptation, which does not always hold in practice.
How to generalize for zero-shotout-of-domain speech synthesis is still an open problem.
An intuitive way [17, 21] to achieve better generalization is to decompose a model into the domain- agnostic and domain-specific parts via disentangled representation learning.
To address the above challenges in style transfer of OOD custom voice we propose GenerSpeech , a gener alizable textto- speech model for high-fidelity zero-shot style transfer of out-of-domain voice, including severaltechniques to model and control the style-agnostic (linguistic content) and style-specific (speakeridentity, emotion, and prosody) variations in speech separately: 1) Multi-level style adaptor .
Wepropose the multi-level style adaptor for the global and local stylization of the custom utterance.
Specifically, a downstream wav2vec 2.0 encoder generates the global latent representations to controlthe speaker and emotion characteristics.
Furthermore, three differential local style encoders model thefine-grained utterance, phoneme, and word-level prosodic representations without explicit labels.
Generalizable content adaptor .
We propose the mix-style layer normalization (MSLN) to effectivelyeliminate the style attributes in the linguistic content representation and predict the style-agnosticvariation, to improve the generalization of GenerSpeech.
We conduct experiments on zero-shot style transfer for out-of-domain text-to-speech synthesis in the
OOD testing sets.
Experimental results demonstrate that GenerSpeech achieves new state-of-the-artzero-shot style transfer results for OOD text-to-speech synthesis.
Both subjective and objectiveevaluation metrics show that GenerSpeech exhibits superior audio quality and similarity comparedwith baseline models.
The extension studies to adaptive style transfer further prove that GenerSpeechperforms robustly in the few-shot data setting.
Related Works2.1 Style Modeling and Transferring in Text-to-Speech
Style modeling and transferring have been studied for decades in the TTS community: The idea ofglobal style tokens [38] represents a success in controlling and transferring the global style.
Sun et al.[31, 32] further study a way to include a hierarchical, fine-grained prosody representation.
Li et al.[19] additionally adopt a multi-scale reference encoder to explore the phoneme-level style modeling.
The models mentioned above resort to the autoregressive generation of mel-spectrogram, sufferingfrom slow inference speed and a lack of robustness.
Recently, non-autoregressive TTS models [27, 13, 25] have been proposed and significantly speed upmel-spectrogram generation, and researchers have experimented on style modeling and transferring inparallel: Meta-StyleSpeech [22] generally adopts a speech encoding network for multi-speaker TTSsynthesis.
SC-GlowTTS [3] proposes a speaker-conditional architecture that explores a flow-baseddecoder in a zero-shot scenario.
Styler [16] models style factor via decomposition.
However, these methods focus on a limited area of style modeling, without simultaneously considering the speakeridentity, emotion and prosody variation.
2.2 Domain Generalization
Previous works on text-to-speech adaptation [5, 22] rely on a strong assumption that the target voicesare accessible for model adaptation, which does not always hold in practice.
In many scenarios,target custom voice is difficult to obtain or even unknown before deploying the TTS model, which isconsidered a more challenging problem: domain generalization (DG).
Many zero-shot DG methods are based on the idea of aligning features between different sources,with the hope that the model can be invariant to domain shift given unseen data.
Li et al.
[20] resortto adversarial learning with auxiliary domain classifiers to learn features that are domain-agnostic.
Li et al.
[18] use meta-learning to simulate train/test domain shift during training and jointly optimizethe simulated training and testing domains.
Ulyanov et al.
[33] add Instance normalization layersto eliminate instance-specific style discrepancy in the field of image style transfer.
However, allthese methods focus on the image domain.
In contrast, our work focuses on style generalization intext-to-speech synthesis, which is relatively overlooked.
GenerSpeech
In this section, we first define and formulate the generalizable text-to-speech model for zero-shot styletransfer of out-of-domain custom voice.
We then overview the proposed GenerSpeech, followingwhich we introduce several critical components including the generalizable content adaptor andmulti-level style adaptor.
Finally, we present the pre-training, training and inference pipeline of
GenerSpeech for high-fidelity out-of-domain text-to-speech synthesis.
3.1 Problem formulation
Style transfer of out-of-domain (OOD) custom voice aims to generate high quality and similarityspeech samples with unseen style (e.g., speaker identity, emotion, and prosody) derived from areference utterance, which has different acoustic conditions from training data.
3.2 Overview
We adopt one of the most popular non-autoregressive TTS models FastSpeech 2 [27] as the modelbackbone.
The overall architecture of GenerSpeech has been presented in Figure 6.
An intuitiveway [17, 21] to achieve better generalization is to decompose a model into the domain-agnosticand domain-specific parts via disentangled representation learning.
Therefore, to improve generalization in text-to-speech synthesis, we design several techniques to model the style-agnostic(linguistic content) and style-specific (e.g., speaker identity, emotion and prosody) variations inspeech separately:1) to improve model generalization, we propose the mix-style layer normalization(MSLN) to eliminatethe style information in the linguistic content representation. 2) to enhance modeling and transferringstyle attributes, we introduce a multi-level style adaptor consisting of a global encoder for speakerand emotion feature embeddings and three differential (utterance, phoneme, and word-level) localencoders for prosodic style representations. 3) to reconstruct details in these expressive speechsamples, we include a flow-based post-net [28] to refine the transform decoder output and generatefine-grained mel-spectrograms.
3.3 Generalizable Content Adaptor
To prevent the degradation in the style transfer from utterances with out-of-domain (OOD) customvoice, we eliminate the style information in phonetic sequences with the proposed Mix-Style Layer
Normalization and predict the style-agnostic prosodic variations.
3.3.1 Mix-Style Layer Normalization
Previous work [1] found that layer normalization could greatly influence the hidden activation andfinal prediction with a light-weight learnable scale vector γ and bias vector β : LN( x ) =
γ x − µσ + β ,where µ and σ are the mean and variance of hidden vector x .
[10, 5] further proposed conditionallayer normalization for speaker adaptation CLN( x, w ) =
γ ( w ) x − µσ + β ( w ) , which can adaptivelyperform scaling and shifting of the normalized input features based on the style embedding.
Here
(d
[Figure]
Adaptor
Style
[Figure]
to-ContentAlignment 𝑤 = [ ] 𝑤 !
𝑤 " 𝑤 # 𝑤 !
𝑤 " 𝑤 # 𝑤& =
[ ] (b) Shuffle Operation
[Figure]
Word
[Figure]
Phoneme Boundary
Boundary
WN Block
Conv Stacks
Pooling Layer
K,V
[Figure]
Net
[Figure]
Mel Decoder
[Figure]
Q
Vector Quantization
[Figure]
+ + + + + +
SSP Predictor
H c
G s G c
G s G c ⊕
[Figure]
two simple linear layers E γ and E δ take style embedding w as input and output the scale and biasvector respectively:
[Equation]
The custom discrepancy between the source and target domain generally hinders the generalizationcapability of learned text-to-speech models.
For disentangling style information and learning styleagnostic representation, a straightforward solution is to refine the sequence conditioned on themismatched style information, which could be regarded as injecting noise to confuse the model andprevent it from generating style-consistent representation.
Leveraging recent progress on domaingeneralization [8, 12, 40], in this work, we design the Mix-Style Layer Normalization for regularizing TTS model training by perturbing the style information in training samples:
[Equation]
where w denotes the style vector, and ˜ w is simply obtained by ˜ w =
Shuffle( w ) (See Figure 6(b)for an illustration).
where λ ∈ R B are sampled from the Beta distribution, and B represents thebatch size.
λ ∼ Beta( α, α ) with α ∈ (0 , ∞ ) being a trade-off between the original style and shufflestyle, and we set α = 0 .
2 throughout this paper.
In the end, the generalizable style-agnostic hiddenrepresentations become:
[Equation]
Consequently, the model refines the input features regularized by perturbed style and learns generalizable style-invariant content representation.
To further ensure diversity and avoid over-fitting, weperturb the style information by randomly mixing the shuffle vectors with a shuffle rate λ sampledfrom the Beta distribution.
See Algorithm 1 in Appendix B for the PyTorch-like pseudo-code.
In the final part, we adopt apitch predictor to generate the style-agnostic prosodic variations.
By utilizing the Mix-Style Layer
Normalization in the generalizable content adaptor, the linguistic content-related variation couldbe disentangled from the global style attributes (i.e., speaker and emotion), which promotes thegeneralization of TTS model towards out-of-domain custom style.
3.4 Multi-level Style adaptor
The out-of-domain custom voice generally contains high dynamic style attributes (e.g., speakeridentities, prosodies, and emotions), making the TTS model difficult to model and transfer.
As shownin Figure 6(d), we propose a multi-level style adaptor for both global and local stylization.
3.4.1 Global Representation
We use a generalizable wav2vec 2.0 model [2] to capture the global style characteristics, includingthe speaker and emotion acoustic conditions.
Wav2vec 2.0 is a recently proposed self-supervised framework for speech representation learning, which follows a two-stage training process of pretraining and finetuning, and has been demonstrated for its efficiency in learning discriminativeembedding.
In practice, we add an average pooling layer and two fully-connected layers on the top ofthe wav2vec 2.0 encoder, which allows to finetune the model on speaker and emotion classificationtasks simultaneously.
The AM-softmax [37] criteria is employed as the loss function for downstreamclassification.
In the multi-task finetuning, the training loss is obtained by the weighted sum of thelosses of these two tasks:
[Equation]
where L s and L e denote the cross entropy loss in the speaker and emotion classification, respectively, 174 and ρ adjusts the loss of these two tasks.
To sum up, the finetuned wav2vec 2.0 model generatediscriminative global representations G s , G e to model the speaker and emotion characteristics, re- 176 spectively.
We put more detailed information about the wav2vec 2.0 model in Appendix A.2.1.
3.4.2 Local Representation
To catch the fine-grained prosodic details, we consider the utterance, phoneme, and word-level threedifferential acoustic conditions.
These multi-level style encoders share the common architecture:
First, the input sequences pass through several convolutional layers to get refined.
Optionally, weconduct pooling operation on the refined series for different level stylization.
In practice, the poolingoperation averages the hidden states inside each representation according to the input boundary.
Later,the refined sequences are fed into the vector quantization later as a bottleneck [35] to eliminate thenon-prosodic information effectively.
We describe these three-level differential prosodic conditionsas follows:
1) Utterance level .
Given arbitrary contexts, speech utterances could have completely different meanings and expressions.
To catch the utterance-level latent representation S p , we remove the optional pooling layer in the local style encoder. 2) Phoneme level .
Considering the rises and falls of the pitch and stress, style patterns between phonemes could be highly dynamic.
To catch thephoneme-level style latent representation S p from speech, we take the phoneme boundary as an extrainput and apply pooling on the refined sequences before feeding into the vector quantization layer.
Word level .
Similar to the phoneme-level stylization, the acoustic conditions (e.g., pitch and stress)on each word are highly variable.
To catch the word-level style latent representation S w from speech,we take the word boundary as an extra input and apply pooling to refine the sequences.
In the Vector Quantization block, the refined sequences are fed into the vector quantization lateras a bottleneck [35] to eliminate the non-style information effectively.
The vector quantizationblock enjoys a carefully-crafted information bottleneck design.
we define a latent embedding spacee ∈ R K × D where K is the size of the discrete latent space (i.e., a K -way categorical), and D isthe dimensionality of each latent embedding vector e i .
Note that there are K embedding vectorse i ∈ R D , i ∈ 1 , 2 , . .
, K .
To make sure the representation sequence commits to an embedding andits output does not grow, we add a commitment loss following previous work [35]:
[Equation]
(5) where z e ( x ) is the output of the vector quantization block, and sg stands for the stop gradient operator.
3.4.3 Style-To-Content Alignment Layer
To align the variable-length local style representations (i.e, S u , S p , and S w ) with the phonetic 206 representations H c , we introduce the Style-To-Content Alignment Layer for learning the alignmentbetween the two modalities of style and content, which is illustrated in Figure 6(e).
In practice, we adopt the popular Scaled Dot-Product Attention [36] as the attention module.
Takingthe module in utterance-level style encoder as an example, where H c is used as the query, and S u isused as both the key and the value:
[Equation]
We add a positional encoding embedding to the style representations before they are fed into theattention module.
For efficient training, we use a residual connection [9] to add the H c .
A largedropout rate is adopted in the dropout layer to prevent the aligned representation from being directlycopied from the linguistic content representations.
For better performance, we stack the style-to-content alignment layer multiple times and graduallystylize the query value (i.e., linguistic content representations).
In the end, we utilize the pitchpredictor to generate the style-specific prosodic variations.
3.5 Flow-based Post-Net
Expressive custom voices usually contain rich and high dynamic variation, while it’s difficult for thewidely-applied transformer decoder to generate such detailed mel-spectrogram samples.
To furtherimprove the quality and similarity of synthesized mel-spectrograms, we introduce a flow-basedpost-net to refine the coarse-grained outputs of the mel-spectrogram decoder.
The architecture of the post-net follows Glow [14] family, which is conditioned on coarse-grainedspectrogram and the mel decoder input.
During training, the flow post-net converts the synthesizedmel-spectrogram into the gaussian prior distribution and calculates the exact log-likelihood of thedata.
During inference, we sample the latent variables from the prior distribution and pass them intothe post-net reversely to generate the expressive mel-spectrogram.
3.6 Pre-training, Training and Inference Procedures
3.6.1 Pre-training and Training
In the pre-training stage of GenerSpeech, we finetune the global style encoder wav2vec 2.0 modelto downstream tasks using the loss objective according to equation 4.
All parameters are adjustableduring this stage, after which we transfer the knowledge of the discriminatively-trained wav2vec 2.0model to generate global style features.
Note that we don’t use a larger dataset consisting of morevarious speeches than training to expand data distribution.
In training GenerSpeech, the reference and target speech remain the same.
The final loss termsconsist of the following parts: 1) duration prediction loss L dur : MSE between the predicted and the GT phoneme-level duration in log scale; 2) mel reconstruction loss L mel : MAE between the GTmel-spectrogram and that generated by the transformer decoder; 3) pitch reconstruction loss L p : MSEbetween the GT and the joint pitch spectrogram predicted by the style-agnostic and style-specificpitch predictor.
More details on pitch prediction have been included in Appendix A.2.2. 4) thenegative log-likelihood of the post-net L pn ; 5) commit loss L c : the objective to constrain vectorquantization layer according to Equation 5.
Also, we put details on training stability in Appendix D.
3.6.2 Inference
GenerSpeech conducts style transfer of custom voice for out-of-domain text-to-speech synthesisin the following pipeline: 1) The text encoder encodes the phoneme sequence, and the expandedrepresentations H c could be obtained according to the inference duration, following which thestyle-agnostic pitch (SAP) predictor generate the linguistic content speech variation invariant tocustom style.
Given reference speech samples, we can obtain the word and phoneme boundariesby forced alignment, which are fed into the multi-level style adaptor to model the style latentrepresentations: The wav2vec 2.0 model generates speaker G s and emotion G e representations to 251 control the global style, and the local style encoder catches the utterance, phoneme and word-levelfine-grained style representations S u , S p , and S w , respectively.
And then the style-specific pitch (SSP) 253 predictor generates the style-sensitive variation.
The mel decoder generates the coarse-grainedmel-spectrograms ˜ M , following which the flow-based post-net converts randomly sampled latentvariables into the fine-grained mel-spectrograms M conditioned on ˜ M and the mel decoder input.
Experiments
4.1 Experimental Setup4.1.1 Dataset
For both pre-training and training stages, we utilize the LibriTTS [23] dataset, which is a multispeaker corpus (2456 speakers) derived from LibriSpeech [39] and contains 586 hours of speech data.
Additionally, we use part of the ESD database [41] which consists of 13 hours of speech from 10speakers with 3 emotions (angry, happy, and neutral) to include more acoustic variation.
To evaluate
GenerSpeech in the out-of-domain scenario, we generalize the source model to other datasets withdifferent acoustic conditions from training data, including the VCTK dataset (a multi-speaker datasetwith 108 unseen speakers) and the leaving part of the ESD database with 2 unseen emotions (surprise and sad).
We randomly choose 20 sentences with speaker and emotion unseen during training toconstruct the out-of-domain (OOD) testing set.
Following the common practice [5, 22], we conduct preprocessing on the speech and text data: 1)convert the sampling rate of all speech data to 16kHz; 2) extract the spectrogram with the FFT size of1024, hop size of 256, and window size of 1024 samples; 3) convert it to a mel-spectrogram with 80frequency bins.
4.1.2 Model Configurations
GenerSpeech consists of 4 feed-forward Transformer blocks for the phoneme encoder and melspectrogram decoder.
We add a linear layer in a style adaptor to transform the 768-dimension globalembedding from wav2vec 2.0 to 256 dimensions.
The default size of the codebook in the vectorquantization layer is set to 128.
We stack two layers in the style-to-content alignment attention block.
We have attached more detailed information on the model configuration in Appendix A.
4.1.3 Training and Evaluation
After the 100,000 pre-training steps, we train GenerSpeech for 200,000 steps using 1 NVIDIA 2080Ti
GPU with a batch size of 64 sentences.
Adam optimizer is used with β 1 = 0 .
9 , β 2 = 0 .
98 , ϵ = 10 − 9 .
We utilize HiFi-GAN[15] (V1) as the vocoder to synthesize waveform from the generated melspectrogram in all our experiments.
We conduct crowd-sourced human evaluations with MOS (mean opinion score) for naturalness and
SMOS (similarity mean opinion score)
[22] for style similarity on the testing set.
Both metricsare rated from 1 to 5 and reported with 95% confidence intervals (CI).
An AXY test [30] of stylesimilarity is conducted to assess the style transfer performance, where raters are asked to rate a7-point score (from -3 to 3) and choose the speech samples which sound closer to the target style interms of style expression.
We further include objective evaluation metrics: Speaker Cosine Similarity(Cos) and F0 Frame Error (FFE) measure the timbre and prosody similarity among the synthesizedand reference audio, respectively.
More information on evaluation has been attached in Appendix C.
4.1.4 Baseline models
We compare the quality and similarity of generated audio samples of our GenerSpeech with othersystems, including 1) Reference, the reference audio; 2) Reference (voc.), where we first convertthe reference audio into mel-spectrograms and then convert them back to audio using HiFi-GAN; 3)
Mellotron [34]: The auto-regressive multi-speaker TTS model based on the Tacotron using globalstyle token (GST).
4) FG-TransformerTTS [4]: The fine-grained style control on auto-regressivemodel Transformer-TTS. 5) Expressive FS2 [27]: The combination of both multi-speaker [6] andmuli-emotion [7] FastSpeech 2, which adds the speaker and emotion d-vectors extracted by thepre-trained discriminative models to the backbone.
Meta-StyleSpeech [22]: The finetuned multispeaker text-to-speech model with meta-learning.
Styler [16]: The expressive text-to-speech modelthat model style factor via speech decomposition.
4.2 Performance
We randomly draw audio samples from the OOD testing sets as references to evaluate GenerSpeechand baseline models towards style transfer for out-of-domain text-to-speech synthesis.
And then, wesynthesize speech using the reference audio and given arbitrary text.
Considering the text consistencybetween the reference and generated speech samples, we could cluster our experiments into twocategories [30]: Parallel and Non-Parallel style transfer.
4.2.1 Parallel Style Transfer
We first demonstrate that our model can generalize to OOD custom voices when the text is unchangedfrom the reference utterance.
We calculate the objective matrixes (i.e., Cos, and FFE) between thegenerated and reference sample.
For easy comparison, the results are compiled and presented in
Table 1, and we have the following observations: 1) For audio quality, GenerSpeech has achievedthe highest MOS with scores of 4 . 06 (VCTK) and 4 .
11 (ESD) compared with the baseline models,especially in ESD dataset. 2)
In terms of style similarity, GenerSpeech score the highest overall
SMOS of 4 .
01 (VCTK) and 4 .
20 (ESD).
The objective results of both Cos and FFE further showthat GenerSpeech surpasses the state-of-the-art models in transferring the style of custom voices.
Informally, The proposed multi-level style adaptor allows GenerSpeech to generate speech samplesthat match the style of the out-of-domain reference substantially more accurately, clearly reflecting
[Table] [Table]
the correct gender, pitch, and formant ranges.
We put more visualizations of mel-spectrogramstowards parallel style transfer in Appendix F [Table]
4.2.2 Non-Parallel Style Transfer
In this section
[Table]
we explore the robustness of our proposed model in non-parallel style transfer, inwhich a TTS system synthesizes different text in the prosodic style of a reference signal.
We randomlychoose 20 reference signals from the testing set and test how TTS models replicate each style whensynthesizing different target phrases.
We conduct a subjective evaluation to assess the style similarity of synthesized speech to reference one.
As shown in Table 2, the side-by-side subjective test indicates that raters prefer GenerSpeech synthesisagainst baselines.
The proposed multi-level style adaptor significantly improves GenerSpeech toinform the speech style, allowing an expressive reference sample to guide the robust stylistic synthesisof arbitrary text.
We further plot the mel-spectrograms and corresponding pitch tracks generated by the TTS systems in
Figure 2, and have the following observations: 1) GenerSpeech can generate mel-spectrograms withrich details in frequency bins between two adjacent harmonics, unvoiced frames, and high-frequencyparts, which results in more natural sounds.
However, some baseline models (especially Mellotron)fail to generate high-fidelity mel-spectrograms in Non-Parallel style transfer; 2) GenerSpeech canresemble the prosodic style of the reference signal and demonstrates its precise style transfer, whichis nearly time-aligned in pitch contours.
However, most baseline models failed to match the prosodicstyle.
They generated the “average” distribution over their input data, generating less expressivespeech especially for long-form phrases.
4.3 Ablation Study
[Figure]
USE, PSE, and represent the respectively
[Table]
As shown in Table 3, we conduct ablation studies to demonstrate the effectiveness of several designs in GenerSpeech,including the generalizable content adaptor, multi-level styleadaptor, and the post-net.
The global representations provide the basic timbre and emotion attributes so that we leavethem unchanged.
We conduct CMOS (comparative mean opinion score) and CSMOS (comparative similarity mean opinionscore) evaluations and have the following findings: 1) Both thequality and similarity scores drop when removing utterance,phoneme, or word level style encoder, which demonstrates theefficiency of the proposed multi-level style adaptor in capturing style latent representations.
Replacing mix-style layernormalization in a generalizable content adaptor with the original one results in decreased quality and similarity, verifying thesignificance of mix-style calculation.
Removing the flowbased post-net has witnessed the degradation of audio quality,proving that the post-net could refine the coarse-grained outputand generate spectrograms with increasing details.
4.4 Style Adaptation
[Table]
Following previous work [5], we further study the performance of ourmodel in style adaptation with different amounts of data.
We randomlysample utterances (each around 5 sec)from the out-of-domain datasets andfinetune the parameters in the multi-level style adaptor for additional 2000 steps.
The CMOS and
CSMOS evaluation results have been illustrated in Table 4: GenerSpeech performs better with theincreasing amount of adaptation data, demonstrating the ability of proposed GenerSpeech towardsadaptive style transfer in the few-shot data setting. The detailed fine-tuning setting has been includedin Appendix E.
Conclusion
In this work, we proposed GenerSpeech, a text-to-speech model towards high-fidelity zero-shotstyle transfer of out-of-domain custom voices.
To achieve better model generalization, we designseveral techniques to learn the style-agnostic and style-specific variations in speech separately:1)
GenerSpeech utilized a multi-level style adaptor to model and transfer various style attributes,including the speaker and emotion global characteristics, and the fine-grained utterance, phoneme,and word-level prosodic representations; 2) The Mix-Style layer normalization was further adoptedto eliminate the style information in linguistic representations for improving the generalization of
GenerSpeech.
Experimental results demonstrated that GenerSpeech achieved new state-of-the-artzero-shot style transfer results for OOD text-to-speech synthesis.
Our further extension study toadaptation further showed that GenerSpeech performed robustly in the few-shot data setting.
Forfuture work, we will further verify the effectiveness of GenerSpeech on more general scenarios suchas multilingual generalization.
We will release our code and pre-trained models after publication, andwe envisage that our work could serve as a basis for future text-to-speech synthesis studies.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Yes
b ) Did you describe the limitations of your work?
[Yes] See Section {+
+}
(c) Did you discuss any potential negative societal impacts of your work
Yes ] See
Section H in the supplementary materials(
d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4.1.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes] We report confidence intervals of subjective metric results
in Appendix C.
in the supplementary materials.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.1.3.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.1.1.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[Yes] See Appendix C in the supplementary
materials.