A GENERATIVE ADVERSARIAL NETWORK FOR STYLE MODELING IN A TEXT - TO - SPEECH SYSTEM
Anonymous authors
Paper under double-blind review
A BSTRACT
The modeling of style when synthesizing natural human speech from text has been the focus of signiﬁcant attention.
Some state-of-the-art approaches train an encoder-decoder network on paired text and audio samples (cid:104) x txt , x aud (cid:105) by encouraging its output to reconstruct x aud .
The synthesized audio waveform is expected to contain the verbal content of x txt and the auditory style of x aud .
Unfortunately, modeling style in TTS is somewhat under-determined and training models with a reconstruction loss alone is insufﬁcient to disentangle content and style from other factors of variation.
In this work, we introduce an end-to-end TTS model that offers enhanced content-style disentanglement ability and controllability.
We achieve this by combining a pairwise training procedure, an adversarial game, and a collaborative game into one training scheme.
The adversarial game concentrates the true data distribution, and the collaborative game minimizes the distance between real samples and generated samples in both the original space and the latent space.
As a result, our model delivers a highly controllable generator, and a disentangled representation.
Beneﬁting from the separate modeling of style and content, our model can generate human ﬁdelity speech that satisﬁes the desired style conditions.
Our model achieves start-of-the-art results across multiple tasks, including style transfer (content and style swapping), emotion modeling, and identity transfer (ﬁtting a new speaker’s voice).
I NTRODUCTION
In the past few years, we have seen exciting developments in Text-To-Speech (TTS) systems with the use of deep neural networks.
These models typically aim to synthesize human-like speech from text in an end-to-end fashion.
Ideally, synthesized speech should convey the given text content in an appropriate auditory style which we refer to as style modeling .
Modeling style is of particular importance for many practical applications such as intelligent conversational agents and assistants.
Yet, this is an incredibly challenging task because the same text can map to different speaking styles, making the problem somewhat under-determined.
To this end, the recently proposed Tacotron-based approaches (Wang et al., 2018; Skerry-Ryan et al., 2018a) use a piece of reference speech audio to specify the expected style.
Given a (cid:104) text, audio (cid:105) input, they assume two independent latent variables: c that encodes content from text, and s that encodes style from the reference audio, where c and s are produced by a text encoder and a style encoder, respectively.
A new audio waveform can be consequently generated by a decoder conditioned on c and s , i.e. p ( x | c, s ) .
Thus, it is straight forward to train the model that minimizes the log-likelihood by a reconstruction loss.
However, this method makes it challenging for s to exclusively encode style as no constraints are placed on the disentanglement of the style components from the content components within the reference audio.
It makes the model easy to simply memorize all the information (i.e. both style and content components) from the paired audio sample. In this case, the style embedding tends to be neglected by the decoder, and the style encoder cannot be optimized easily.
To help address some of the limitations of the prior work, we build a model, which provides enhanced controllability and disentanglement ability .
Rather than only training on single paired textaudio sample (the text and audio are aligned with each other), i.e. ( x txt , x aud )
→ ˜ x , we adopt a pairwise training procedure to enforce our model to correctly map from one text to two audio references ( x txt , x + aud , x − aud ), i.e. ( x txt , x + aud )
→ ˜ x + ; ( x txt , x − aud )
→ ˜ x − .
Where x + aud is paired with x txt , and x − aud is unpaired (randomly sampled).
Training the model involves solving an adversarial game and a collaborative game.
The adversarial game concentrates the true joint data distribution p ( x, c ) by using a conditional GAN loss .
The collaborative game is built to minimize the distance of generated samples from the real samples in both original space and latent space.
Speciﬁcally, we introduce two additional losses, the reconstruction loss and the style loss .
The style loss is produced by drawing inspiration from image style transfer, which can be used to give explicit style constraints.
During training, the the generator and discriminator combat each other to match a joint distribution.
While at the same time, they also collaborate with each other in order to minimize the distance of the expected sample and the synthesized sample in both original space and hidden space.
As a result, our model delivers a highly controllable generator, and a disentangled representation.
B ACKGROUND
TTS can be formulated as a cross-domain mapping problem, i.e. given the source domain Src (text) and target domain Trg (audio), we want to learn a mapping F : Src → Trg such that the distribution of F ( Src ) matches the distribution Trg .
When modeling style in TTS, F shall be conditioned on a style variable (where the style could speciﬁed in many forms, such as a reference audio waveform or a label).
Given a text and audio pair ( x txt , x aud ) , the goal under this formulation is to synthesize a new audio waveform that contains the textual content speciﬁed by x txt and the auditory style speciﬁed by x aud .
This is solved by training on paired data ( x txt and x aud are aligned with each other) with a reconstruction loss in some previous works, e.g. Tacotron-based systems (Wang et al., 2018; Skerry-Ryan et al., 2018a).
Here, we describe this solution via a conditional probabilistic model admitting two independent sources of variation: a content variable c 1: T with T words speciﬁed by text x txt , and a style variable s given by the reference audio x aud .
Given ( x txt , x aud ) , we can sample:
[Equation]
, and [Equation] p θ ( x | c 1: T , s ) is a likelihood function described by a decoder network, Dec .
We deﬁne deterministic encoders Enc c that maps x txt to their corresponding content components, and Enc s that parameterizes the approximate posterior q φ ( s | x aud ) .
It is natural to consider the conditional likelihood to be written as x ∼ p θ ( x | Enc c ( x txt ) , Enc s ( x aud )) , and the training objective could be maximizing the log-likelihood:
[Equation]
Ideally, the learned mapping F should be bijective , i.e. there should be a one-to-one correspondence between input conditions and the output audio waveform.
However, we argue that training only on paired data with maximum likelihood objective is insufﬁcient to learn a bijective mapping.
Unlike x txt that purely contains content components, x aud consists of both style components s and other factors z , such as verbal content that matches with x txt .
Therefore, the model needs to be able to disentangle s from z .
Otherwise, in the case of training on paired data by maximum likelihood objective, the model could simply learn to copy the waveform information from x aud to the output and ignore s .
When given the same x txt but different x aud to such a model, it may still map sample to the same ˜ x , i.e., F is not bijective any more.
In the following sections, we introduce a way to prevent this degenerate issue.
A PPROACH
Our proposed approach makes use of pair-wise training and GAN architecture.
The training pipeline is illustrated in Figure 1.
This training procedure could also be considered as the swapping of style components.
After swapping, the content components of both observations will remain the same, while the sources of style will change, and be aligned with x + aud and x − aud , respectively.
The two games involved in training the proposed model are described in detail in Sec.
3.1 and Sec. 3.2.
3. A DVERSARIAL GAME
Because ˜ x − need not be aligned with the content factors of x − aud , we cannot enforce the reconstruction of x − aud .
Instead, we enforce that both ˜ x + and ˜ x − are assigned high probabilities of belonging to the target domain by means of generative adversarial networks Goodfellow et al.
Speciﬁcally we use a conditional GAN (Mirza & Osindero, 2014) to model a joint distribution of audio and content (i e. D ( x, c ) ) which provides a stronger constraint by enforcing the decision to be always conditioned on the content variable c .
We deﬁne the min-max adversarial game:
[Equation]
Unlike the traditional binary setting of D , we make D a ternary classiﬁer with D ( · ) i representing the probability of x being either “fake from paired input” ( D ( · ) 1 ), “fake from unpaired input” D ( · ) 2 , or “real audio sample” D ( · ) 3 .
This ternary setting makes the discriminator more powerful because it has to distinguish subtle differences between samples generated from paired and unpaired input.
A similar setting has also been used in cross-domain image generation by (Taigman et al., 2016).
Our generator consists of both the encoders Enc c and Enc s and the decoder Dec .
The discriminator network is used only during training.
3. C OLLABORATIVE GAME
Although the adversarial game theoretically drives p θ ( x, c, s ) toward the true data distribution, we ﬁnd it to be insufﬁcient to ﬁnd the desired distribution, as there is little supervision from the observation what s should represent.
Especially for x − aud , the lack of a pairwise relationship makes it difﬁcult to ﬁnd the correct correspondence.
As a result, G might generate high-ﬁdelity samples ˜ x − with incorrect s , and D will still accept it as long as its style is different from ˜ x + .
Therefore, we impose explicit constraints on the generated samples with a style loss and a reconstruction loss.
Style Loss.
In the computer vision literature, Gatys et al.
(2016) have shown that the artistic style of an image can be effectively represented by the gram matrix of features maps produced by a convolutional neural network.
The gram matrix captures patch-level appearance statistics, such as texture, in a location-invariant manner.
It is thus natural to expect that a gram matrix of feature maps computed from a mel-spectrogram captures local statistics of an audio signal in the frequency-time domain, representing low-level characteristics of sound, e.g. loudness, stress, speed, pitch, etc.
This has motivated some initial attempts in the auditory style modeling (Dmitry Ulyanov, 2016; Barry & Kim, 2018).
Let X and ˜ X be the feature maps of the mel-spectrograms from the reference and the synthesized audio samples, respectively.
We compute the gram matrices W and G as the inner-product of vectorized feature maps X and ˜ X , respectively:
[Equation]
The style loss L sty is the L 2 distance between G and W over all pairs of ﬁlters i and j :
[Equation]
where N f is the number of ﬁlters.
To produce the features maps, most existing approaches in image style transfer use the VGG-19 (Simonyan & Zisserman, 2014) pretrained on ImageNet (Russakovsky et al., 2015).
However, mel-spectrograms look quite different from the natural images of the ImageNet, making the VGG-19 unsuitable for our work.
We found that a simple four-layer
CNN with random weights, denoted by R , perform just well for our purpose; similar ﬁndings have been reported recently by Ulyanov et al.
(2017), showing that the structure of a CNN is sufﬁcient to capture a great deal of low-level image statistics.
Reconstruction Loss.
As ˜ x
+ is expected to be the same as x + aud , we include Eq. 2 and encourage the reconstruction in the original mel-spectrogram space:
[Equation]
(8) where f ( · ) and g ( · ) denote the deterministic encoding function of Enc c and Enc s , respectively.
We also encourage reconstruction in the latent space by introducing an inference network C : x aud → z c which approximates the posterior p ( z c | x aud ) as z c ∼ p c ( z c | x aud )
=
C ( x aud ) .
C reduces to an Nway classiﬁer if z c is categorical.
In our model, C and Enc s share all layers and there is one ﬁnal fully-connected layer to output parameters for the conditional distribution p c ( z c | x aud ) .
To train p c ( z c | x aud ) , we deﬁne a collaboration game in the latent space:
[Equation]
Minimizing the ﬁrst term w.r.t.
C guides C toward the true posterior p ( z c | x aud ) , while minimizing the second term w.r.t.
G enhances G with extra controllability, i.e. it minimizes the chance that G could generate samples that would otherwise be falsely predicted by C .
Note that we also minimize the second term w.r.t.
C , which proves effective during training that uses synthetic samples to augment the predictive power of C .
In summary, minimizing both L sty and L rec can be seen as a collaborative game between players C , R and G that drives p θ ( x | c, s ) to match p ( x | c, s ) , and p c ( z c | x ) to match the posterior p ( z c | x ) , the reconstruction loss is thus given by:
[Equation]
3. I MPLEMENTATION DETAILS
We train our model with a combination of the GAN loss, style loss, and reconstruction loss, the overall loss function is given by:
[Equation]
(11) where α
= 0 .
1 , β = 10 .
Our model is built upon Tacotron (Wang et al., 2017b) that predicts melspectrograms directly from character sequences.
The predicted mel-spectrogram can be synthesized directly to speech using either the WaveNet vocoder (van den Oord et al., 2016) or the GrifﬁnLim method (Grifﬁn & Lim, 1984).
In our experiments, we use the Grifﬁn-Lim for fast waveform generation.
For Enc c , we use the same text encoder architecture of Skerry-Ryan et al.
(2018b).
The style encoder Enc s is a combination of reference encoder and style token layers proposed in
Wang et al.
We combine the encoded s and c as in Tacotron, i.e. for a content sequence c of length L , we concatenate the style embedding with each state of the text embeddings.
The inference network C takes as input a style embedding and processes it through a fully-connected layers followed by batch normalization and Relu is added on top of each convolution layer.
The output is mapped to an N-way classiﬁcation layer.
R is a 2-D fully-convolution neural network with four layers, with ﬁlter dimensions of 32, 32, 64, 64, respectively.
The discriminator D has the similar architecture with the reference encoder, except that a style and content fusion unit is added before it, such that the predicted spectrogram and content information are jointly fed into the network.
Finally, a fully-connected layer followed by ReLU maps the output to a 3-way classiﬁcation layer.
Note that, instead of using the character level content embedding c 1: T , here we adopt the global sentence embedding , which is the average of hidden unit activation over the sequence.
A detailed diagram can be seen in Appendix 5.
We train our model for at least 200k steps with a minibatch size of 32 using the Adam optimizer.
During training, R is ﬁxed weights.
For testing, C , R and D are not needed, and we just need to send a (cid:104) text, audio (cid:105) pairs into the model (unpaired audios are not needed in the testing stage), which is shown in Figure 1.
R ELATED W ORK
Text-To-Speech (TTS) :
Recently, rapid progress has been achieved in TTS with end-to-end trained neural networks, e.g., WaveNet (van den Oord et al., 2016), DeepVoice (Arik et al., 2017),
VoiceLoop (Taigman et al., 2018), Char2Wav
(Jose Sotelo, 2017), (Saito et al., 2017; Saruwatari, 2018; Yang et al., 2017) and Tacotron (Skerry-Ryan et al., 2018b).
Consequently, modeling style in TTS has become a subject of extensive study.
DeepVoice2 (Gibiansky et al., 2017) and DeepVoice3 Ping et al.
(2018) learn one or more lookup tables that store information about different speaker identities.
However, they are limited to synthesizing voices of speaker identities seen during training.
Unlike DeepVoice2 and DeepVoice3, Nachmani et al.
(2018), which is based on VoiceLoop, can ﬁt unseen speakers’ voice at testing time.
There is also a collection of approaches that are based on Tacotron, e.g., Tacotron-prosody (Wang et al., 2017a), prosody-Tacotron (SkerryRyan et al., 2018a) and GST (Wang et al., 2018).
prosody-Tacotron uses an encoder to compute a style embedding from a reference audio waveform, where the embedding provides style information that is not provided by the text.
The Global-Style-Token (GST) extends prosody-Tacotron by adding a new attention layer that captures a wide range of acoustic styles.
Domain mapping by GANs : Recently, GANs have shown promising results in various domain mapping problems.
Cycle-GAN (Zhu et al., 2017a) and UNIT (Liu et al., 2017) perform imageto-image translation by adding a cycle-consistency loss to the learning objective of GANs.
Further research has extended this to cross-domain translation.
StackGAN (Zhang et al., 2016) generates images from text, and DA-GAN (Ma et al., 2018) operates across different domains, e.g., object transﬁguration, human face to cartoon face, skeleton to natural object.
Another line of work performs one-sided domain mapping without using the cycle consistency loss, e.g.,(Taigman et al., 2016).
(Zhang et al., 2017) and (Hirokazu Kameoka, 2018) are mapping within text and speech domains.
Moving beyond one-to-one domain mapping, Bicycle GAN Zhu et al.
(2017b) maps samples from one domain to multiple target domains.
Our work can also be considered as a one-sided crossdomain mapping that does not require cycle consistency, which makes the training more practical.
We also follow the concept of Bicycle GAN that promotes a one-to-many mapping.
To the best of our knowledge, ours is the ﬁrst to formulate TTS as a cross domain mapping problem using GANs.
Style transfer : The recent success in image style transfer Gatys et al.
(2016) has motivated approaches that model the acoustic style of sound using spectrogram.
For example, Dmitry Ulyanov (2016) uses a simple 1-D convolutional layer with a ReLU to compute feature maps and then obtain style features by computing the gram matrix.
Barry & Kim (2018) followed the same concept and adopted two different audio representations, the mel-spectrogram and the constant Q transform spectrogram.
Inspired by this, in this work, we adopt the image style transfer concept to impose explicit style constraints on audio mel-spectrogram.
E XPERIMENTS
We evaluate our model from three perspectives: content vs. style disentanglement ability (Sec.
5.1), effectiveness of style modeling (Sec. 5.2), and controllability (Sec. 5.3).
We use two datasets in our experiments: EMT-4 : An in-house dataset of 22,377 American English audio-text samples, with a total of 24 hours.
All the audio samples are read by a single speaker, in four emotion categories: happy, sad, angry and neutral.
For each text sample, there is only one audio sample labeled with one of the four emotion styles.
VCTK :
A publicly available, multi-speaker dataset containing [Table] recordings of clean speech from 109 speakers, with a total of 44 hours.
As the raw audio clips have different speciﬁcations, we preprocess them by downsampling the audio to 24kHz and trimming leading and trailing silence, reducing the median duration from 3.3 seconds to 1.8 seconds.
We compare our method with three state-of-the-art approaches: prosody-Tacotron (Skerry-Ryan et al., 2018a) is similar to our model but trained on the reconstruction loss only.
The style embeddings are obtained from the reference encoder directly.
GST (Wang et al., 2018) incorporates the
Global Style Tokens to prosody-Tacotron.
DeepVoice2 (Gibiansky et al., 2017) learns a look-up table capturing embeddings for different speaker identity.
As DeepVoice2 is particularly designed for multi-speaker modeling, so comparisons with DeepVoice2 is only performed on VCTK.
We also conduct extensive ablation study to investigate the importance of different loss terms.
5. C ONTENT VS . STYLE DISENTANGLEMENT ABILITY
Reconstruction error of style embeddings : If the style encoder Enc s has successfully disentangled style from other factors of variation in the audio input, we expect the style embedding s to contain very little information about the content of the audio input.
Therefore, we should expect poor performance when we try to reconstruct the audio sample purely from s .
This motivates us to evaluate our model with the task of reconstructing audio samples from style embeddings.
To this end, we train an autoencoder, where the encoder has the same architecture as Enc s and the decoder has six deconvolutional layers, with each layer having batch normalization and ReLU activation.
To set the baseline, we ﬁrst train the autoencoder from scratch using only the L 2 reconstruction loss; this results in the reconstruction error of 0.12.
Next, we use precomputed style embeddings from different approaches and train only the decoder network using the reconstruction loss.
We report the results under the columns “Recon. error” in Table 1.
prosody-Tacotron achieves the lowest reconstruction error, suggesting that the approach has the weakest ability to disentangle style from other factors in audio input.
GST shows improvement over prosody-Tacotron, which demonstrates the effectiveness of the style token layer that acts as an information bottleneck from audio input to style embeddings.
DeepVoice2 performs much better than both prosody-Tacotron and GST on the VCTK dataset.
This shows the model particularly works well on modeling speaker identities.
Compared to the three state-of-the-art approaches, our model performs the best on both datasets.
We also evaluate the importance of different loss terms in our model.
We can see that the adversarial loss L adv provides a signiﬁcant improvement over the baseline models, which suggests the effectiveness of our adversarial loss and pairwise training.
When we add the style loss we get further improvements.
We also remove the adversarial loss and add the style and reconstruction losses to the baseline GST; this produces even worse results.
It is because, when training only on paired data with GST’s encoder-decoder network, the reconstruction loss already imposes very strong supervision.
In this case, additional constraints might on the contrary impaired the performance due to the risk of over ﬁtting.
While as our model is adversarially trained, the GAN loss regularizes the model, thus under this case, the style loss and reconstruction loss can help optimizing in a better way.
Content and style swapping : We follow the qualitative evaluation method from Mathieu et al.
and generate audio samples by combining content and style embeddings from different ground-truth pairs of text and audio.
Speciﬁcally, we randomly sample four (text, audio) pairs from the EMT-4 dataset, one for each emotion category, and create 16 permutations of text (content) and audio (style).
The samples located on the diagonal line can be considered as results of parallel style transfer , i.e. the given reference audio sample is aligned with the text.
While for the other ones, they are results by unparalleled style transfer .
Each column has the same content and varies in style, each row shares the same style, but with different content.
By hearing the samples, we can see that the results are comparable for both parallel transfer and unparalleled transfer, which means the content and style components are disentangled.
Even when transferring on two samples which are separated by a large distance in the latent space, e.g. sad to happy (row 2, line 4), the style can also be correctly mapped (compare this to sad to sad (row 4, line 4)).
We have uploaded the synthesized audio clips to our demo page; the readers are strongly encouraged to listen to the samples.
5. E FFECTIVENESS OF STYLE MODELING
Speaker style modeling : We further evaluate the effectiveness of our approach on modeling styles by means of speaker veriﬁcation.
Speciﬁcally, we compare our style embeddings with the i-vector representation used in modern speaker veriﬁcation systems (Kinnunen et al., 2017), on the speaker classiﬁcation task.
We report the results under the columns “Embeddings” in Table 2.
We can see that, despite the fact that the i-vectors are speciﬁcally designed for speaker classiﬁcation, our model can still achieve comparable results, which suggests that our model can produce generic feature representation for various auditory styles including speaker identity.
Visualization of style embedding : Figure 2 shows the t-SNE projection (van der Maaten & Hinton, 2008) of style embeddings from (a) the EMT-4 dataset and (b) the VCTK dataset.
To create the plots, we randomly sampled 1,000 instances from each dataset: (a) 250 instances from each of the four emotion categories, and (b) 125 instances from 9 speakers (3 male and 5 female).
We can see that the projections show clear boundaries between different style (emotion and speaker) categories.
Interestingly, “sad” is far from the other three emotion categories; we believe that this is because sad speech usually have much lower pitch compared to other emotion categories.
“neutral” is projected to the middle, which has roughly the same distance with other emotion categories.
Also, we can see that there is a clear boundary between male samples and female samples.
5. C ONTROLLABILITY
A good TTS system should allow users to control both content and style of the output.
We consider two factors that affect the controllability of a TTS model: the ﬁdelity , i.e. the synthesized speech should contain the desired content in a clearly audible form, and the naturalness , i.e. the synthesized speech should contain the desired style.
WER of synthesized samples : To validate the ﬁdelity, we assess the performance of synthesized samples in a speech recognition task.
We use a pre-trained ASR model based on WaveNet (van den
Oord et al., 2016) to compute Word Error Rate (WER) for the samples synthesized by each model.
Results are shown in Table 1.
Our model performs comparably with, and sometimes even better than, the state-of-the-art approaches.
Note that WER measures only the correctness of verbal content, not its auditory style.
The results suggests that all the methods we have compared perform reasonably well in controlling the verbal content in TTS.
When trained with more constraints on GST, i.e.
GST+
L sty + L rec , the performance gets worse.
We suspect that this is because the autoencoder training procedure used in GST already gives strong supervision to the decoder.
Thus, when added [Table] with more constraints, the model has overﬁtted to the training data.
Our model does not have such problem because of the unpaired samples used during training, which act as strong regularizer.
Classiﬁcation accuracy on synthesized samples : As the style we are modeling are all categorical, we evaluate the synthesized samples by a classiﬁcation task.
We ﬁrst train two classiﬁers on each dataset, which have 98 % and 83 % accuracy for EMT-4 and VCTK, respectively.
We selectsamples synthesized from test set on EMT-4.
To assess on VCTK, we test samples from both seen and unseen speakers, where ‘seen speakers’ mean the speakers are part of the training set, while the reference audio are selected from the test set.
‘unseen speaker’ means the speakers have never be seen during training, which means the model is asked to ﬁt a new speaker’s voice on testing stage.
The results are shown in Table 2.
As we can see, on the EMT-4 dataset, our model performs the better than prosody-Tacotron and GST.
When tested on the seen data on VCTK, DeepVoice performs the best, but it fails to generalize to unseen speakers.
Our model performs well in both cases.
Style transfer : To qualitatively evaluate Our model, we conduct style transfer.
In this experiment, we want to compare our model against GST in how well they model varied styles in EMT-4.
We randomly selected 15 sentences, where 10 of the sentences are from the test set, and 5 of them are picked on web (out of the dataset).
To perform style transfer, we select four different reference audios from ‘happy’, ‘angry’, ‘sad’ and ‘neutral’, all of the reference audio samples are unseen during training.
Each sentence is paired with these four reference audio samples for synthesizing, which will produce 60 new audio samples in total.
The results can be found in our demo page.
We also compare our model against GST on the task of unparalleled transfer at scale.
Speciﬁcally, we follow the same setting in Wang et al.
(2018) to ran side-by-side subjective study on 7-point ratings (-3 to 3) from “model A is the closest to the reference style” to “model B is the closest to the reference style”, where model B was ours.
We recruited seven participants.
Each listened to all 60 permutation of content and rated each set of audio style (emotions) comparing the result of our model versus the prosody-Tacotron model.
They rated each pair of audio outputs on the 7-point scale.
We performed a single-sample T-Test on the resulting ratings averaged across all participants.
µ> 0 means our model was judged as closer to the reference.
Overall the emotion samples the participants rated our model as signiﬁcantly closer to the reference ( µ =0.872, p (cid:28) 0.001).
For each of the styles individually our model was consistently rated as signiﬁcantly closer to the reference (neutral: µ =0.295, p=0.01, happy: µ =0.905, p (cid:28) 0.001, sad: µ =1.646, p (cid:28) 0.001, angry: µ =0.641, p (cid:28) 0.001).
These results provide further evidence that our model can synthesize speech with the correct content and distinct auditory styles that are closer to the reference than the state-of-the-art comparison.
C ONCLUSION
We propose an end-to-end conditional generative model for TTS style modeling.
The proposed model is built upon Tacotron, with an enhanced content-style disentanglement ability and controllability.
The proposed pairwise training approach that involves a adversarial game and a collaborative game together, result in a highly controllable generator with disentangled representations.
Beneﬁting from the separate modeling of content c and style s , our model can synthesize high ﬁdelity speech signals with the correct content and realistic style, resulting in natural human-like speech.
We demonstrated our approach on two TTS datasets with different auditory styles (emotion and speaker identity), and show that our approach establishes state-of-the-art quantitative and qualitative performance on a variety of tasks.
For future research, an important direction can be training on unpaired data under an unsupervised setting.
In this way, the requirements for a lot of work on aligning text and audios can be much released.
A PPENDEX A N ETWORK A RCHITECTURES
We provide architecture details of our model.
Figure 3 shows both the content and style encoder networks ( Enc c and Enc s , respectively) as well as the inference network ( C ).
Figure 4 shows the decoder network ( Dec ) and Figure 5 shows the discriminator network ( D ).
We further provide network parameter settings in the captions of each ﬁgure token layers, 10 global style tokens (GSTs) are randomly initialized.
The reference embedding is used as a query for a multi-head attention unit.
A learned linear weight is then output from the multi-head attention unit, and the style embedding is computed as a weighted sum.
The inference network ( C ) shares the same architecture and parameters with Enc s , except that a new N-way classiﬁer (which consists of a fully connected layer followed by Softmas) is added on top.
B A TTENTION PLOTS
In this section, we show attention plots of our model and a baseline model, comparing the robustness of these models for different lengths of the reference audio.