S EMI -S UPERVISED G ENERATIVE M ODELING FOR C ONTROLLABLE S PEECH S YNTHESIS
A BSTRACT
We present a novel generative model that combines state-of-the-art neural textto-speech (TTS) with semi-supervised probabilistic latent variable models.
By providing partial supervision to some of the latent variables, we are able to force them to take on consistent and interpretable purposes, which previously hasn’t been possible with purely unsupervised TTS models.
We demonstrate that our model is able to reliably discover and control important but rarely labelled attributes of speech, such as affect and speaking rate, with as little as 30 minutes supervision.
Even at such low supervision levels we do not observe a degradation of synthesis quality compared to a state-of-the-art baseline.
Audio samples are available on the web 1 .
I NTRODUCTION
The ability to reliably control high level attributes of speech, such as emotional expression (affect) or speaking rate, is often desirable in speech synthesis applications.
Achieving this control however is made difﬁcult by the necessity of acquiring a large quantity of high quality labels.
In this paper we show that semi-supervised latent variable models can take us a signiﬁcant step closer towards solving this problem
Combining state-of-the-art neural text-to-speech (TTS) systems with probabilistic latent variable models provides a natural framework for discovering aspects of speech that are rarely labelled or even difﬁcult to describe.
Both inferring the latent prosody and generating samples with sufﬁcient variety requires reasoning about uncertainty and is thus a natural ﬁt for deep generative models.
There has been recent progress in applying stochastic gradient variational Bayes (SGVB) (Kingma & Welling, 2013; Rezende et al., 2014) to training probabilistic neural TTS models.
Battenberg et al.
(2019) and Hsu et al.
(2018) have shown that it is possible to use latent variable models to discover features such as speaking style, speaking rate, arousal, gender and even the quality of the recording environment.
However, these models are formally non-identiﬁable (Hyv¨arinen & Pajunen, 1999; Locatello et al., 2019) and this implies that repeated training runs will not reliably discover the same latent attributes.
Even if they did, a lengthy human post-processing stage is necessary to identify what the model has learned on any given training run.
In order to be of practical use for control, it is not enough for the models to discover latent attributes, they need to do so reliably and in a way that is robust to random initialization and to changes in the model.
We demonstrate that the addition of even modest amounts of supervision can be sufﬁcient to achieve this reliability.
By augmenting state-of-the art neural TTS with semi-supervised deep generative models within the
VAE framework (Kingma et al., 2014; Narayanaswamy et al., 2017), we show that it is possible to not only discover latent attributes of speech but to do so in a reliable and controllable manner.
In particular we are able to achieve reliable control over affect, speaking rate and F0 variation (F0 is the [Figure] fundamental frequency of oscillation of the vocal folds).
Further, we provide demonstrations that it is possible to transfer controllability to speakers for whom we have no labels.
Our core contributions are:
• To combine semi-supervised latent variable models with Neural TTS systems, producing a system that can reliably discover attributes of speech we wish to control.
• To demonstrate that as little as 30 minutes supervision can be sufﬁcient to improve prosody and allow control over speaking rate, fundamental frequency (F0) variation and affect, a problem of interest to the speech community for well over two decades (Schr ¨ oder, 2001).
• To imbue TTS models with control over affect, F0 and speaking rate whilst still maintaining prosodic variation when sampling.
G ENERATIVE M ODEL
Our generative model, shown in ﬁgures 1 and 2a, consists of an autoregressive distribution over a sequence of acoustic features, x 1 ...t , that are generated conditioned on a sequence of text, y 1 ...
k , and on two latent variables, z u and z s .
The latent variables can be discrete or continuous.
z s represents the variations in prosody that we seek to control and is semi-supervised.
z u is fully unobserved and represents latent variations in prosody (intonation, rhythm, stress) that we wish to model but not explicitly control.
Once trained, our model can be used to synthesize acoustic features from text.
Similar to Tacotron 2 (Shen et al., 2018), we then generate waveforms by training a second network such as WaveNet (van den Oord et al., 2016) or WaveRNN ( Kalchbrenner et al., 2018) to act as
a vocoder
In our case we use WaveRNN.
We parameterize our likelihood p ( x 1 ...t | y 1 ...
k , z u , z s , θ ) by a sequence-to-sequence neural network with attention (Shen et al., 2018; Graves, 2013; Bahdanau et al., 2014) that is shown schematically in ﬁgure 1.
Details largely follow Tacotron (Wang et al., 2017) and are given in appendix A.
At each time step we model a mel-spectrogram frame with a ﬁxed variance isotropic Laplace distribution whose mean is output by the neural network.
We condition each of the latent variables by concatenating the vectors z u and z s to the representation of the text-encoder, before the application of the attention mechanism.
In the case of continuous z we use a standard normal prior and in the case of discrete
z we use a uniform categorical prior with one-hot encoding.
[Figure]
2. S EMI -S UPERVISED T RAINING
Following Kingma et al.
(2014); Narayanaswamy et al.
[Equation]
we train our model via stochastic gradient variational Bayes (SGVB).
That is we approximately maximize the log-likelihood of our training data by maximizing a variational lower bound using stochastic gradient ascent.
Since we are training with semi-supervision we in fact need two lower bounds: one for the data points for which z s is observed; one for the case where z s is unobserved.
In our models the fully latent variable z u is always continuous but the semi-supervised latent z s can be continuous or discrete.
The conditional independence structure of our variational distributions is shown in ﬁgures 2b and 2c.
On supervised data, the per-datapoint bound is:
[Equation]
Where q ( z u | x, y, z s , φ ) is a parametric variational distribution introduced to approximately marginalize z u .
θ are the parameters of the generative model and φ are the parameters of the variational distributions.
The intractable integrals are approximated with reparameterized samples.
For the cases where z s is unobserved and discrete, the bound is:
[Equation]
and when z s is continuous we replace the sum above with an integral and again approximate with reparameterized samples.
The variational distributions are parameterized by a neural network that takes as input the text, spectrograms and other conditioning variables and outputs the parameters of the distribution.
The exact structure of this network is given in appendix A. We have implicitly assumed that q ( z u , z s | x, y, φ ) may be factorized as q ( z u , z s | x, y, φ ) =
q ( z u | x, y, z s , φ ) q
( z s | x, y, φ ) with shared parameters between these two distributions (see appendix A).
Optimizing the variational objective with respect to the parameters φ encourages the variational distributions to match the posterior of the generative model p ( z u , z s | x, y, θ ) .
Unlike previous work (Hsu et al., 2018), we do not assume that the posterior on the latents is independent of the text, as this dependence likely exists in the model due to explaining away.
That is to say that although the text and the latents are independent in our prior, observing the spectrogram correlates them in the posterior because they both explain variation in the spectrogram.
This has been shown to be signiﬁcant by Battenberg et al.
If we deﬁne:
[Equation]
then we can write the overall objective over both the supervised and unsupervised points succinctly as 2 :
[Equation]
where summation would again be replaced by integration for continuous z s and γ (shown in equation 4) is a weighting factor that pre-multiplies the loss for any supervised point.
This weighting was also used in previous work such as Narayanaswamy et al.
(2017), who showed it to be beneﬁcial at very low levels of supervision.
Writing the objective in this form allows an intuitive interpretation for the semi-supervised training procedure.
When supervision is provided, our objective function is evaluated at the observed value of z s .
When supervision is not provided, we evaluate the objective function for every possible value of z s and take a (potentially inﬁnite for continuous z s ) weighted average.
The weighting in the average is given by q ( z s | x, y, φ ) , which is simultaneously trained to approximate the posterior p ( z s | x, y, θ ) .
In other words, on unsupervised utterances, we evaluate our objective for each possible value of the latent attribute and weight by the (approximate) posterior probability that this value of the latent was responsible for generating the utterance.
As q ( z s | x, y, φ ) is trained to approximate p ( z s | x, y, θ ) we can expect it to become a reasonable classiﬁer/regressor for the semi-supervised latent attribute as the model improves.
For example when z s represents an affect label, p ( z s | x, y, θ ) is the posterior probability, of the model, over affect given text and speech.
By taking the most likely posterior class, this distribution can be used as an affect classiﬁer.
However, this variational distribution is only trained on unsupervised training points and so does not beneﬁt directly from the supervised data.
To overcome this problem we follow Kingma et al.
and add a classiﬁcation loss to our objective.
The overall objective becomes:
[Equation]
Where α is a hyperparameter which adjusts the contribution of this term.
D ATA
We have used a proprietary high quality labeled data-set of 40 English speakers.
The training set consists of 72,405 utterances with durations of at most 5 seconds (45 hours).
The validation and test sets each contain 745 utterances or roughly 30 minutes of data.
We vary the amount of supervision in the experiments below.
We also experimented with transferring controllability to a fully unlabeled data-set of audiobook recordings by Catherine Byers (the speaker from the 2013 Blizzard
Challenge), which exhibits high variation in affect and prosody and to other speakers who were less expressive.
We strongly encourage the reader to listen to the synthesized samples on our demo page 3 .
In this work we chose to focus on learning to control affect with a discrete representation, as well as speaking rate and F0 variation with a continuous representation, as these are challenging aspects of prosody to control.
Our method could be applied to other factors without modiﬁcation.
3. A FFECT C ONTROL
The best way to represent emotion is an actively researched area and many models of affect exist.
In this work we chose to follow the circumplex model of emotion (Russell, 1980) which posits that most affective states can be represented in a 2 dimensional plane with one axis representing arousal and the other axis representing valence.
Arousal measures the level of excitement or energy and valence measures positivity or negativity.
Figure 3, shows a chart of emotions plotted in the arousalvalence plane where we can see that, for example, high arousal and high valence corresponds to joy or happiness whereas high arousal and low valence might correspond to anger or frustration.
Our data-set was recorded under studio conditions with trained voice actors who were prompted to read dialogues in one of three valences:-2, -1, +2 and two arousal values:-2 (low), +2 (high).
This was achieved by prompting the actors to read dialogues in either a happy, sad or angry voice at two levels of arousal.
This results in 6 possible affective states which we chose to model as discrete and use as our supervision labels.
3. S PEAKING R ATE AND F0 V ARIATION C ONTROL
In order to demonstrate that we can control continuous attributes we also created approximate realvalued labels for speaking rate and arousal for all of our data.
We generate the approximate speaking rate as number of syllables per second in each utterance.
F0, also known as the fundamental frequency, measures the frequency of vibration of the vocal folds during voiced sounds.
Variation in
F0 is highly correlated with arousal and roughly measures how expressive an utterance is.
To create approximate arousal labels we extracted the F0 contour from each of our utterances, using the YIN algorithm (De Cheveign ´ e & Kawahara, 2002), and measured its standard deviation.
We then performed a whitening transform on these two approximate labels in order to match it to our standard normal prior.
These artiﬁcial labels would of course be cheap to obtain for the entire data-set and would not justify the use of semi-supervision in real applications.
But, our objective here is to evaluate/demonstrate the efﬁcacy of semi-supervision rather than to speciﬁcally control a particular attribute.
We have chosen syllable rate and F0 standard deviation, because they both correspond to subjectively distinct variations of interest, and they are more easily quantiﬁable than affect and so provide strong evidence of controllability.
For the continuous latents we are not only able to interpolate speaking-rates and
variations but also to extrapolate outside of our training data.
We provide examples on our demo page of samples with signiﬁcantly greater/lower speed and F0 variation than typically observed in natural speech.
E XPERIMENTS AND R ESULTS
To evaluate the efﬁcacy of semi-supervised latent variable models for controllable TTS we trained the model described in section 2 on the above data-sets at varying levels of supervision as well as for [Table] varying settings of the hyperparameters: α which controls the supervision loss and γ , which over emphasizes supervised training
We found that a value of was optimal for the discrete and the which corresponds to simply optimizing the
ELBO experiment we report the for best and to experiments with no over-weighting of the supervised points.
All models were trained using the
ADAM optimizer with learning rate of 10 − 3 and run for 000 training steps with a batch size of distributed across 32 Google Cloud TPU chips.
All models were implemented using tensorﬂow 1 (Abadi et al., 2016).
Assessing the degree of control is challenging as interpreting affect is subjective.
We used two objective metrics of control as well as subjective evaluation from human raters and a third objective metric of overall quality.
For affect, the ﬁrst objective metric we introduced was the test-set accuracy of a 6-class affect classiﬁer trained on the ground truth training data and applied to generated samples from the model (shown in ﬁgure 4a).
The classiﬁer is a convolutional neural network whose structure mirrors the posterior network q ( z s | x, y, φ ) and its exact architecture is given in appendix A.
We also provide subjective metrics of controllability, shown in table 1.
For speaking rate control, we measure the mean syllable rate error on a held out test-set.
The syllable rate error is calculated as the absolute difference in syllable rate between the desired syllable rate and that measured from the synthesized sample.
We calculate an analogous error rate for F0 variation.
Whilst the two metrics above measure controllability they don’t tell us if this comes at the expense of a degradation in synthesis quality.
To probe quality we use three further metrics.
The ﬁrst was MelCepstral-Distortion-Dynamic-Time-Warping (MCD-DTW)
(Kubichek, 1993) on a held out test-set, shown in ﬁgure 4d.
MCD-DTW is a measure of the difference between the ground-truth spectrogram and the synthesized mel spectrogram that is known to correlate well with human perception (Kubichek, 1993).
The second metric of quality was crowd sourced mean-opinion-scores (MOS).
The third metric of quality is speech recognition word error rate (WER) and character error rate (CER) on audio samples.
The MOS and speech recognition results are summarized in table 2.
To demonstrate that semi-supervision by including unlabelled data is beneﬁcial, we also provide
MOS and speech recognition errors for fully supervised subsets of the data in table 2.
These show that at least close to 5 hours of data is required to train a reasonable quality TTS model, far above the 30 minutes supervision needed to control prosodic aspects of speech.
We provide further details of all of these metrics in the appendix B, and sample spectrograms are provided in appendix C.
D ISCUSSION
The classiﬁcation accuracy (see ﬁgure 4a), subjective metrics (see table 1) and error-rate results (see ﬁgure 4b-4c) provide a clear demonstration that using semi-supervised latent variables, we are able to achieve control of both continuous and discrete attributes of speech.
There is not a signiﬁcant degradation in the overall quality and this is evidenced by the mean opinion scores which are above the baseline, Tacotron, and also speech recognition errors (see table 2).
We also include a baseline of our Tacotron model augmented only by the unsupervised latent z s , to aid comparison. [Figure] [Figure]
affect MOS 4.52 ± 0. 4.09 ± 0. [Figure]
.24 ±
[Figure] [Figure] 4.28 ± [Figure]
[Figure]
4.16 ± 0.
[Figure]
4.17 ± 0.09 WER
[Figure]
6.09 CER 2.
[Table]
[Table] The MCD-DTW scores for F0 variation and affect are improved at all levels of supervision (ﬁgure 4d). Whilst the MCD-DTW is degraded for speaking rate, this is likely a misleading metric when targeting changes in timing as the dynamic-time-warping component of MCD-DTW changes exactly the aspect we wish to control.
For speaking rate the combination of MOS and samples is a better indication of the overall quality.
We are able to reduce the supervision level to levels as low as 1% or 30 minutes and still have a signiﬁcant degree of control.
We show on our demo page 4 that even at 3 minutes of supervision we can still achieve control of speaking rate and that we are able to extrapolate outside the range of values seen during training.
On the affect data our classiﬁcation accuracy doesn’t degrade signiﬁcantly until we reach 10% (300 minutes) supervision and remains signiﬁcantly above chance down to levels as low as 1% (30 minutes), see ﬁgure 4a and table 1.
Obtaining 30 minutes of supervised data is likely within reach of most teams constructing TTS systems.
Unlike previous work on generative modelling for control (Hsu et al., 2018; Wang et al., 2018), we do not require a post-processing stage to determine what our latent variables control and we can pre-determine what aspects we wish to control through choice of data.
By separating our latent variables into those that are partially supervised and those which are fully unsupervised we retain the ability to model other latent aspects of prosody; this means that we can still draw samples of varying prosody whilst holding constant the affect, speaking rate or F0 variation.
We observe the greatest degree of affect control, as measured by classiﬁer accuracy, when α =and γ = 100 .
This means that to achieve the highest controllability we needed to 1) provide extra information to our approximate posterior q ( z s | x, y, φ ) and 2) to over-represent the supervised data at low levels of supervision.
Although both of these hyperparameters have been used in the literature before (Narayanaswamy et al., 2017; Kingma et al., 2014) and shown to be either beneﬁcial or necessary, they aren’t strictly required by our probabilistic framework and so it is worth considering why they are needed.
There are three potential sources of error in any generative model trained with
SGVB: the model itself may be mis-speciﬁed such that the true data-generating distribution is not in the model class, the parametric family chosen to approximate the posterior may be overly restrictive and ﬁnally the optimization landscape may contain undesirable local minima.
These problems have afﬂicted previous work with deep latent variable models trained with SGVB, resulting in models that don’t use their latent variables unless trained with complex annealing schedules (Bowman et al., 2015).
In our case we believe that the necessity to set α and γ arises from a combination of model mis-speciﬁcation and local minima.
If α is set to 0, then at the start of training q
( z s | x, y, φ ) is trained only to approximate p ( z s | x, y, θ 0 ) , which is randomly initialized.
We found empirically that in our discrete-latent experiments this resulted in q ( z s | x, y, φ ) collapsing early in training to a point-mass on a single class.
Having ended up in this undesirable local minimum the posterior distribution never recovered, despite this being an obviously poor approximation to the model posterior later in training.
The addition of the classiﬁcation loss and supervision weighting were sufﬁcient to overcome this collapse and allow q to continue to model the posterior.
The optimization landscape is strongly affected by the relative size of the conditional likelihood and
KL terms in our objective.
These are in turn strongly affected by our choice of conditional independence assumptions and output-distributions.
Thus, a natural direction for further work is to increase the expressivity of the conditional likelihood p ( x | y, z s , z u , φ ) to reduce model mis-speciﬁcation.
This could be done by learning the variance of the Laplace-distribution we currently use or by parameterizing more expressive output distributions that do not assume conditional independence across spectrogram channels.
We conjecture that with more expressive output distributions, it may be possible to reduce the need for the α and γ terms in the objective.
In this work we chose to use quite simple unconditional diagonal Gaussian priors, as our primary goal was to demonstrate the practicality of semi-supervision.
Another natural extension would be to use conditional-priors p ( z | y ) and to use more expressive priors such as mixtures as was done in Hsu et al.
5. R ELATED W ORK
There has been enormous recent progress in neural TTS with numerous novel models proposed in recent years to synthesize speech directly from characters or phonemes (Shen et al., 2018;
Arik et al., 2017; Gibiansky et al., 2017; Ping et al., 2017; Vasquez & Lewis, 2019; Taigman et al., 2017).
Differentiating factors between these models include the degree of parallelism, with some models using Transformer based architectures (Ren et al., 2019), the choice of conditional independence assumptions made (Vasquez & Lewis, 2019) or the number of separately trained components (Gibiansky et al., 2017).
Our work here is largely orthogonal to the exact structure of the conditional likelihood p ( x | y, z s , z u ) and could be combined with all of the above methods.
Much of the recent research focus has been on modeling latent aspects of prosody.
Early attempts include Global Style Tokens (Wang et al., 2018) which attempted to learn a trainable set of styleembeddings.
Wang et al.
(2018) condition the Tacotron decoder on a linear combination of embedding vectors whose weights during training are predicted from the ground-truth spectrogram.
They were able to achieve prosodic control but there is no straightforward way to sample utterances of varying prosody.
More recently, attempts have also been made to combine probabilistic latent variable models trained using SGVB (Akuzawa et al., 2018; Wan et al., 2019).
These models use a fully unsupervised and non-identiﬁable approach, which makes it difﬁcult to disentangle or interpret their latent variables for control.
Hsu et al.
(2018) attempt to overcome this problem by using a Gaussian mixture as the latent prior and so perform clustering in the latent space.
Battenberg et al.
(2019) introduce a hierarchical latent variable model to separate the modelling of style from prosody.
However, all of these methods are fully unsupervised and this results in latents that can be hard to interpret or require complex post-processing.
The work most similar to ours is Wu et al.
(2019) which also attempts to achieve affect control using semi-supervision with a heuristic approach based on Global Style Tokens (Wang et al., 2018).
Wu et al.
(2019) add a cross-entropy objective to the weightings of the style-tokens that encourages them to be one-hot on points with supervision.
Similar to our method, they are able to achieve control over affect but unlike our method they do not have a principled probabilistic interpretation nor the ability to simultaneously model aspects of prosody other than emotion.
The result is that their method is not able to draw samples of varying prosody for the same utterance with ﬁxed emotion.
Furthermore, whilst our method can be applied to both continuous and discrete controllable factors, its not clear how to extend the style-token based approach to handle continuous latent factors.
In the wider generative modelling literature, the combination of semi-supervision and deep latent variable models was ﬁrst introduced in Kingma et al.
(2014) who focus on using unlabelled data to improve classiﬁcation accuracy.
The potential to use the same technique for controllable generation was recognized by Narayanaswamy et al.
(2017) who also provided demonstrations on image synthesis tasks.
Since that work, interest in learning disentangled latent variables has grown but generally pursued alternate directions such as re-weighting the ELBO (Higgins et al., 2017), augmenting the objective to encourage factorization (Kim & Mnih, 2018) or using adversarial training (Mathieu et al., 2016).
The ability to transfer controllability to speakers for whom we do not have supervision is referred to as domain transfer and our model bears similarities to that introduced by
Ilse et al.
but they use a mixture in their latent space more similar to Hsu et al.
5. E THICAL C ONSIDERATIONS
As with many advances in speech synthesis, progress in controllability raises the prospect that bad actors may misuse the technology either for misinformation or to commit fraud.
Improvements in data efﬁciency and realism increase these risks and, when publishing, a consideration has to be made as to whether the beneﬁts of the developments outweigh the risks.
It is the opinion of the authors in this case that, since the focus of this work is on improved prosody, with potential beneﬁts to human-computer interfaces, the beneﬁts likely outweigh the risks.
We nonetheless urge the research community to take seriously the potential for misuse both of this work and broader advances in TTS.
C ONCLUSION
We have shown that the combination of semi-supervised latent variable models with neural TTS presents a practical and principled path towards building speech synthesizers we can control.
Unlike previous fully unsupervised methods, we are able to consistently and reliably learn to control predetermined aspects of prosody.
Our method can be applied to any latent attribute of speech for which a modest amount of labelling can be obtained, whether it be continuous or discrete.
In our experiments we found that 30 minutes of supervision was sufﬁcient, a volume of data that is within the reach of most research teams.
We are able to learn to control subtle characteristics of speech such as affect and for continuous attributes we have provided demonstrations of extrapolation to ranges never seen during training, and to speakers with no supervision.
Augmenting existing stateof-the-art TTS systems with latent variables does not degrade synthesis quality and we evidence this with crowd sourced mean opinion scores.
Unlike similar heuristic methods, our probabilistic formulation, allows us to draw samples of varying prosody whilst holding constant some attribute we wish to control.
B E VALUATION
Mel spectrograms The mel spectrograms the model predicts are computed from 24 kHz audio using a frame size of 50 ms, a hop size of 12.5 ms, an FFT size of 2048, and a Hann window.
From the FFT energies, we compute 80 mel bins distributed between 80 Hz and 12 kHz.
MCD-DTW
To compute mel cepstral distortion (MCD) (Kubichek, 1993), we use the same mel spectrogram parameters described above and take the discrete-cosine-transform to compute the ﬁrst 13 MFCCs (not including the 0th coefﬁcient).
The MCD between two frames is the Euclidean distance between their MFCC vectors.
Then we use the dynamic time warping (DTW) algorithm
(Velichko & Zagoruyko, 1970) (with a warp penalty of 1.0) to ﬁnd an alignment between two spectrograms that produces the minimum MCD cost (including the total warp penalty).
We report the average per-frame MCD-DTW.
Affect classiﬁer
The affect classiﬁer has a very similar structure to the variational posterior.
The input to the classiﬁer starts from the mel spectrogram x and passes it through a stack of 6 convolutional layers, each using ReLU non-linearities, 3x3 ﬁlters, 2x2 stride, and batch normalization.
The 6 layers have 32, 32, 64, 64, 128, and 128 ﬁlters, respectively.
The output of this convolution stack is fed into a unidirectional LSTM with 128 units.
The ﬁnal output of the LSTM is then passed through a softmax non-linearity to get logits over the training classes.
We use the same data splits described in section 3 to train and evaluate the classiﬁer.
The classiﬁer is tuned on the validation set achieving 84.33% classiﬁcation accuracy, generalizing well to the test set with 83.94% accuracy.
Mean opinion scores We use a human rating service similar to Amazon’s Mechanical Turk, with a large pool of English speakers to collect MOS evaluations.
The MOS template is shown in ﬁgure 5.
A human rater is presented with a single speech sample and is asked to rate perceived naturalness on a scale of 1 to 5, where 1 is “Bad” and 5 is “Excellent”.
We have selected the utterances of one male, and one female speaker in our test set, totalling 371 utterances to evaluate.
For each sample, we collect 1 rating, and no rater is used for more than 6 items in a single evaluation set.
In total, 270 unique raters completed the 6 evaluation sets presented in table 2.
Since raters are randomly selected for each set, some raters have assessed multiple methods.
Across the 6 evaluation sets, the average and median of total number of ratings per rater was 8.24, and 6, respectively.
To analyze the data from these subjective tests, we average the scores and compute 95% conﬁdence intervals.
Natural human speech is typically rated around 4.5.
Samples used for MOS from our model were drawn using the mean of z u , whilst sampling z s .
., happy in the ﬁgure).
Both utterances are generated with the same text.
To evaluate the control over valence, we present baseline (i.e, no control) against utterances generated in speciﬁc valence category (angry, happy and sad).
To evaluate the control over arousal, we present samples generated at low arousal against samples generated at high arousal, and ask the rater to choose the utterance that is more vocally aroused.
We use mean of z u to generate all the samples.
C S AMPLE S PECTROGRAMS
Controlling affect Table 5 shows the effect of varying the valence and arousal latent variable on the spectrogram and F0 track.
We can see that a low valence for sadness corresponds to the ﬂattest
F0 track, and high arousal manifests in higher F0 values and variations.
Controlling speaking rate and pitch variations Table 6 shows the effect of varying speaking rate, and F0 variation control variables on a sample spectrogram and F0 track.
When controlling speaking rate (ﬁrst column), the duration reduces as we increase the input speaking rate control, while the F0 variation remains stable.
When controlling the F0 variation (second column), the pitch dynamic range increases, while the duration remains constant, which demonstrates controllablity and also some degree of disentanglement.
D R EPRODUCING RESULTS ON L IBRI TTS PUBLIC DATASET
To verify the reproduciblity of our results on a public dataset, we trained models to control speaking rate and F0 variation on clean subset of LibriTTS dataset (Zen et al., 2019).
We only use the utterances below 5 seconds, which is 62 hours of data.
We have done no tuning on this dataset, and directly used the hyperparameters we used for our internal dataset.
Figures 7a and 7b show the
[Figure]
Angry
[Figure]
Happy
[Figure] [Figure]
a)
[Figure]
-rate error as a
[Figure]
errors of producing the desired speaking rate, and F0 standard deviation, which generally go down as function of supervision level, with exception of 10% supervision for controlling F0 variation 5 .
Given, this is a lower quality dataset, with many more speakers and with much smaller data per speaker and also the fact that we have done zero hyperparameter tuning on this dataset, this result look very encouraging.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]