S TYLE E QUALIZATION : U NSUPERVISED L EARNING OF C ONTROLLABLE G ENERATIVE S EQUENCE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Controllable generative sequence models with the capability to extract and replicate the style of speciﬁc examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks.
However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference.
In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models.
By introducing a style transformation module that we call style equalization , we enable training using different content and style samples and thereby mitigate the traininginference mismatch.
To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets.
Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data.
Moreover, the proposed method enables style interpolation between sequences and generates novel styles.
I NTRODUCTION
The goal of controllable generative sequence models is to generate sequences containing target content in a target style.
With the capability to select speaker voices, multi-speaker text-to-speech models have been successfully adopted in many voice assistants (Gibiansky et al., 2017; Ping et al., 2018;
Hayashi et al., 2020).
Many applications, however, require style controllability beyond selecting speaker voices.
For example, to perfectly reconstruct a speech example, we need to replicate not only the speaker’s voice characteristics but also all aspects of style about the sample, including but not limited to prosody, intonation dynamics, background noise, echo, and microphone response appeared in the given sample.
To analyze failures or biases of a downstream recognizer, we need a style representation that models the entire style distribution, beyond speaker identity.
In these applications, style represents all information (except the content) to exactly reconstruct a sample, as illustrated in
Fig.
1a.
Notice that to represent the time-dependent information of a sample, style is itself a sequence and changes over time, instead of a ﬁxed vector.
Moreover, even when the same speaker utters the same content, the resulting audios can contain different styles.
To capture the large variation, the style representation should be learned in an unsupervised manner from a reference sample, rather than using a few human-annotated attributes.
Our goal is to learn a controllable generative sequence model that controls its style with a reference example ( e.g ., an existing audio) and controls the content with a content sequence ( e.g ., text), as shown in
Fig.
1b.
Our training dataset X is composed of (cid:8) ( x i , c i ) (cid:9) i =1 ,...,n , where x i = (cid:2) x i 1 . . .
x iT i | x it ∈ R d (cid:3) is the i-th sample and c i = (cid:2)
c
i 1 . . .
c iN i | c ij ∈ R m (cid:3) is the corresponding content sequence.
Note that in general, x i and c i have different lengths, i.e ., T i (cid:54)
=
N i , and we do not have the alignment between them.
For example, in text-to-speech synthesis, x i is the mel-spectrogram of an audio sample,
c i is the corresponding phonemes of the spoken words, and we do not have the mapping between the phonemes and mel-spectrogram.
We also do not have any style supervision, including speaker or attribute labels, nor any grouping of the data based on style.
While the unsupervised setting requires only the essential information ( i e ., samples and their content) it makes learning a controllable generative sequence model very challenging.
The main challenge is the mismatch between the inputs used during training and inference.
As shown in Fig.
1c, during inference we pair arbitrary content A and reference sample B as inputs.
However, due to the lack of ground truth containing content A and in the style of B, during training we pair content A and sample
A.
In other words we train the model under the parallel setting where the reference style input contains the input content, but we use the model in the non-parallel setting (where the reference style contains a different content than the target content) during inference.
Due to the training-inference mismatch, a well-performing model during training may perform poorly during inference.
If a generative model learns to utilize the content information in the style example, during inference the generative model will generate wrong content.
This phenomenon is called content leakage (Hu et al., 2020).
In an extreme case, a model can learn to copy the reference sample to the output; despite its perfect training loss, it is useless because it always generates wrong content in practice.
This paper proposes a simple but effective technique to deal with the training-inference mismatch when we learn controllable auto-regressive models in an unsupervised manner.
As shown in Fig.
1d, we train the model under the non-parallel setting, i.e ., we pair arbitrary content A with an arbitrary sample B from the training dataset.
Instead of directly using sample B as style (in which case we have no ground truth), we jointly learn a style transformation function, which estimates the style difference between A and B and transforms the style of sample B to the style of A.
The generative model then takes content A and the transformation output (that contains the style of A) to reconstruct
sample A.
The proposed method enables us to use sample A as the ground truth while learning in the non-parallel setting—the intended usage during inference.
Additionally, our method provides
NEW a systematic way to interpolate between the style of two samples by scaling the estimated style difference between two reference samples.
We call the method style equalization .
Note that for style equalization to work, the style transformation and difference estimator need to be carefully designed, such that no content information from content A can be transferred through sample
B. We defer the discussion to Sec. 4.
The proposed method is general and can be applied to different sequence signals.
We apply the proposed method on two signal domains, speech and online-handwriting, and evaluate the performance carefully via quantitative evaluation (by computing content error rates) and conducting qualitative user studies.
Experimental results show that our method outperforms various unsupervised controllable
FIX sequence generative models, even when they have additional style supervision like speaker labels.
On
LibriTTS, style equalization achieves close style replication (3.5 real oracle vs. 3.5 proposed in style
FIX opinion score) and content reproduction errors (6.6% real oracle vs. 9.5% proposed) to real samples.
R ELATED WORK
Controllable generative sequence models are not new in the literature; however, the majority of these methods require style supervision, whereas the paper develops an unsupervised-style method.
Tableprovides an overview of the related works.
NEW
Unsupervised-style sequence models.
Unsupervised methods extract style information directly from samples, i.e ., without any style labels or pretrained style embeddings.
Existing unsupervised methods train models under the parallel setting, as shown in Fig. 1c.
To prevent content leakage, most existing methods introduce a bottleneck on the capacity of the style encoder by representing style as a single (time-invariant) vector and limiting its dimension (Wang et al., 2018;
Hsu et al., 2018;
Hu et al., 2020; Ma et al., 2018).
Wang et al.
(2018) propose Global Style Token (GST),
which
NEW represents a style vector as a linear combination of a learned dictionary (called style tokens) shared across the dataset.
The number of style tokens (the implicit dimension of the style vector) is carefully controlled to prevent content leakage.
As we will see in Sec. 3, the bottleneck not only reduces the amount of content information contained in the style vector but also sacriﬁces style information.
Alternative loss formulations have also been proposed to limit content information contained in the style representation.
Hu et al.
(2020) minimize the mutual information between the style vector and the content sequence but requires a pretrained content encoder and adversarial learning, which makes training their model difﬁcult.
Hsu et al.
(2018) approximate the posterior distribution of the style vector using a mixture of Gaussian distributions with a small number of mixtures.
Ma et al.
(2018) utilize a discriminator conditioned on both the generated output and the content (similar to a content
NEW recognizer).
Akuzawa et al.
(2018) anneal the Kullback-Leibler divergence to control the amount of
NEW information contained in style.
Henter et al.
(2018) utilize phoneme segmentation (McAuliffe et al., NEW 2017) to avoid learning the alignment between content c and output x .
Priming is a technique that is introduced to control the style of auto-regressive generative sequence models (Graves, 2013; Aksan et al., 2018).
Since the hidden state of a Recurrent Neural Network (RNN) contain all information about current generation, including style, we can initialize the RNN by pre-rolling the reference sample through the RNN.
Utilizing priming requires the content of the reference style.
For example, Aksan et al.
(2018) learn a character recognizer and use it during inference.
Moreover, since the hidden state contains residual content from the reference example, it
NEW often generates unexpected artifacts at the beginning of the sequence, as will be seen in Sec.
Supervised-style sequence methods.
Many existing controllable generative models require style supervision, either directly by passing attribute labels as inputs or implicitly by grouping training data with their attribute labels.
In the following, we brieﬂy introduce various supervised controllable sequence models.
While using style supervision avoids training-inference mismatch, it limits the style control over a few sparsely-deﬁned attribute classes.
For instance, given a speech audio, we can recognize the spoken texts, the accent, or even the speaker, but provided solely with these attribute labels, it is impossible to exactly reconstruct the original speech audio.
The sparsely-deﬁned attributes are insufﬁcient to capture the entire style information.
User identiﬁcations or their embeddings have been used to learn multi-speaker text-to-speech models
(Jia et al., 2018; Gibiansky et al., 2017; Kameoka et al., 2020; Donahue et al., 2020; Chen et al., 2021; Dhariwal et al., 2020; Valle et al., 2020;
Kim et al., 2020; Hayashi et al., 2020; Sun et al., 2020),voice conversion models (Qian et al., 2019) and handwriting models (Kotani et al., 2020;
NEW
Bhunia et al., 2021; Kang et al., 2020; Davis et al., 2020).
In addition to user identiﬁcations, predeﬁned features like pitch, phoneme duration, loudness, and timbre have also been used by existing methods (Ren et al., 2020;
Qian et al., 2020; Dhariwal et al., 2020; Valle et al., 2020).
Instead of using speaker labels as input, Kameoka et al.
; Kaneko and Kameoka (2018); Kaneko et al.
(2019a;b) group training samples by their speaker labels and apply adversarial learning to learn voice conversion models that change speaker voices while keeping the content of the input.
Image methods.
Controllable generative models have also been developed for images (Härkönen et al., 2020; Esser et al., 2019;
Singh et al., 2019; Lample et al., 2017;
Karras et al., 2020; Brock et al., 2019; Collins et al., 2020;
Shen et al., 2020; Esser et al., 2020; Goetschalckx et al., 2019; Pavllo et al., 2020; Zhang et al., 2018), which control the object class, pose, lighting, etc ., of an image.
Many image style transform methods have also been developed (Isola et al., 2017; Zhu et al., 2017; Gatys et al., 2016).
However, there is a fundamental difference between image and sequence problems.
In image generative models, we do not need to learn the content-output alignment.
The content is usually deﬁned globally as an image class or as pixel labels, e.g ., segmentation map.
In contrast, our content is given as text, the output is mel-spectrogram of a waveform, and the content and output have different lengths.
To utilize the input content sequence, generative sequence models need to align the content and the output sequences and translate text to the output signal modality.
The complication exacerbates the training-inference mismatch for sequence methods, since copying the style input is easier than utilizing the input content.
C ONTROLLABLE GENERATIVE SEQUENCE MODELS
We focus on learning controllable auto-regressive generative models, p ( x t | z t , x 1 ..t − 1 , c ) , where x = [ x 1 , . . .
, x T ] is the output sequence, c is the content sequence, and
z =
[ z 1 , . .
, z T | z t ∈ R (cid:96) ] is the reference style information.
Note that, in our model, style is also represented as a sequence that changes over time.
Under the style-unsupervised setting, we are given a dataset X =
( x i , c i ) , i ∈ { 1 . . .
n }} that contains the ground-truth output sequence x i and the corresponding content c i , but we do not have style supervision on z .
Therefore, we treat z t as a latent variable with a learnable prior distribution p ( z t | x 1 ..t − 1 , c ) and optimize the log-likelihood of x conditioned on c , E ( x , c ) log p ( x | c ) .
Speciﬁcally, we maximize a variational lower bound of the likelihood as VRNN (Chung et al., 2015):
NEW
E
[Equation] [Equation]
) where D KL represents the Kullback-Leibler (KL)-divergence.
In eq.
(1), we use the chain rule to expand p ( x | c ) into p ( x 1 | c ) p ( x 2 | x 1 , c ) · · · p ( x T | x 1 ..T − 1 , c ) , introduce the variational approximation q ( z t | x , c ) of the posterior distribution p ( z t | x , c ) for all t , and apply Jensen’s inequality.
Note that since q is a variation approximation of the posterior distribution, it can be conditioned on any variable.
Fig.
2a shows an overview of the network used in the paper — the input content c is processed by the content attention, the style encoder (shown in green) models q ( z t | x r , c ) , and the decoder (shown in gray) models p ( x t | z t , x 1 ..t − 1 , c ) using output from the style encoder and the content attention.
Note that during inference, x r is the reference example that we replicate the style of, and it generally contains an unrelated content, i.e ., c r (cid:54)
=
c .
During training, the ground-truth sample x is used as the reference style to optimize eq.
(1), i.e ., x r = x and c r = c , which is different from inference.
Therefore a generative model can learn to copy the style input to the output (and ignore the content input), leading to incorrect generation during inference.
This phenomenon can be remedied by limiting the capacity of the style encoder, e.g ., by decreasing the dimensionality of the style representation.
However, to achieve the lower bound of eq.
(1) and hence a higher generation quality, we need the style encoder to contain enough capacity such that q ( z t | x , c ) ≡ p ( z t | x , c ) for all t .
In this paper, we provide an alternative training procedure that bypasses this trade-off — we use a high-capacity style encoder (shown in Fig. 2b) and prevent content leakage.
S TYLE EQUALIZATION
Let ( x , c ) and ( x (cid:48) , c (cid:48) ) be two samples from the training set.
To match the inference setting, we should train the model using c as content and x (cid:48) as style input.
However, neither x nor x (cid:48) can be used as the ground-truth output sequence.
If we use x as the ground-truth output sequence but x (cid:48) as style input, the generative model will learn to ignore the style encoder since x (cid:48) contains unrelated style information.
In other words, the variational approximation q ( z | x (cid:48) , c ) is a poor approximation to the true posterior p ( z | x , c ) .
Alternatively, if we use x (cid:48) as the ground-truth output sequence, the content given in c will be ignored.
We introduce a learnable style transformation function M ( x (cid:48) , δ ) that we use to transform the style of x (cid:48) by the amount speciﬁed by a vector δ ∈ R k .
Instead of directly using x (cid:48)
as the style input, we ﬁrst estimate the style difference vector between x and x (cid:48) with a learnable function φ , i.e .
δ = φ ( x (cid:48) , x ) ; we then transform x (cid:48) with M ( x (cid:48) , δ ) .
By jointly optimizing M , φ , and the rest of the generative model using eq.
(1), the model learns to transfer style information from x through M to maximize the log-likelihood of the ground-truth x .
In other words, we approximate the posterior distribution p ( z | x , c ) with q ( z |M ( x (cid:48) , φ ( x (cid:48) , x )) , c ) , which is a better approximation than q ( z | x (cid:48) , c ) .
We call this method style equalization .
Note that, for style equalization to be successful, M◦ φ should not transfer any content-related information ( e.g ., copy the entire sequence) from x but only its style information so that the decoder will utilize the transferred style and will rely on provided content input to generate the output.
Therefore the design of M is critical.
Design of M and φ .
An important observation we use in the design of M and φ is that content information ( i.e ., sequence of phonemes or characters) is strongly time-dependent whereas the style can be reasonably well approximated by a time-independent representation ( e.g ., voice characteristics of a speaker and microphone response, etc .).
By designing the style difference estimator φ such that no sequence information is stored in the difference vector δ , we can satisfy that content-related information is not leaked, but the function can still transfer the time-independent style information.
We achieve this using convolutional ﬁlters and average pooling over the time dimension.
As shown in Fig.
2c, to estimate the style difference vector between two sequences x and x (cid:48) , we ﬁrst compute their style features f and f (cid:48) using a convolutional network.
Note that f and f (cid:48) are s -dimensional feature sequences with different lengths.
We deﬁne
[Equation]
and
[Equation]
where avg represents taking mean across time, A ∈ R k × s is a learnable linear transform, and f (cid:48)
+
A (cid:62) φ ( x (cid:48) , x ) means that the vector A (cid:62) φ ( x (cid:48) , x ) ∈ R s is added to each time step of f (cid:48) .
Intuitively, the design assumes that the style information lies on a k -dimensional subspace, and we equalize the style between x (cid:48) and x by minimizing their differences in the subspace.
It also satisﬁes the identity property — φ ( x r , x r ) = 0 and M ( x r , 0) =
f r — which allows us to treat style equalization as a training procedure and remove it from the model during inference (as shown in Fig. 2b).
Interpolation between two styles.
Once learned, M and φ can be used to manipulate style during inference.
Given two style references, x s and x t , we interpolate between them with
M ( x s , α φ ( x s , x t )) , where a scalar α ∈ R controls the interpolation.
By changing α , we traverse a one-dimensional manifold that starts from the original style (with α = 0 ) and ends at the target style (with α = 1 ).
Note that, unlike existing generative models that support style interpolation in post-processing, M and φ are trained to transform style by design.
E XPERIMENTS
To demonstrate the generality of the proposed method, we train and evaluate it on two different
NEW signals, speech and handwriting, with the same model architecture design.
In the following, we will introduce the model architecture used in the experiments, the baselines, the metrics, and the results.
More details are provided in the supplemental material.
5. M ODEL ARCHITECTURE
Our model is auto-regressive and composed of (i) a decoder that is modeling p ( x t |
z t , x 1 ..t − 1 , c ) , (ii) a content attention module, (iii) a style encoder that is modeling q ( z t |· , c ) , and (iv) a network that models the prior distribution of z t .
Fig.
2a shows an overview of the model.
The backbone of the model, namely the content attention and the decoder, uses a standard architecture that was proposed by Graves (2013) for handwriting synthesis and later extended to speech in variations of
Tacotron (Shen et al., 2018; Wang et al., 2018).
The variational approximation q ( z t |· ) and the prior distribution are modeled as multivariate Gaussian distributions with a diagonal covariance matrix.
Our style encoder is composed of a convolutional network and a multi-head attention layer, as shown in Fig. 2b.
The convolutional network extracts the style feature sequence f r from the reference style input x r .
We use multi-head attention to extract relevant style information at every time step from f r with the query computed from the hidden state of the LSTM, h t , which contains information about past generations, and the currently focused content a t .
Thus, our style representation is a time-varying sequence.
The intuition is that if the model utters a particular phoneme, it should be able to ﬁnd the information in the style reference and mimic it.
For style equalization, we insert M and φ into the style encoder, as shown in Fig.
2c.
Since style equalization is only able to transfer time-independent information, when we utilize this procedure, the network will not be able to learn time-dependent style.
To enable learning time-dependent style information during training, half of the batches, we use x (cid:48)
= x , which means that the difference vector δ = 0 , hence the decoder directly uses the ground-truth style information which contains time-dependent style information.
We analyze the effect of style attention and its ability to represent time-varying style information in Sec. 5.4.
5. S PEECH SYNTHESIS
We train and evaluate the proposed method on two multi-speaker speech datasets.
VCTK dataset
NEW (Yamagishi et al., 2019) contains 110 speakers and 44 hours of speech, and LibriTTS dataset (Zen et al., 2019) contains 2,311 speakers and 555 hours of speech in the training set.
Baselines.
We compare the proposed method with Global Style Tokens (GST-n) (Wang et al., 2018) with various numbers of tokens n. For the sake of completeness, we also compare with Tacotron 2 (Shen et al., 2018) (even though it does not have style control), Tacotron-S and GST-nS. Tacotron-S
NEW and GST-nS are Tacotron and GST-n with pretrained style embedding (Snyder et al., 2018) that was trained on the VoxCeleb dataset (Chung et al., 2018), respectively.
The VoxCeleb dataset contains 2 , 000 hours of speech from 7 , 000 speakers and has a large variation in recording conditions.
We use ESPnet-TTS (Hayashi et al., 2020), a widely used implementation of the baselines and follow their training recipe.
All the baselines are trained using the same dataset as the proposed model 2 .
They achieve similar performance as those listed in the original papers.
All the methods output 80 -dimensional mel-spectrograms with the sampling rate equal to 22,050 Hz and the window size equal to 1,024, which are converted to waveforms using a pretrained WaveGlow vocoder (Prenger et al., 2019).
The content input is represented as phonemes.
[Table]
[Table] [Table]
Metrics.
We measure the content generation errors as Word Error Rate (WER), using a pretrained speech recognition model, ESPnet (kamo naoyuki, 2021).
To evaluate the style replication accuracy of the methods, we use a speaker classiﬁcation network (Deng et al., 2019) and measure the style similarity between reference and output generations.
We report cos-sim , which is the average
NEW cosine similarity between reference and output generations, and avgRank , which is the average rank of the reference speaker out of all speakers based on their cosine similarities.
We also compute the style opinion score following the protocol used by Zhao et al.
To evaluate the style similarity between a generated output and a style reference, users were given pairs of reference and synthesized audio, and asked if “the two samples could have been produced by the same speaker in a similar environmental condition", and asked to score with “ 4 (Absolutely same)", “3 (Likely same)", “2 (Likely different)", “1 (Absolutely different)".
We synthesized 100 samples using each method with the same style example and target content.
A total of 15 users participated in the study, and we collected 630 responses in total.
We also provide an oracle (a pseudo upper-bound) where we select a different real speech sample from the same speaker from the dataset, and evaluate style similarity and content error.
This provides a good calibration for our evaluation metrics and opinion studies.
Results.
Table 1 shows the results on VCTK dataset.
Let us ﬁrst look at the parallel-text setting.
Without any style control, Tacotron achieves a low cosine similarity.
While GST and GST-nS improve cosine similarities, the proposed method achieves the highest similarity and lowest avgRank.
When comparing the WERs between the parallel and non-parallel settings, we can see the adversarial effect of content leakage — the models with a high-capacity style encoder (GST-64, GST-16S, and
GST-64S) produce a much higher word-error rate in the non-parallel setting than in the parallel setting, while the small capacity GST-16 is largely unaffected.
In comparison, the proposed method achieves a high cosine similarity and a similar word-error rate as the oracle, which demonstrates robustness to content leakage despite using the high-capacity style encoder.
Table 2 shows the results on LibriTTS dataset.
As can be seen from the results, the large variety ( e.g ., more accents, higher background noise, difﬁcult microphone effects, etc) makes it a more difﬁcult dataset than VCTK, as reﬂected by the high WERs of Tacotron.
The proposed method, on the other hand, is able to learn from the noisy data and mimic the wide variety of styles in the dataset, including the voice characteristics and the background noise.
In addition, since the training set contains more than 2 , 000 speakers, the proposed method learns to generalize and mimic the voices of unseen speakers in the validation set, as demonstrated by the low WER and high cosine similarity in the right-most column of Table 2.
Table 3 shows the results of style opinion score evaluation on the two datasets.
The proposed method achieves the highest score among the synthesizers.
Furthermore, it achieves similar scores to the oracle.
The result demonstrates the effectiveness of the proposed method in style replication.
The method also enables sampling styles from the prior distribution and interpolating between two styles.
Please see the supplemental material for these results.
5. O NLINE - HANDWRITING SYNTHESIS
Online-handwriting synthesis aims to generate sequences of pen movements on a writing surface.
A handwriting sample is represented as a sequence of ( x, y, p ) triplets, where x and y are the coordinates of the pen on the surface, and p is a binary variable indicating whether the pen touches the surface over time.
We apply the proposed method to a subset of a proprietary dataset collected for research and development.
The subset consists of 600k online handwriting samples that were written by 1, people in English, French, German, Italian, Spanish, and Portuguese.
Baselines and metrics.
We compare with the method proposed by Graves (2013) that uses priming for style encoding and an ablation of our model which uses the same style encoder but without style equalization.
Note that all these models use the same decoder and content attention.
Similar to speech, we measure content generation error as Character Error rate (CER) with a pretrained handwriting recognizer, and we conduct style-opinion-score study on 12 users and collected 320 responses.
We also provide an oracle where we select a different real handwriting sample from the same writer from the dataset and compute the metrics.
Results.
Fig.
3a shows synthesized handwriting from each of the methods on unseen style examples.
As can be seen, while Graves (2013) with priming can replicate the reference style, it outputs artifacts at the beginning, and the style replication is worse than the proposed method.
The model without style equalization produces high-quality replication in the parallel setting; however, it suffers severely from content leakage under the non-parallel setting and produces wrong content.
In comparison, the proposed method generates correct content and replicate the style.
We demonstrate the capability to interpolate between reference styles ( e.g ., between cursive and printed style) in Fig.
3b.
Our method also supports sampling styles from the learned prior distribution, as shown in Fig. 3c.
Fig.
3d shows the CERs and style opinion scores for unseen style references.
Due to the residual information in the hidden state, priming signiﬁcantly increases the CERs for Graves (2013).
Without style equalization, the model fails to synthesize legible handwriting in the nonparallel-text setting due to the content leakage caused by the high-capacity style encoder.
In comparison, by adding style equalization, we successfully reduce content leakage and replicates style, as demonstrated by the
CER and the style opinion score that are close to that of real handwriting samples.
5. A NALYSIS OF STYLE ATTENTION
Here we analyze how our model utilizes the time-dependent style information contained in the reference example x r by examining the attention weights of the style attention module.
As discussed in Sec. 4, while our style representation is a time-dependent sequence, style equalization can only transfer time-independent global style information from the style reference.
Therefore, when we apply
[Figure]
style equalization to all the samples during training, we learn a time-independent representation.
This can be seen from Fig. 4a where the attention weights become constant over time during inference.
We encourage our style encoder to learn the time-dependent aspects of style by applying style equalization only to 50% of the training samples.
As can be seen from the time-varying attention weights in Fig.
4b-d, this training procedure enables the style encoder to utilize style information more efﬁciently by focusing time instances with similar signal and context.
For example, when the style reference contains the target content, i.e .,
c r = c (Fig. 4b), the attention weights form a block-diagonal pattern, indicating the model focuses to the time instances of the style signal that match the current content and context.
In Fig.
4c, we show attention weights when c r (cid:54) =
c , where the weights are still well localized over time to gather pieces of time-dependent style information from matching signal in x r .
When we use style equalization to transform style during inference, Fig.
4d, the weights are more diffuse since it can only transfer global style.
The ﬂexibility to apply and remove style equalization (and hence the representation bottleneck) during training is one of the main differences compared to existing methods (Wang et al., 2018;
Hu et al., 2020; Hsu et al., 2018), which operate on ground-truth input, suffer from the training-inference mismatch, and thus require tight bottlenecks to the style representation.
In contrast, with style equalization, we are able to use high-dimensional style ( i.e ., large s and k ) and retain time-dependent style information.
C ONCLUSION
This paper proposes a simple but effective method, style equalization, to learn a generative sequence model where style and content can be controlled separately.
The generative model supports 1) accurate replication of styles from a single style reference, 2) interpolation between two reference styles, and 3) generating new reference styles.
Experiments on speech and handwriting domains show that the method obtains state-of-the-art synthesis results.
A B ROADER IMPACT
The technology we develop in the paper, like other artiﬁcial intelligence technologies, potentially has both positive and negative impacts to our society (Brundage et al., 2018).
One potential risk associated with all generative models is creating fake digital content.
If deployed irresponsibly, speech and handwriting synthesis could facilitate deceptive interactions, including mimicking a person’s voice or handwriting in automatic speared phishing attacks, gaining security access, or directing public opinions.
Examples of responsible deployments of the technology include (but not limited to):
• A system-level authentication every time the technology is used to register style or generate output.
• An encryption system to protect registered style.
• A watermarking system (for both speech and handwriting) such that the generations can be easily identiﬁed by human or detection systems (Hua et al., 2016).
Beyond system-level security measures technologies to identify fake digital content have also been rapidly developed (Chen et al., 2020; Lyu, 2020; Güera and Delp, 2018; Dolhansky et al., 2020; Alegre et al., 2013; Wu et al., 2017; Evans et al., 2013; Kinnunen et al., 2020).
Despite the potential negative societal impacts we believe that the technology will have a larger positive impact, such as enabling
new
accessibility
capabilities ( e.g .,
helping
mute people
speak
in their
voices
and paralyzed people
write in their handwriting) and better human-computer interaction ( e.g ., by improving downstream speech and handwriting recognizers).
B D ETAILS OF THE TRAINING PROCEDURE
Training with style equalization is straightforward — we jointly optimize all the model parameters by maximizing the log-likelihood lower bound as described in eq.
(1) of the main paper.
We found the following optional steps to be useful to improve training and the quality of the generation.
• During training, we found it helpful to add a small amount of Gaussian noise to the ground-truth x t − 1 that is fed back to the bottom LSTM.
Intuitively, we are simulating the noise caused by sampling the output distribution during inference at training time.
We found that it makes the inference more stable.
A similar method is proposed by Meng et al.
(2021) to learn auto-regressive models.
• To encourage the basis A that is used by φ in eq.
(2) in the main paper to capture a wide variety of styles, we maintain A as an orthonormal basis.
We normalize each column a i in A to be unit norm in the architecture and minimize | a (cid:62) i a j | 2 for all i and j (cid:54)
=
i .
The minimization is conducted by minimizing the trace of ( A (cid:62) A ) 2 , which is estimated efﬁciently using Hutchinson’s trace estimator (Hutchinson, 1989) with 100 random samples from a standard Normal distribution.
The estimated value is used as a regularization with loss weight equal to 1.
Overall, we optimize the following objective function
[Equation]
where θ represents all network parameters (except A ).
We use the reparameterization trick that is commonly used in variational autoencoders (Kingma and Welling, 2014) with one sample to estimate the inner expectation in eq.
and we use ADAM (Kingma and Ba, 2015) with β 1 = 0 .
9 , β 2 = 0 .
98 , and a learning rate schedule used by Vaswani et al.
(2017) with a warm-up period of 4 , 000 iterations to optimize the objective function.
The learning rate increases rapidly within the warm-up period to 10 − 4 and decreases slowly after then.
C M ODEL ARCHITECTURE DETAILS
In this section, we provide more details about the model architecture used in the paper.
Fig.
5 shows an overview of the model, and Table 4 lists the individual block formulations used in Fig.
We use the same architecture for both handwriting and speech synthesis, except for the hyper-parameters, which we list at the end of the section.
The model can be separated into a backbone and a style encoder.
The backbone is composed of a decoder, content attention, and an LSTM at the bottom.
It is a standard model architecture and has been used and extended in many works of handwriting and speech synthesis (Graves, 2013;
Shen et al., 2018; Wang et al., 2018; Hsu et al., 2018).
The bottom LSTM is one-layer, and it accumulates information from the past outputs x 1 ..t − 1 and the previously attended content a 1 , . .
, a t − 1 into its hidden state h t .
The content attention, which is proposed by Graves (2013), utilizes moving Gaussian windows to calculate attention weights for the content.
The focused content a t at time t is a linear combination of the elements in c based on the attention weights.
The decoder at the top is a two-layer LSTM that takes all available information, including h t , z t , and a t , and outputs the parameters of p ( x t | z t , x 1 ..t − 1 , c ) .
As mentioned in Sec.
B, we add a small Gaussian noise to x t − 1 that is passed to the bottom LSTM.
[Table]
Our style encoder is composed of a 4-layer convolutional network and a multi-head attention layer.
Given a style reference input x r , the convolutional network extracts the feature sequence f r .
We apply low-pass ﬁltering before every sub-sampling (Zhang, 2019) to avoid aliasing caused by sub-sampling in the convolutional network.
Given the past outputs h t and the currently focused content c t , we use multi-head attention to extract relevant information from f r .
The query vector is computed from h t and c t using a linear layer, and the key and the value vectors are the individual feature vectors contained in the sequence f r without positional encoding.
The intuition is that if the model plans to write a speciﬁc character or utter a speciﬁc word, it should ﬁnd the information in the style reference and mimic it.
The variational approximation q ( z t |· ) is a multivariate Gaussian distribution with a diagonal covariance matrix.
The prior distribution p ( z t | x 1 ..t − 1 , c ) is also modeled as a multivariate
Gaussian distribution with a diagonal covariance matrix, and we a two-layer feed-forward network to compute its means and standard deviations from h t and c t .
When the style equalization is used, M and φ are inserted into the style encoder, as shown in Fig. 5(b).
Now we summarize the hyper-parameters used for handwriting and speech.
Please also see Table 4.
• For handwriting, the dimension of all LSTMs are 512 .
The ﬁnal linear layer outputs a 122dimensional vector, which is used to parameterize the output distribution.
The output distribution includes a mixture of 20 bivariate Gaussian distributions that model the pen movement, a Bernoulli distribution for pen lifting, and a Bernoulli distribution for sequence stops.
The posterior and the prior Gaussian distributions are 256 -dimensional.
The convolutional network in the style encoder has four layers; all of them use kernel size 3, stride 2, and no padding.
Their feature dimensions are 3 → 32 → 64 → 128 → 256 .
We use dropout with a dropping rate equal to 0.1 after each nonlinearity in the convolutional network.
The multihead attention has 4 heads, the dimension of
the query, key, and value vectors are all 256.
The dimension of δ ( i.e ., k ) is 128 .
The input content c is represented as 200-dimensional one-hot vectors of the input characters.
The standard deviation of the Gaussian noise is 0 . 1 , and during inference, we reduce the standard deviation of the output distribution to 0 .
9 of the original one.
The model is trained for 100 epochs on a machine withA100 GPUs, and the training took 36 hours.
• For speech, all LSTMs have the same 2048 dimension.
The ﬁnal linear layer outputs a 484dimensional vector, which is used to parameterize the output distribution.
The output distribution includes a mixture of three 80-dimensional Gaussian distributions with diagonal covariance that models the mel-spectrogram and a Bernoulli distribution for sequence stops.
The convolutional network in the style encoder has four layers; all of them use kernel size 3, stride 2, no padding.
Their feature dimensions are 80 → 256 → 384 → 512 → 512 .
We use dropout with a dropping rate equal to 0.1 after each nonlinearity in the convolutional network.
The multihead attention has 4 heads, the dimension of the query, key, and value vectors are all 256.
The input sentence is represented as phonemes, which contain 148 symbols.
We follow the preprocessing used by Shen et al.
(2018) for phoneme and mel-spectrogram extraction.
We also follow Shen et al.
and use a bidirectional LSTM and a convolutional network to encode the dependencies between phonemes.
The architecture is the same as that used by Shen et al.
The posterior and prior Gaussian distributions are 512 -dimensional, and the dimension of δ ( i.e ., k ) is 64 for VCTK dataset and 192 for LibriTTS dataset.
The standard deviation of the added noise is 0 . 2 , and during inference, we reduce the standard deviation of the output distribution to 0 .
74 of the original one.
The VCTK model is trained for 70 epochs on a machine with 8 A100 GPUs, and the training took
12 hours; the LibriTTS model is trained for 25 epochs on a machine with 8 A
GPUs, and the training took 3 days.
D S TYLE CLASSIFIER NETWORK
As we mentioned in Sec. 5.2, we train a speaker classiﬁer using the objective function proposed by Deng et al.
Using the features extracted by the speaker classiﬁer, we measure the style similarity between two waveforms using the cosine similarity between the features.
The style classiﬁer comprises a convolutional network, an LSTM, and a linear layer that transforms the last hidden state into the feature that we use to compute the cosine similarity.
The input to the convolutional network is the 80-dimensional phoneme, which is extracted using the same procedure as the one used by
Shen et al.
The convolutional network has four layers; all of them use kernel size 3, stride 2, valid padding, and swish non-linearity
(Ramachandran et al., 2018).
Their feature dimensions are 80 → 256 → 384 → 512 → 512 .
We use dropout with a dropping rate equal to 0.1 after each non-linearity in the convolutional network.
The LSTM has one layer, and its dimension is 512.
We split the training set of LibriTTS-all-960 into the training, validation, and test sets by a ratio of 85%, 7.5%, 7.5%, respectively.
We use the same learning rate schedule and optimizer mentioned above to train the classiﬁer.
The classiﬁer achieves 96.5% validation accuracy.
E M ORE SPEECH AND HANDWRITING GENERATION RESULTS
Our results can be viewed at https://apple.github.io/ml-style-equalization .
In the website, we show an extensive list of speech and handwriting samples generated by the proposed and the baseline methods.
Note that it may take a while for the speech results to load, and if the audio players do not contain the play button, please increase the size of the browser window.
To remove the effect of the vocoder when comparing synthesized speech samples with real speech samples, all real speech samples (including those in the style opinion score evaluations) are converted to mel-spectrogram and reconstructed back to waveform using the same vocoder that is used by the generative models, i.e ., waveglow (Prenger et al., 2019).
For speech synthesis, the webpage contains
• a video showcasing the generation of speech with various styles and content • nonparallel-text generation with seen speakers from LibriTTS-all- • nonparallel-text generation with unseen speakers from LibriTTS-all- • an ablation study that compares training with and without style equalization
• interpolation between two unseen style reference speech • generated speech with random styles sampled from the learned prior • nonparallel-text generation with seen speakers from VCTK • parallel-text generation with seen speakers from VCTK
For handwriting synthesis, the webpage contains
• a video showcasing the online generation of handwriting with various styles and content • nonparallel-text generation with unseen style • parallel-text generation with unseen style • generated handwriting with random styles sampled from the learned prior • interpolation between two unseen style reference handwriting
F A DDITIONAL ABLATION STUDY : THE ROLE OF x (cid:48)
During training, the proposed style equalization randomly selects a sample from the training dataset
NEW as x (cid:48) , which is unrelated to the ground-truth x .
One question is naturally raised: “Since x (cid:48) is unrelated to x , do we really need it to be a sample?”.
Theoretically, since our design of M and φ in eq.
(2) prevents content leakage and transfers time-independent ground-truth style information from x through z , as long as x (cid:48) does not contain the content information about x , the learned model should be able to control style and content separately during inference.
To verify the hypothesis, we conduct the following ablation studies:
x (cid:48) is a ﬁxed vector: We initialize x (cid:48) as a random vector but ﬁxed it during training and inference.
x (cid:48) is a random noise: We randomly sample x (cid:48) from the standard Gaussian distribution.
As can be seen from Table 5, both methods are able to train models that can separately control content and style (low WER and avgRank and high cos-sim ).
They achieve higher style replication quality than GST-nS, which has additional speaker information.
Nevertheless, the proposed model ( i.e ., using real samples as x (cid:48) ) is able to utilize both time-independent and time-dependent style information of x (see discussion in Sec. 5.4), and thus, our model still outperforms the two models.
There are additional disadvantages when using random noises as x
(cid:48) instead of real samples: • During inference, the two models still need to run M and φ .
In comparison, our usage of x (cid:48) allows us to remove M and φ when mimicking a reference example.
• Interpolation between two reference examples becomes non-trivial.
By design, our usage of x (cid:48) enables the model to learn to transform the style from one real sample to another.
In contract, the other two models only learn to transfer style to a random noise (or a ﬁxed vector).
Thus, while interpolation is straight-forward for our model, it is not for the two methods.
[Table]
G O VERVIEW OF RELATED WORKS
NEW
We provide a high-level summary of various controllable sequence generative models in Table 6.
In the table, we compare the methods on their needs of : (1) user identiﬁcation or pretrained embedding, (2) phoneme or character segmentation, (3) content recognizer or pretrained encoder, (4) their training loss and procedure, and (5) the applications shown in the papers.
As can be seen and to the best of our knowledge, while there exist many controllable sequence generative models, our proposed method is the ﬁrst that does not require user ID, segmentation, pretrained recognizer and proven to be applicable on both speech and handwriting domains
[Table] [Table]
H U TILIZING TIME - DEPENDENT STYLE INFORMATION
NEW
In the section, we answer the question: “does our model utilize time-dependent style information?”
When a model fully utilizes time-dependent style information, in the parallel setting (where the input text appears exactly in the reference example), the generated output should be a reconstruction of the reference example.
On the other hand, if a model utilizes only time-independent style information, its outputs in the parallel setting can still be different from the reference example, i.e ., the same sentence spoken by the same user can be very different in different contexts.
The difference in style replication quality provides a basis to the following evaluation.
We ﬁrst evaluate the capability of our model to utilize time-dependent style information by comparing the results between parallel (where all the time-dependent style information is available) and nonparallel (where little time-dependent and mostly time-independent style is available) settings.
As can be seen in Table 1, Table 2 and Fig.
3d, our results (WER, CER, and cos-sim) in the parallel setting are better than those in the non-parallel setting.
The results show that the model is capable of utilizing the time-dependent information in the reference example.
Next, we verify this capability by visualizing the style attention weights in the three scenarios in
Fig.
4b-d.
In the parallel setting (Fig. 4b), the model strongly attends to the correct time instance; in the non-parallel setting (Fig. 4c), the model attends to localized pieces in the reference example; in the transformed setting (Fig. 4d), the style attention weights are more diffused over the sequence.
Additionally, we qualitatively evaluate the capability via the handwriting/speech generation results provided in the supplementary material.
As can be seen, the reproduction is “qualitatively closer“ in the parallel setting and in the nonparallel setting when a character appears in the style reference within a similar context, e.g ., in Fig.
6b, the character ‘A’ and ‘a’ in the ﬁrst non-parallel text example and ‘p’ in the second example.
We also provide a nonparallel example that has overlapping content as the reference style in the third example in Fig.
6b.
As can be seen, the generated result is qualitatively similar to the reference example at the overlapped regions even when they are separated by unrelated text.
The ablation study in Table 5 compares our model (which is trained to utilize time-dependent style information via applying style equalization to 50% of the batches) with two models that always transform style inputs (and thus can only use time-independent style information).
While all models in the comparison use the same style encoder, our model, with its capability to utilize time-dependent style information, achieves the highest cosine similarities.
The style attention weights shown in
Fig. 4a and Fig.
4b also verify that our training method enables the use of time-dependent information.
Finally, we compare the same model trained with our method (which applies style equalization to 50% of the batches) and entirely without style equalization in Fig.
3d.
While both models can utilize time-dependent information, the model trained without style equalization suffers from content leakage (as shown by the high CER in the nonparallel text setting).
In comparison, the proposed model is less effected by the settings.
In summary, the proposed method (utilizing the style attention module and applying style equalization on half of the batches) enables the model to utilize time-dependent style information while avoiding catastrophic content leakage.