Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization
Abstract
To leverage crowd-sourced data to train multi-speaker text-to-speech (TTS) modelsthat can synthesize clean speech for all speakers, it is essential to learn disentangledrepresentations which can independently control the speaker identity and background noise in generated signals.
However, learning such representations can bechallenging, due to the lack of labels describing the recording conditions of eachtraining example, and the fact that speakers and recording conditions are oftencorrelated, e.g. since users often make many recordings using the same equipment.
This paper proposes three components to address this problem by: (1) formulatinga conditional generative model with factorized latent variables, (2) using dataaugmentation to add noise that is not correlated with speaker identity and whoselabel is known during training, and (3) using adversarial factorization to improvedisentanglement.
Experimental results demonstrate that the proposed method candisentangle speaker and noise attributes even if they are correlated in the trainingdata, and can be used to consistently synthesize clean speech for all speakers.
Ablation studies verify the importance of each proposed component.
Introduction
Recent development of neural end-to-end TTS models [28, 2] enables control of both labelled andunlabelled speech attributes by conditioning synthesis on both text and learned attribute representations [29, 23, 12, 1, 7, 11].
This opens the door to leveraging crowd-sourced speech recordedunder various acoustic conditions [20] to train a high-quality multi-speaker TTS model that is capable of consistently producing clean speech.
To achieve this, it is essential to learn disentangledrepresentations that control speaker and acoustic conditions independently.
However, this can bechallenging for two reasons.
First, the underlying acoustic conditions of an utterance, such as the typeand level of background noise and reverberation, are difﬁcult to annotate and therefore such labelsare often unavailable.
This hinders the use of direct conditioning on the acoustic condition labels in away similar to conditioning on one-hot speaker labels [2].
Second, speaker identity can have strongcorrelations with recording conditions, since a speaker might make most of their recordings in thesame location using the same device.
This makes it difﬁcult to learn a disentangled representation byassuming statistical independence [8].
We address this scenario by introducing three components: a conditional generative model withfactorized latent variables to control different attributes, data augmentation by adding backgroundnoise to training utterances in order to counteract the inherent speaker-noise correlation and to createground truth noisy acoustic condition labels, and adversarial training based on the generated labelsto encourage disentanglement between latent variables.
We utilize the VCTK speech synthesisdataset [25], and background noise signals from the CHiME-4 challenge [26] to synthesize a dataset containing correlated speaker and background noise conditions for controlled experiments.
Weextensively evaluate disentanglement performance on the learned latent representations as well asthe synthesized samples.
Experimental results identify the contribution of each component, anddemonstrate the ability of the proposed model to disentangle noise from speakers and consistentlysynthesize clean speech for all speakers, despite the strong correlation in the training data.
Proposed Method
[Figure]
2.1 Conditional factorized variational autoencoder for TTS
We base our TTS model on Tacotron 2 [22], which takes a text sequence as input, and outputs asequence of mel spectrogram frames.
To control speech attributes other than text, two additionallatent variables, z s and z r , are introduced to condition the generative process, where the formermodels speaker identity, and the latter models residual unlabelled attributes (e.g. acoustic conditions).
Prior distributions for both variables are deﬁned to be isotropic Gaussian.
The full TTS model can bewritten as a conditional generative model with two latent variables: p ( speech | z s , z r , text ) .
Two variational distributions are introduced: q ( z s | speech ) and q ( z r | speech ) , to approximatethe intractable posteriors of the latent variables, following the variational autoencoder (VAE) framework [16].
Each distribution is deﬁned to be diagonal-covariance Gaussian, whose mean and varianceare parameterized by a neural network encoder.
Note that z s , z r , and text are assumed to beconditionally independent given speech , in order to simplify inference.
In contrast to learning anembedding for each speaker, learning an inference model for z s can be used to infer speaker attributesfor previously unseen speakers.
To factorize speaker and residual information, an auxiliary speaker classiﬁer that takes z s as inputis trained jointly with the TTS model.
This encourages information that is discriminative betweenspeakers to be encoded in z s , and leaves residual information to z r .
A simple fully-connected networkis used for the speaker classiﬁer.
2.2 Speaker invariant data augmentation
When acoustic conditions are correlated with speakers, information about e.g. background noise levelcan be used to discriminate between speakers, and therefore can be encoded into z s .
To counteractsuch behavior, one can decorrelate these factors by leveraging prior knowledge that adding noiseshould not affect speaker identity.
We propose to augment the original training set with a noisy copy that mixes each utterance witha randomly selected piece of background noise at a randomly sampled signal-to-noise ratio (SNR),but reuses the same transcript and speaker label as the original utterance.
This operation can be seenas ﬂattening the SNR distribution of each speaker, in order to make SNRs less discriminative aboutspeakers.
2.3 Augmentation-adversarial training
To increase the degree of disentanglement, it is also useful to proactively discourage z s from encodingacoustic condition information.
If the ground truth acoustic condition labels are available, domain adversarial training [4] can be applied directly to encourage z s not to be informative about theacoustic condition.
Nevertheless, such labels are often unavailable in crowdsourced datasets such as[20].
In order to utilize adversarial training in such a scenario, we propose to use the augmentation label(original/augmented) to replace the acoustic condition label (clean/noisy).
This augmentation labelcan be seen as a noisy acoustic condition label: an augmented utterance must be noisy, but an originalone can be either .
If z s is invariant to acoustic conditions, then it is also invariant to augmentationlabels, implying that the latter is a necessary condition for the former.
Following [4], invariance of z s to augmentation is measured using the empirical H -divergencebetween the z s distribution of the augmented data and that of the original data, given a hypothesisclass H .
As suggested in [3], the empirical H -divergence can be approximated with the Proxy
A -distance : 2(1 − 2 (cid:15) ) , where (cid:15) is a generalization error of an augmentation classiﬁer trained topredict if z s is inferred from an augmented utterance.
2.4 Model and training objective
The complete model is illustrated in Figure 1, composed of three modules: a synthesizer,p ( speech | z s , z r , text ) , an inference network with two encoders, q ( z s | speech ) and q ( z r | speech ) ,and an adversarial factorization module with speaker and augmentation classiﬁers, p ( y s | z s ) andp ( y a | z r ) , where y s and y a denotes speaker and augmentation labels.
The parameters of the synthesizer, the two encoders, the speaker classiﬁer, and the augmentation classiﬁers are accordinglydenoted as θ , φ s , φ r , ψ s , and ψ a , respectively.
Training of the proposed model aims to maximize the conditional likelihood and the informationz s contains about speakers, while minimizing the H -divergence between the z s inferred from theoriginal utterances and that from the augmented ones.
The objective function can be formulated ascombining an evidence lower bound (ELBO) with a domain adversarial training [4] objective:
[Equation]
where λ 1 , λ 2 > 0 are the loss weights for the two classiﬁers, and ELBO ( θ, φ s , φ r ; speech, text ) isformulated as:
[Equation]
Note that the augmentation classiﬁer is optimized with a different objective than the rest of themodel.
To train the entire model jointly, a gradient reversal layer [4] is inserted after the input to theaugmentation classiﬁer, which scales the gradient by − λ 2 .
Related Work
Our formulation of a TTS model with latent variables are closely related to [29, 23, 1, 7, 11], whichfocus on modeling unlabeled speech attributes.
In contrast to this work, [29, 23, 1, 7] do not addressdisentangling attributes to enable independent control when different attributes are highly correlatedin the training data, while [11] learns to disentangle speaker attributes from the rest by encodingthose with small within-speaker variance to z s .
The proposed augmentation-adversarial training combines data augmentation for speech [13] withdomain adversarial neural networks (DANNs)
[4] for disentangling correlated attributes.
These twomethods have been mainly applied for training robust discriminative models [9, 24, 26, 21], andare less studied in the context of building generative models.
In addition, our method provides twoadvantages.
First, while DANNs require domain labels, our proposed method enables adversarialtraining even when the ground truth domain labels are unavailable.
Second, domain adversarial training aims to remove domain information while preserving target attribute information; however, ifdomain and target attribute have very strong correlations, the two objectives conﬂict with each other,and one of the them will be compromised.
Our proposed method alleviates such issues by using dataaugmentation to decorrelate the two factors.
Learning disentangled representations for deep generative models has gain much interest recently [10, 30].
Several studies also explored adversarial training for disentanglement, such asusing maximum mean discrepancy [17] and generative adversarial network [19].
We particularlyemphasize disentangling statistically correlated attributes, and apply H -divergence based adversarialtraining on latent variables.
Experiments
We artiﬁcially generated a noisy speech dataset with correlated speaker and noise conditions usingthe VCTK corpus [25] and background noise from the CHiME-4 challenge [26].
The motivation hereis to simulate real noisy data while evaluating the model under carefully controlled conditions.
VCTKcontains 44 hours of clean English speech from 109 speakers.
We downsample the signals to 16 kHzto match the background noise sample rate, and split it into training and testing sets in a 9:1 ratio.
The CHiME-4 corpus contains 8.5 hours of background noise recorded in four different locations(bus, cafe, pedestrian area, and street junction), which we split into three partitions: train , test ,and aug .
To simulate speaker-correlated noise, we randomly selected half the speakers to be noisy,and mixed all of their train and test utterances with noise sampled from train and test respectively,at SNRs ranging from 5 - 25 dB. As described in Section 2.2, we generated an augmented set bymixing every (potentially noisy) training utterance with a noise signal sampled from aug at SNRsranging from 5 - 25 dB. Utterances in the augmented set are annotated with y a = 1 , and those in theoriginal noisy training set are annotated with y a = 0 .
4.1 Model and training setup
The synthesizer network use the sequence-to-sequence Tacotron 2 architecture [22], with extra inputz s and z r concatenated and passed to the decoder at each step.
If not otherwise mentioned, z s is64-dim and z r is 8-dim.
The generated speech is represented as a sequence of 80-dim mel-scaleﬁlterbank frames, computed from 50ms windows shifted by 12.5ms.
We represent input text as asequence of phonemes, since learning pronunciations from text is not our focus.
The speaker and the residual encoders both use the same architecture which closely follow the attributeencoder in [11].
Each encoder maps a variable length mel spectrogram to two vectors parameterizingthe mean and log variance of the Gaussian posterior.
Both classiﬁers are fully-connected networkswith one 256 unit hidden layer followed by a softmax layer to predict the speaker or augmentationposterior.
The synthesizer, encoders, and speaker classiﬁer are trained to maximize Eq (1) with λ 1 = λ
= 1 ,and the augmentation classiﬁer is trained to maximize Eq (2).
The entire model is trained jointly witha batch size of 256, using the Adam optimizer [15], conﬁgured with an initial learning rate of 10 − 3 ,and an exponential decay that halves the learning rate every 12.5k steps, starting at 50k steps.
4.2 Latent space disentanglement
We quantify the degree of disentanglement by training speaker and noise classiﬁers on z s and z rseparately.
The classiﬁcation accuracy on a held-out set is used to measure how much information alatent variable contains about the prediction targets.
A simple linear discriminative analysis classiﬁeris used for all four tasks.
If the classiﬁer input contains no information about the target, the besta classiﬁer can do is to predict the highest prior probability class.
Since the distributions of bothspeaker and acoustic conditions are close to uniform, a speaker-uninformative input should result inabout 1% accuracy, and a noise-uninformative input should result in about 50% .
Results are shown in Table 1, comparing the full proposed model with two alternative models: onewhich removes adversarial training, denoted as “ - adv ,”, and a second which further removes dataaugmentation, denoted as “ - adv - aug .”
Without data augmentation and adversarial training, thesecond alternative completely fails to disentangle speaker from noise, i.e. its speaker encoding z s can infer both, while its residual encoding z s cannot infer either.
The ﬁrst alternative learns to encodeacoustic condition into z r , reaching 96 . 5% accuracy on noise prediction; however, part of suchinformation still leaks to z s , as indicated by the 85% noise prediction accuracy.
The full proposedmodel achieves the highest noise prediction accuracy using z r , and the lowest accuracy using z s ,implying the best allocation of acoustic information.
Nevertheless, adversarial training also results inslight degradation of speaker information allocation, where the speaker prediction accuracy using z rincreases from 1 . 4% to 2 .
[Table]
4.3 Visualization
We further analyze the latent space of the proposed model by visualizing the learned speaker andresidual representations using t-SNE [18], which is a technique for projecting high-dimensionalvectors to a two-dimensional space.
Results are shown in Figure 2, where each point correspondingto a projected z r (left column) or z s (right column) inferred from a single utterance.
Points arecolor-coded according to speaker, gender, and accent labels in each row.
In the left column, projected z r are clearly separated by acoustic condition, but not by gender orspeaker.
In contrast, projected z s shown in the right column forms many small clusters, with onespeaker each cluster; Moreover, as shown in the middle row, clusters of speakers are further separatedaccording to their genders.
In the lower right panel, projected z s of noisy utterances and cleanutterances are overlaid, demonstrating that z s have similar distributions conditioning on differentacoustic conditions.
4.4 Evaluation of synthesized speech
To evaluate how well the two latent variables, z s and z r , can control the synthesized speech, wesample ﬁve clean speakers and ﬁve noisy speakers, and select one testing utterance for each speakerwith duration ≥ 3 s. For each of the ten utterances, the two latent variables are inferred using thecorresponding encoders.
We construct an evaluation set of 100 phrases that does not overlap with the
VCTK corpus, and synthesize them conditioned on each combination of z r and z s , including thoseinferred from different utterances.
The total 10,000 synthesized samples are divided into four groups,depending on the set of speakers (clean/noisy) z r and z s are inferred from.
4.4.1 Control of acoustic conditions
To quantify the ability to control noise, we use waveform amplitude distribution analysis (WADA)
[14]to estimate an SNR without a clean reference signal.
We compare to a baseline multi-speaker Tacotronmodel, which removes the residual encoder and replaces the speaker encoder with a lookup table of64-D speaker embeddings.
The upper half of Table 2 presents the estimated SNRs of synthesizedspeech using this baseline, conditioning on the same ﬁve clean speakers and the ﬁve noisy speakersmentioned above.
The difference in SNR between clean and noisy speakers indicates that the acousticcondition is tied to speaker identity in this baseline model.
Results of the proposed model and the two alternatives mentioned in Section 4.2 are shown in thelower half of Table 2.
By conditioning on z r inferred from clean utterances, the proposed model isable to synthesize clean speech even for noisy speakers whose training utterances all had backgroundnoise.
Moreover, when conditioning on the same set of z r , the proposed achieves the smallestdiscrepancy in SNR between different z s sets.
On the other hand, the "-adv" variant has a largerdiscrepancy between different z s sets, indicating worse disentanglement comparing to the full model,while the "-adv-aug" variant fails to control noise through z r .
These results are in line with the noiseprediction results using z s and z r shown in Table 1.
Figure 3 illustrates synthesized samples for a
[Figure] [Figure]
noisy speaker, comparing the baseline to our proposed model.
Our model is capable of controllingnoise using z r , and can generate clean speech for the noisy speaker, while the baseline output alwayscontains background noise.
4.4.2 Control of speaker identity
We next examine if z s can control the speaker identity of synthesized speech, using a text-independentspeaker veriﬁcation system [27] to is compute speaker discriminative embeddings, called d-vector s
[6],from the reference and synthesized speech samples.
The system is trained to optimize a generalized
[Table]
end-to-end speaker veriﬁcation loss, so that the embeddings of two utterances are close to each otherif they are from the same speaker, and far way if from different speakers.
We build a nearest-neighbor classiﬁer, which assigns an input signal the speaker label of the referencesignal whose d-vector is closest to that of the input, measured using Euclidean distance.
To preventbackground noise from affecting d-vector quality, we only evaluate synthesized samples conditionedon z r from clean utterances.
Table 3 shows that the synthesized samples closely resemble the speakercharacteristics of their corresponding reference samples, regardless of z r used for conditioning.
Theresults indicate that speaker identity is controlled by z s , while being invariant to change in z r .
[Table]
[Table]
3 Subjective naturalness evaluation
To quantify ﬁdelity, we rely on crowd-sourced mean opinion scores (MOS), which rates the naturalness of the synthesized samples by natives speakers using headphones, with scores ranging from1 to 5 in 0.5 increments.
Results shown in Table 4 compares the baseline and the proposed modelconditioning on z r from clean utterances.
Note that because the Grifﬁn-Lim algorithm [5] is used forsynthesizing waveforms from power spectrograms, the MOS scores would be lower than those using a WaveNet vocoder [22]. When conditioning on z r from clean utterances, the proposed model achievesa higher MOS score than the baseline. In contrast, the MOS drops signiﬁcantly when conditioning onz r inferred from noisy utterances.
The results indicate that disentangling speaker and noise improvesthe naturalness of the generated speech, and the proposed model can synthesize more natural speechwith less background noise than the baseline when conditioning on z r inferred from clean signals.
[Table] 4.5 Hyperparameter sensitivity [Table] Finally, we study the sensitivity of disentanglement performance with respect to the choice of speakerencoding dimensions.
As shown in the previous two sections, good latent space disentanglementtranslates to good performance in terms of control of speaker identity and acoustic conditions forsynthesis.
In this section, we only evaluate latent space disentanglement when changing the dimensionof z s
Table 5 compares performance of the proposed model when the dimensionality of z s is 32, 64,128, and 256.
Variants without data augmentation or adversarial training fail to disentangle in allconﬁgurations.
When the dimension of z s increases, both the proposed model and "-adv" reportworse separation of information, as indicated by increased noise prediction accuracy using z s .
Speciﬁcally, the "-adv" fails to encode noise information in z r when z s has 128 dimensions, whichcould result from a bad initialization of model parameters; however, such a behavior also indicatesthat when adversarial training is not applied, the disentanglement performance may rely heavily onthe model initialization.
On the other hand, the proposed model is least sensitive to the change ofz s dimensionality.
It always achieves the highest noise prediction accuracy using z r , and the lowestnoise prediction accuracy using z s . [Table] [Table]
Conclusion and Future Work
We build a neural network TTS model which incorporates conditional generative modeling, dataaugmentation, and adversarial training to learn disentangled representations of correlated and partiallyunlabeled attributes, which can be used to independently control different aspects of the synthesizedspeech.
Extensive studies on a synthetic dataset verify the effectiveness of each element of theproposed solution, and demonstrate the robustness to the choice of hyperparameters.
The proposed methods for disentangling correlated attributes is general, and can potentially be appliedto other pairs of correlated factors, such as reverberation and speaker, or to other modalities, such ascontrollable text-to-image generation.
In addition, for future work, we would also like to investigatethe capability of the proposed method to disentangle pairs of attributes which are both unsupervised.