Investigation of using disentangled and interpretable representations with language conditioning for cross-lingual voice conversion
Abstract
We study the problem of cross-lingual voice conversion in non-parallel speechcorpora and one-shot learning setting.
Most prior work require either parallelspeech corpora or enough amount of training data from a target speaker.
However,we convert an arbitrary sentences of an arbitrary source speaker to target speaker’sgiven only one target speaker training utterance.
To achieve this, we formulate theproblem as learning disentangled speaker-speciﬁc and context-speciﬁc representations and follow the idea of (1) which uses Factorized Hierarchical Variational
Autoencoder (FHVAE).
After training FHVAE on multi-speaker training data,given arbitrary source and target speakers’ utterance, we estimate those latentrepresentations and then reconstruct the desired utterance of converted voice tothat of target speaker.
We use multi-language speech corpus to learn a universalmodel that works for all of the languages.
We investigate the use of a one-hotlanguage embedding to condition the model on the language of the utterance beingqueried and show the effectiveness of the approach.
We conduct voice conversionexperiments with varying size of training utterances and it was able to achieve reasonable performance with even just one training utterance.
We also investigate theeffect of using or not using the language conditioning.
Furthermore, we visualizethe embeddings of the different languages and sexes.
Finally, in the subjectivetests, for one language and cross-lingual voice conversion, our approach achievedmoderately better or comparable results compared to the baseline in speech qualityand similarity.
Introduction
The task of Voice Conversion (VC) (2; 3) is a technique to convert source speaker’s spoken sentencesinto those of a target speaker’s voice.
It requires to preserve not only the target speaker’s identity,but also phonetic context spoken by the source speaker.
To tackle this problem, many approacheshave been proposed (4; 5; 6).
However, most prior work require parallel spoken corpus and enoughamount of data to learn the target speaker’s voice.
Recently, there were approaches proposed forvoice conversion with non-parallel corpus (7; 8; 9).
But they still require that speaker identity wasknown priori , or included in training data for the model.
Recently, Hsu et al.
(1) proposed to use disentangled and interpretable representations to overcomethese limitations by exploiting Factorized Hierarchical Variation Autoencoder.
They achievedreasonable quality with just single utterance from a target speaker but it was still not satisfactory.
Nevertheless, most prior work focus on voice conversion within one language.
But we believe that ifwe can capture disentangled representations of phonetic or linguistic contexts and speaker identities,the model should be capable for more challenging cross -lingual setting, which means that source and target speakers are from different languages.
Therefore, we focus on investigating cross-lingual voiceconversion, and propose to follow the same spirit from Hsu et al.
(1) and improve the performance.
Our contributions are:
• We build a voice model which is trained on utterances from 5 different languages to let themodel observe as much speaker and phonetic variations as possible.
• We conduct cross-lingual voice conversion experiments and our approach achieved significantly better or comparable results than baselines in speech quality and similarity in thesubjective tests.•
We examine the effect of using additional one-hot embedding along with speaker embeddingthat determines the input utterance language.
Related Work
Voice conversion has been an important research problem for over a decade.
One popular approach totackle the problem is spectral conversion such as Gaussian mixture models (GMMs) (4) and deepneural networks (DNN) (5).
However, it requires parallel spoken corpus and dynamic time warping(DTW) is usually used to align source and target utterances.
To overcome this limitation, nonparallel voice conversion approaches were proposed, for instance, eigenvoice (6), i-vecotor (10), and
Variational Autoencoder (7; 9) based models.
However, eigenvoice based approach (6) still requiresreference speaker to train the model, and VAE based approaches (7; 9) require speaker identities to beknown priori as included in training data for the model.
i-vector based approach (10) looks promisingwhich remains to be studied further.
The i-vectors are converted by replacing the source latent variableby the target latent variable.
The Gaussian mixture means are then reconstructed from the convertedi-vector.
The Gaussians with adjusted means are then applied to the source vector to perform theacoustic feature conversion.
Siamese autoencoder has also been proposed for decomposing speakeridentity and linguistic embeddings (11).
However, this approach requires parallel training data tolearn the decomposing architecture.
This decomposition is achieved by means of applying somesimilarity and non-similarity costs between the Siamese architectures.
Nonetheless, cross-lingual voice conversion is also a challenging task since target language is notknown in training time, and only few work has proposed, including GMMs based approach (12) andeigenvoice based approach (13), but still have inherent limitations as above.
Recently, deep generative models have been applied and successful for unsupervised learning tasks,and include Variational Autoencoder (VAE) (14), Generative Adversarial Networks (GAN) (15), andauto-regressive models (16; 17).
Among them, VAE can infer latent codes from data and generatedata from them by jointly learning inference and generative networks, and VAE has been also appliedfor voice conversion (7; 9).
However, in their models, speaker identities are not infered from data andinstead required to be known in model training time.
GAN has been also exploited for non-parallelvoice conversion (18) with the cycle consistency contraint (19), but it still has the limitation that itneeds to know the target speaker in training time and be trained for each target.
To understand the disentangled and interpretable structure of latent codes, several work were proposed,namely, DC-IGN (20), InfoGAN (21), β -VAE (22), and FHVAE (1).
These approaches to uncoverdisentangled representation may help voice conversion with very limited resource from target speaker,since it might infer speaker identity information from data without supervision, as illustrated in
FHVAE (1).
However, the qualities of converted voices were not good enough, therefore, we focus onthe model structure of FHVAE and investigate to improve it, even with more challenging cross-lingualvoice conversion setting.
Model
Variational autoencoder (14) (VAE) is a powerful model to uncover hidden representation and generatenew data samples.
Let observations be x and latent variables z .
In the variational autoencoder model,the encoder (or inference network) q φ ( z | x ) outputs z given input x , and decoder p Φ ( x | z ) generatesdata x given z .
The encoder and decoder are neural networks.
Training is done by maximizingvariational lower bound (or also called evidence lower bound):
[Equation]
[Figure]
where D KL is Kullback-Leibler divergence.
However, VAE considers no structure for latent variable z .
Assuming structure for z could bebeneﬁcial to exploit the inherent structures in data.
Here we describe Factorized Hierarchical
Variational Autoencoder proposed by Hsu et al (1).
Let a dataset D consist of N seq i.i.d.
sequences
X i .
For each sequence X i , it consists of N iseg
X i,j observation segments.
Then we deﬁne factorizedlatent variables of latent segment variable Z i,j 1 and latent sequence variable Z i,j 2 .
In the context ofvoice conversion, Z i,j 1 is responsible for generating phonetic contexts and Z i,j 2 is for speaker identity.
When generating data X i,j , we ﬁrst sample Z i,j 2 from isotropic Gaussian centered at µ i shared forthe entire sequence, and also Z i,j 1 independently.
Then we generate X i,j conditioned on Z i,j 1 and
Z i,j 2 .
Thus, joint probability with a sequence X i is:
[Equation]
This is illustrated in Figure 1.
For inference, we use variational inference to approximate the trueposterior and have:
[Equation]
Since sequence variational lower bound can be decomposed to segment variational lower bound, wecan use batches of segment instead of sequence level to maximize:
[Equation]
where ˜ µ i is the posterior mean of µ i .
Please refer to Hsu et al.
(1) for more details.
Additionally,
Hsu et al.
also proposed discriminative segment variational lower bound to encourage Z i 2 to be moresequence-speciﬁc by adding the additional term of inferring the sequence index i from Z i,j 2 .
For ourexperiments, we exploit this FHVAE model and sequence-to-sequence model (23) as the structure ofencoder-decoder for sequential data.
We propose adding an input language embedding to the input ofthe model.
This language embedding will be used to determine the input utterance language using aone-hot representation of the in-training languages.
For performing the voice conversion, we compute the average Z 2 from the training utterance(s) ofsource and target speakers.
For a given input utterance, we compute Z 1 and Z 2 of the input utterance.
There are two ways to perform voice conversion.
First, we can replace Z 2 values of the source speakerwith the average Z 2 from the target speaker.
This approach resulted in too mufﬂed generated result.
Second, we compute a difference vector between source and target average Z diff= Z trg− Z src 2 .
This difference vector is added to Z 2 from the input utterance as Z converted= Z 2 + Z diffand thendecoded using FHVAE to achieve the speech features.
In an informal listening test, we decided to thesecond approach since it resulted in signiﬁcantly higher quality generated speech.
Experiments
4.1 Datasets
We used the TIMIT corpus (24) which is a multi-speaker speech corpus as the training data for
FHVAE model.
We used the training speakers as suggested by the corpus to train the model.
For
English test speakers, we select speakers from TIMIT testing part of the corpus.
We also use aproprietary Chinese speech corpus (hereon referred to as CH) with 5200 speakers each uttering onesentence.
We use Microsoft’s Indian Language Speech Corpus for Indian language.
We also useproprietary Korean and Japanese multi-speaker speech corpora. Additionaly. we consider using thecombination of all languages corpus for training the model.
For Korean, Japanese, and Indian corpora,we randomly exclude 10 percent of the speakers from each corpus for training purposes.
For Chinesetest speakers, we utilize speakers from the THCHS-30 speech corpus (25).
To observe the effect ofhaving more utterances per speaker but less speakers we also train the model on VCTK corpus (26).
Finally, for objective testing (which requires availability of parallel data), we utilized four CMU-arcticvoices (BDL, SLT, RMS, CLB)(27).
As speech features, we used 40th-order MCEPs (excluding theenergy coefﬁcient, dimensionality D=39), extracted using the World toolkit (28) with a 5ms frameshift.
All audio ﬁles are transformed to 16kHz and 16 bit before any analysis.
4.2 Experimental setting
For the encoder and decoder in FHVAE model, we use Long Short Term Memory (LSTM) (29) asthe ﬁrst layer with 256 hidden units with a fully-connected layer on top.
We use 32 dimensions foreach latent variable Z 1 and Z 2 .
The models were trained with stochastic gradient descent.
We usea mini-batch size of 256.
The Adam optimizer (30) is used with β 1 = 0.95, β 2 = 0.999, (cid:15) = 10 − 8 ,and initial learning rate of 10 − 4 .
The model is trained for 500 epochs and select the model bestperforming on the development set.
From now on, we use the abbreviation VAE for FHVAE model.
In our experiments, we consider twomodels: VAE-UNC (unconditioned) (1) and VAE-CND (conditioned) which mean models trainedeither with or without the language conditioning input.
We consider four gender conversions (F:female, M: male): F2F, F2M, M2F, M2M.
We also consider 25 cross-language conversions: allpermutations of 5 languages considered as source and target.
The voice conversion samples areavailable at: https://shamidreza.github.io/nips2018samples
4.3 Visualizing embeddings
In this experiment, we investigate the speaker embeddings Z 2 by visualizing them in Figure 2.
Forvisualizing the speaker embeddings, we use the 10 test speakers (5 male, 5 female) from each languagetest corpus.
We use VAE-UNC and VAE-CND.
In Figures 2, we show the speaker embeddingscomputed from 1 utterance where the 2D plot of the speaker embeddings (computed using PCA)are shown.
In all subplots, the female and male embedding cluster locations are clearly separated.
Furthermore, the plot shows that the speaker embeddings of unique speakers fall near the samelocation.
One phenomenon that we notice is that the speaker embeddings for different languages andgender fall to different locations for VAE-UNC, however, they fall closer to each other in VAE-CND.
This might be due to the conditioning on language improving the representation ability of the model.
Furthermore, we investigate the phonetic context embedding Z 1 for a sentence for four English testspeakers on VAE-UNC.
The phonetic context matrix over the computed utterances (compressed using
PCA) is shown in Figures 3.
Ideally, we want the matrices should be close to each other since thephonetic context embedding is supposed to be speaker-independent.
The ﬁgure show the closenessof the embeddings at the similar time frames.
There is still some minor discrepancy between the
[Figure] [Figure]
[Figure] [Figure]
embeddings which shows room for further improvement of model architecture and/or larger speechcorpus.
4.4 Subjective evaluation
To subjectively evaluate voice conversion performance, we performed two perceptual tests.
Theﬁrst test measured speech quality, designed to answer the question “how natural does the convertedspeech sound”?, and the second test measured speaker similarity, designed to answer the question“how accurate does the converted speech mimic the target speaker"?.
The listening experiments were all
F2F
F2M M2F
M2M 0 .
[Figure]
C M
O S
VAE-UNC vs. VAE-CND
[Figure]
4.4.1 Speech quality
[Figure]
To evaluate the speech quality of the converted utterances, we conducted a Comparative Mean Opinion
Score (CMOS) test.
In this test, listeners heard two stimuli A and B with the same content, generatedusing the same source speaker, but in two different processing conditions, and were then asked toindicate whether they thought B was better or worse than A, using a ﬁve-point scale comprisedof +2 (much better), +1 (somewhat better), 0 (same), -1 (somewhat worse), -2 (much worse).
Werandomized the order of stimulus presentation, both the order of A and B, as well as the order of thecomparison pairs.
We utilized three processing conditions: VAE-UNC, VAE-CND.
We assessed the
VC approach effect by directly comparing VAE-UNC vs. VAE-CND utterances.
The experiment wasadministered to 50 listeners with each listener judging 100 sentence pairs.
We achieved +0.15 ± 0.14mean score towards VAE-CND.
Although this is a positive difference, we did not ﬁnd statisticallysigniﬁcant difference between the quality of VAE-UNC and VAE-CND.
The language-breakdown ofthe results are shown in Figure 4.
4.4.2 Speaker similarity
To evaluate the speaker similarity of the converted utterances, we conducted a same-different speakersimilarity test (31).
In this test, listeners heard two stimuli A and B with different content, andwere then asked to indicate whether they thought that A and B were spoken by the same, or by twodifferent speakers, using a ﬁve-point scale comprised of +2 (deﬁnitely same), +1 (probably same), 0(unsure), -1 (probably different), and -2 (deﬁnitely different).
One of the stimuli in each pair wascreated by one of the two conversion methods, and the other stimulus was a purely MCEP-vocodedcondition, used as the reference speaker.
The listeners were explicitly instructed to disregard thelanguage of the stimuli and merely judge based on the fact whether they think the utterances are fromthe same speaker regardless of the language.
Half of all pairs were created with the reference speakeridentical to the target speaker of the conversion (expecting listeners to reply “same", ideally); theother half were created with the reference speaker being the same gender, but not identical to thetarget speaker of the conversion (expecting listeners to reply different).
We only report “same" scores.
The experiment was administered to 50 listeners, with each listener judging 100 sentence pairs.
Theresults are shown in Figure 5.
We did ﬁnd a consistent improvement of VAE-CND performance over
VAE-UNC, however these differences where not statistically signiﬁcant.
Conclusions
We proposed to exploit FHVAE model for challenging non-parallel and cross-lingual voice conversion,even with very small number of training utterances such as only one target speaker’s utterance.
We usemulti-language corpus to learn disentangled representations from speech.
We also introduce a one-hotlanguage embedding to the model in order to improve the model performance.
We perform severalvisualizations to show the effect of the model.
In the subjective tests, we found some improvement inthe quality and similarity performance of the system, although the differences where not statisticallysigniﬁcant.