I MPROVING Z ERO -S HOT V OICE S TYLE T RANSFER VIA D ISENTANGLED R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Voice style transfer, also called voice conversion, seeks to modify one speaker’s voice to generate speech as if it came from another (target) speaker.
Previous works have made progress on voice conversion with parallel training data and pre-known speakers.
However, zero-shot voice style transfer, which learns from non-parallel data and generates voices for previously unseen speakers, remains a challenging problem.
In this paper we propose a novel zero-shot voice transfer method via disentangled representation learning.
The proposed method ﬁrst encodes speaker-related style and voice content of each input voice into separate low-dimensional embedding spaces, and then transfers to a new voice by combining the source content embedding and target style embedding through a decoder.
With information-theoretic guidance, the style and content embedding spaces are representative and (ideally) independent of each other.
On real-world VCTK datasets, our method outperforms other baselines and obtains state-of-theart results in terms of transfer accuracy and voice naturalness for voice style transfer experiments under both many-to-many and zero-shot setups.
I NTRODUCTION
Style transfer, which automatically converts a data instance into a target style, while preserving its content information, has attracted considerable attention in various machine learning domains, including computer vision (Gatys et al., 2016;
Luan et al., 2017; Huang & Belongie, 2017), video processing (Huang et al., 2017; Chen et al., 2017), and natural language processing (Shen et al., 2017; Yang et al., 2018; Lample et al., 2019).
In speech processing, style transfer was earlier recognized as voice conversion (VC) (Muda et al., 2010), which converts one speaker’s utterance, as if it was from another speaker but with the same semantic meaning.
Voice style transfer (VST) has received long-term research interest, due to its potential for applications in security (Sisman et al., 2018), medicine (Nakamura et al., 2006), entertainment (Villavicencio & Bonada, 2010) and education (Mohammadi & Kain, 2017), among others.
Although widely investigated, VST remains challenging when applied to more general application scenarios.
Most of the traditional VST methods require parallel training data, i.e. , paired voices from two speakers uttering the same sentence.
This constraint limits the application of such models in the real world, where data are often not pair-wise available.
Among the few existing models that address non-parallel data (Hsu et al., 2016; Lee & Wu, 2006; Godoy et al., 2011), most methods cannot handle many-to-many transfer (Saito et al., 2018; Kaneko & Kameoka, 2018; Kameoka et al., 2018), which prevents them from converting multiple source voices to multiple target speaker styles.
Even among the few non-parallel many-to-many transfer models, to the best of our knowledge, only two models (Qian et al., 2019; Chou & Lee, 2019) allow zero-shot transfer, i.e. , conversion from/to newly-coming speakers (unseen during training) without re-training the model.
The only two zero-shot VST models (AUTOVC (Qian et al., 2019) and AdaIN-VC (Chou & Lee, 2019)) share a common weakness.
Both methods construct encoder-decoder frameworks, which extract the style and the content information into style and content embeddings, and generate a voice sample by combining a style embedding and a content embedding through the decoder.
With the combination of the source content embedding and the target style embedding, the models generate the transferred voice, based only on source and target voice samples.
AUTOVC (Qian et al., 2019) uses a GE2E (Wan et al., 2018) pre-trained style encoder to ensure rich speaker-related information in style embeddings.
However, AUTOVC has no regularizer to guarantee that the content encoder does not encode any style information.
AdaIN-VC (Chou & Lee, 2019) applies instance normaliza- tion (Ulyanov et al., 2016) to the feature map of content representations, which helps to eliminate the style information from content embeddings.
However, AdaIN-VC fails to prevent content information from being revealed in the style embeddings.
Both methods cannot assure that the style and content embeddings are disentangled without information revealed from each other.
With information-theoretic guidance, we propose a disentangled-representation-learning method to enhance the encoder-decoder zero-shot VST framework, for both style and content information preservation.
We call the proposed method I nformation-theoretic D isentangled E mbedding for V oice C onversion (IDE-VC).
Our model successfully induces the style and content of voices into independent representation spaces by minimizing the mutual information between style and content embeddings.
We also derive two new multi-group mutual information lower bounds, to further improve the representativeness of the latent embeddings.
Experiments demonstrate that our method outperforms previous works under both many-to-many and zero-shot transfer setups on two objective metrics and two subjective metrics.
B ACKGROUND
In information theory, mutual information (MI) is a crucial concept that measures the dependence between two random variables.
Mathematically, the MI between two variables x and y is
[Equation]
where p ( x ) and p ( y ) are marginal distributions of x and y , and p ( x , y ) is the joint distribution.
Recently, MI has attracted considerable interest in machine learning as a criterion to minimize or maximize the dependence between different parts of a model (Chen et al., 2016; Alemi et al., 2016;
Hjelm et al., 2018; Veli ˇ ckovi ´ c et al., 2018; Song et al., 2019).
However, the calculation of exact MI values is challenging in practice, since the closed form of joint distribution p ( x , y ) in equation (1) is generally unknown.
To solve this problem, several MI estimators have been proposed.
For MI maximization tasks, Nguyen, Wainwright, and Jordan (NWJ) (Nguyen et al., 2010) propose a lower bound by representing equation (1) as an f -divergence (Moon & Hero, 2014):
[Equation]
with a score function f ( x , y ) .
Another widely-used sample-based MI lower bound is InfoNCE (Oord et al., 2018), which is derived with Noise Contrastive Estimation (NCE) (Gutmann &
Hyv ¨ arinen, 2010).
With sample pairs { ( x i , y i ) } Ni =1 drawn from the joint distribution p ( x , y ) , the
InfoNCE lower bound is deﬁned as
[Equation]
For MI minimization tasks, Cheng et al.
(2020) proposed a contrastively learned upper bound that requires the conditional distribution p ( x | y ) :
[Equation]
where the MI is bounded by the log-ratio of conditional distribution p ( x | y ) between positive and negative sample pairs.
In the following, we derive our information-theoretical disentangled representation learning framework for voice style transfer based on the MI estimators described above.
P ROPOSED M ODEL
We assume access to N audio (voice) recordings from M speakers, where speaker u has N u voice samples X u
= { x ui } N u
i =1 .
The proposed approach encodes each voice input x ∈ X
=
∪ Mu =1 X u into a speaker-related (style) embedding s = E s ( x ) and a content-related embedding c =
E c ( x ) , using respectively a style encoder E s ( · ) and a content encoder E c ( · ) .
To transfer a source x ui from speaker u to the target style of the voice of speaker v , x vj , we combine the content embedding c ui =
E c ( x ui ) and the style embedding s vj = E s ( x vj ) to generate the transferred voice ˆ x u → v,i = D
( s vj , c ui ) with a decoder D ( s , c ) .
To implement this two-step transfer process, we introduce a novel mutual information (MI)-based learning objective, that induces the style embedding s and content embedding c into independent representation spaces ( i.e. , ideally, s contains rich style information of x with no content information, and vice versa).
In the following, we ﬁrst describe our MI-based training objective in Section 3.1, and then discuss the practical estimation of the objective in Section 3.2 and 3.3.
3. MI- BASED D ISENTANGLING O BJECTIVE
From an information-theoretic perspective, to learn representative latent embedding ( s , c ) , it is desirable to maximize the mutual information between the embedding pair ( s , c ) and the input x .
Meanwhile, the style embedding s and the content c are desired to be independent, so that we can control the style transfer process with different style and content attributes.
Therefore, we minimize the mutual information I ( s ; c ) to disentangle the style embedding and content embedding spaces.
Consequently, our overall disentangled-representation-learning objective seeks to minimize
[Equation]
As discussed in Locatello et al.
(Locatello et al., 2019), without inductive bias for supervision, the learned representation can be meaningless.
To address this problem, we use the speaker identity u as a variable with values { 1 , . . .
, M } to learn representative style embedding s for speaker-related attributes.
Noting that the process from speaker u to his/her voice x ui to the style embedding s ui (as u → x → s ) is a Markov Chain, we conclude I ( s ; x ) ≥ I ( s ; u ) based on the MI data-processing inequality (Cover & Thomas, 2012) (as stated in the Supplementary Material).
Therefore, we replace
I ( s ; x ) in L with I ( s ; u ) and minimize an upper bound instead:
[Equation]
In practice, calculating the MI is challenging, as we typically only have access to samples, and lack the required distributions (Chen et al., 2016).
To solve this problem, below we provide several MI estimates to the objective terms I ( s ; c ) , I ( x ; c | s ) and I ( u ; s ) .
3. MI L OWER B OUND E STIMATION
To maximize I ( u ; s ) , we derive the following multi-group MI lower bound (Theorem 3.1) based on the NWJ bound developed in Nguyen et al.
(Nguyen et al., 2010).
The detailed proof is provided in the Supplementary Material.
Let µ ( − ui ) v = µ v represent the mean of all style embeddings in group
X v , constituting the style centroid of speaker v ; µ ( − ui ) u is the mean of all style embeddings in group
X u except data point x ui , representing a leave- x ui -out style centroid of speaker u .
Intuitively, we minimize (cid:107) s ui − µ ( − ui ) u (cid:107) to encourage the style embedding of voice x ui to be more similar to the style centroid of speaker u , while maximizing (cid:107) s ui − µ ( − ui ) v (cid:107) to enlarge the margin between s ui and the other speakers’ style centroids µ v .
We denote the right-hand side of equation (7) as ˆ I 1 .
Theorem 3.1.
Let µ ( − ui ) v {+= 1 N v (cid:80) N v k =
+}
s vk if u (cid:54) = v ; and µ ( − ui ) u
= 1 N u − 1 (cid:80)
j (cid:54)
=
i s uj .
Then,
I ( u ; s ) ≥ E (cid:104) 1 N
M (cid:88) u
=
N u (cid:88) i = (cid:104) − (cid:107) s ui − µ ( − ui )
u (cid:107) − e − N M (cid:88) v [Equation]
To maximize I ( x ; c | s ) , we derive a conditional mutual information lower bound below:
Theorem 3.2.
Assume that given s = s u , samples { ( x ui , c ui ) } N u i =1 are observed.
With a variational distribution q φ ( x | s , c ) , we have I ( x ; c | s ) ≥ E
[ˆ I ] , where
[Equation]
Based on the criterion for s in equation (7), a well-learned style encoder E s pulls all style embeddings s ui from speaker u together.
Suppose s u is representative of the style embeddings of set X u .
If we parameterize the distribution q φ ( x | s , c ) ∝ exp( −(cid:107) x − D ( s , c ) (cid:107) 2 ) with decoder D ( s , c ) , then based on Theorem 3.2, we can estimate the lower bound of
I ( x ; c | s ) with the following objective:
[Equation]
[Table]
<latexit sha1_base
[Table]
+PasorQBZmzcYCKSGYnblO0T3rBUyQzbcJVkGy8/2Y4Iq1dyzwwJYHSXo81zv/FxjXMjieOq6oGpuj2oVktEtBJM4Gk4IZREOsACDU81JrQkhhCIcwp3nkml37XYUESszbFTm8ODJdU8KohS7JgJHwDBLUYv2VhMIZ90Hlt4VRLSVThsC21AdDe9eIkwU0Hhgm3ARXZpG4pQ9A4RMiV78WNtmIruqMRCH48mLQ6C2bUQNYNbqrJ9ZVz/dd4xYtmUmlzvDvEcghlN8PLqiQKtHSvfKtm+LwEYoxe+UPfSPq/7WgoQzdhQbLr63ATnA/6WdrP3h91hyftquyh5+gFeoky9AYN0Tt0hkaIogp9Rl/Q1+hT9C36Hv3YUqNOm/MM7Vj08w8RcPwh</latexit>
When maximizing ˆ I 2 , for speaker u with his/her given voice style s u , we encourage the content embedding c ui to well reconstruct the original voice x ui , with small (cid:107) x ui − D ( c ui , s u ) (cid:107) . Addi- tionally, the distance (cid:107) x uj − D ( c ui , s u ) (cid:107) is minimized, ensuring c ui does not contain information to reconstruct other voices x uj from speaker u . With ˆ I 2 , the correlation between x ui and c ui is ampliﬁed, which improves c ui in preserving the content information. 3. MI U PPER B OUND E STIMATION
The crucial part of our framework is disentangling the style and the content embedding spaces, which guarantees (ideally) that the style embedding s excludes any content information and vice versa.
Therefore, the mutual information between s and c is expected to be minimized.
To estimate
I ( s ; c ) , we derive a sample-based MI upper bound in Theorem 3.3 base on (4).
Theorem 3.3.
If p ( s | c ) provides the conditional distribution between variables s and c , then
[Equation]
The upper bound in (9) requires the ground-truth conditional distribution p ( s | c ) , whose closed form is unknown.
Therefore, we use a probabilistic neural network q θ ( s | c ) to approximate p ( s | c ) by maximizing the log-likelihood F ( θ )
= (cid:80) Mu =1 (cid:80) N u
i =1 log q θ
( s ui | c ui ) .
With the learned q θ ( s | c ) , the objective for minimizing
I ( s ; c ) becomes:
[Equation]
When weights of encoders E c , E s are updated, the embedding spaces s , c changes, which leads to the changing of conditional distribution p ( s | c ) .
Therefore, the neural approximation q θ ( s | c ) must be updated again.
Consequently, during training, the encoders E c , E s and the approximation q θ ( s | c ) are updated iteratively.
In the Supplementary Material, we further discuss that with a good approximation q θ ( s | c ) , ˆ I 3 remains an MI upper bound.
3. E NCODER -D ECODER F RAMEWORK
With the aforementioned MI estimates ˆ I 1 , ˆ I 2 , and ˆ I 3 , the ﬁnal training loss of our method is
[Equation]
11) where β is a positive number re-weighting the two objective terms.
Term ˆ I 3 − ˆ I 1 − ˆ I 2 is minimized w.r.t the parameters in encoders E c , E s and decoder D ; term
F ( θ ) as the likelihood function of q θ ( s | c ) is maximized w.r.t the parameter θ .
In practice, the two terms are updated iteratively with gradient descent (by ﬁxing one and updating another).
The training and transfer processes of our model is shown in Figure 1.
We name this MI-guided learning framework as I nformation-theoretic
D isentangled E mbedding for V oice C onversion (IDE-VC).
R ELATED W ORK
Many-to-many Voice Conversion Traditional voice style transfer methods mainly focus on one-toone and many-to-one conversion tasks, which can only transfer voices into one target speaking style.
This constraint limits the applicability of the methods.
Recently, several many-to-many voice conversion methods have been proposed, to convert voices in broader application scenarios.
StarGANVC (Kameoka et al., 2018) uses StarGAN (Choi et al., 2018) to enable many-to-many transfer, in which voices are fed into a unique generator conditioned on the target speaker identity.
A discriminator is also used to evaluate generation quality and transfer accuracy.
Blow (Serr ` a et al., 2019) is a ﬂow-based generative model (Kingma & Dhariwal, 2018), that maps voices from different speakers into the same latent space via normalizing ﬂow (Rezende & Mohamed, 2015).
The conversion is accomplished by transforming the latent representation back to the observation space with the target speaker’s identiﬁer.
Two other many-to-many conversion models, AUTOVC (Qian et al., 2019) and AdaIN-VC (Chou & Lee, 2019), extend applications into zero-shot scenarios, i.e. , conversion from/to a new speaker (unseen during training), based on only a few utterances.
Both AUTOVC and AdaIN-VC construct an encoder-decoder framework, which extracts the style and content of one speech sample into separate latent embeddings.
Then when a new voice from an unseen speaker comes, both its style and content embeddings can be directly extracted.
However, as aforementioned in the Introduction, both methods do not have explicit regularizers to reduce the correlation between style and content embeddings, which limits their performance.
Disentangled Representation Learning Disentangled representation learning (DRL) aims to encode data points into separate independent embedding subspaces, where different subspaces represent different data attributes.
DRL methods can be classiﬁed into unsupervised and supervised
DRL.
Under unsupervised setups, Burgess et al.
(2018), Higgins et al.
and Kim & Mnih (2018) use latent embeddings to reconstruct the original data while keeping each dimension of the embeddings independent with correlation regularizers.
This has been challenged by Locatello et al.
(2019) in that each part of the learned embeddings may not be mapped to a meaningful data attribute.
In contrast, supervised DRL methods effectively learn meaningful disentangled embedding parts by adding different supervision to different embedding components.
Between the two embedding parts, the correlation is still required to be reduced to prevent the revealing of information to each other.
The correlation-reducing methods mainly focus on adversarial training between embedding parts (Hjelm et al., 2018; Kim & Mnih, 2018), and mutual information minimization (Chen et al., 2018; Cheng et al., 2020).
By applying operations such as switching and combining, one can use disentangled representations to improve empirical performance on downstream tasks, e.g. conditional generation (Burgess et al., 2018), domain adaptation (Gholami et al., 2020), and few-shot learning (Higgins et al., 2017).
E XPERIMENTS
We evaluate our IDE-VC on real-world voice dataset under both many-to-many and zero-shot VST setups.
The selected dataset is CSTR Voice Cloning Toolkit (VCTK) (Yamagishi et al., 2019), which includes 46 hours of audio from 109 speakers.
Each speaker reads a different sets of utterances, and the training voices are provided in a non-parallel manner.
The audios are downsampled at 16kHz.
In the following, we ﬁrst describe the evaluation metrics and the implementation details, and then analyze our model’s performance with other baselines under many-to-many and zero-shot
VST settings.
5. E VALUATION M ETRICS
Objective Metrics We consider two objective metrics: Speaker veriﬁcation accuracy ( Veriﬁcation ) and the Mel-Cepstral Distance ( Distance ) (Kubichek, 1993).
The speaker veriﬁcation accuracy measures whether the transferred voice belongs to the target speaker.
For fair comparison, we used a third-party pre-trained speaker encoder Resemblyzer 1 to classify the speaker identity from the transferred voices.
Speciﬁcally, style centroids for speakers are learned with ground-truth voice samples.
For a transferred voice, we encode it via the pre-trained speaker encoder and ﬁnd the speaker with the closest style centroid as the identity prediction.
For the Distance , the vanilla Mel-Cepstral Distance (MCD) cannot handle the time alignment problem described in Section 2.
To make reasonable comparisons between the generation and ground truth, we apply the Dynamic Time Warping (DTW) algorithm
(Berndt & Clifford, 1994) to automatically align the time-evolving sequences before calculating MCD.
This DTW-MCD distance measures the similarity of the transferred voice and the real voice from the target speaker.
Since the calculation of DTW-MCD requires parallel data, we select voices with the same content from the VCTK dataset as testing pairs.
Then we transfer one voice in the pair and calculate DTW-MCD with the other voice as reference.
Subjective Metrics Following Wester et al.
(Wester et al., 2016), we use the naturalness of the speech ( Naturalness ), and the similarity of the transferred speech to target identity ( Similarity ) as subjective metrics.
For Naturalness , annotators are asked to rate the score from 1-5 for each transferred speech.
For the Similarity , the annotators are presented with two audios (the converted speech and the corresponding reference), and are asked to rate the score from 1 to 4.
For both scores, the higher the better.
Following the setting in Blow (Serr ` a et al., 2019), we report Similarity deﬁned as a total percentage from the binary rating.
The evaluation of both subjective metrics is conducted on Amazon Mechanical Turk (MTurk) 2 .
More details about evaluation metrics are provided in the
Supplementary Material.
5. I MPLEMENTATION D ETAILS
Following AUTOVC (Qian et al., 2019), our model inputs are represented as mel-spectrogram.
The number of mel-frequency bins is set as 80.
When voices are generated, we adopt the WaveNet vocoder (Oord et al., 2016) pre-trained on the VCTK corpus to invert the spectrogram signal back to a waveform.
The spectrogram is ﬁrst upsampled with deconvolutional layers to match the sampling rate, and then a standard 40-layer WaveNet is applied to generate speech waveforms.
Our model is implememted with Pytorch and takes 1 GPU day on an Nvidia Xp to train.
Encoder Architecture The speaker encoder consists of a 2-layer LSTM with cell size of 768, and a fully-connected layer with output dimension of 256.
The speaker encoder is initialized with weights from a pretrained GE2E (Wan et al., 2018) encoder.
The input of the content encoder is the concatenation of the mel-spectrogram signal and the corresponding speaker embedding.
The content encoder consists of three convolutional layers with channel number of 512, and two layers of a bidirectional LSTM with cell dimensions 32.
Following the setup in AUTOVC (Qian et al., 2019), the forward and backward outputs of the bi-directional LSTM are downsampled by 16.
Decoder Architecture Following AUTOVC
(Qian et al., 2019), the initial decoder consists of a three-layer convolutional neural network (CNN) with 512 channels, three LSTM layers with cell dimension 1024, and another convolutional layer to project the output of the LSTM to dimension of 80.
To enhance the quality of the spectrogram, following AUTOVC (Qian et al., 2019), we use a post-network consisting of ﬁve convolutional layers with 512 channels for the ﬁrst four layers, and 80 channels for the last layer.
The output of the post-network can be viewed as a residual signal.
The ﬁnal conversion signal is computed by directly adding the output of the initial decoder and the post-network.
The reconstruction loss is applied to both the output of the initial decoder and the ﬁnal conversion signal.
Approximation Network Architecture As described in Section 3.3, minimizing the mutual information between style and content embeddings requires an auxiliary variational approximation q θ ( s | c ) .
For implementation, we parameterize the variational distribution in the Gaussian distribution family q θ ( s | c ) =
N ( µ θ ( c ) , σ 2 θ ( c ) · I ) , where mean µ θ ( · ) and variance σ 2 θ ( · ) are two-layer [Table] [Table] fully-connected networks with tanh( · ) as the activation function.
With the Gaussian parameterization, the likelihoods in objective ˆ I 3 can be calculated with closed form.
5. S TYLE T RANSFER P ERFORMANCE
For the many-to-many VST task [Table] we randomly select 10% of the sentences for validation and 10% of the sentences for testing from the VCTK dataset, following the setting in Blow (Serr ` a et al., 2019).
The rest of the data are used for training in a non-parallel scheme.
For evaluation, we select voice pairs from the testing set, in which each pair of voices have the same content but come from different speakers.
In each testing pair, we conduct transfer from one voice to the other voice’s speaking style, and then we compare the transferred voice and the other voice as evaluating the model performance.
We test our model with four competitive baselines:
Blow (Serr ` a et al., 2019) 3 , AUTOVC (Qian et al., 2019), AdaIN-VC (Chou & Lee, 2019) and StarGAN-VC (Kameoka et al., 2018).
The detailed implementation of these four methods are provided in the Supplementary Material.
Tableshows the subjective and objective evaluation for the many-to-many VST task.
Both methods with the encoder-decoder framework, AdaIN-VC and AUTOVC, have competitive results.
However, our
IDE-VC outperforms the other baselines on all metrics, demonstrating that the style-content disentanglement in the latent space improves the performance of the encoder-decoder framework.
For the zero-shot VST task, we use the same train-validation dataset split as in the many-to-many setup.
The testing data are selected to guarantee that no test speaker has any utterance in the training set.
We compare our model with the only two baselines, AUTOVC (Qian et al., 2019) and AdaINVC (Chou & Lee, 2019), that are able to handle voice transfer for newly-coming unseen speakers.
We used the same implementations of AUTOVC and AdaIN-VC as in the many-to-many VST.
The evaluation results of zero-shot VST are shown in Table 2, among the two baselines AdaIN-VC performs better than AUTOVC overall.
Our IDE-VC outperforms both baseline methods, on all metrics.
All three tested models have encoder-decoder transfer frameworks, the superior performance of IDE-VC indicates the effectiveness of our disentangled representation learning scheme.
More evaluation details are provided in the supplementary material.
5. D ISENTANGLEMENT D ISCUSSION
Besides the performance comparison with other VST baselines, we demonstrate the capability of our information-theoretic disentangled representation learning scheme.
First, we conduct a tSNE (Maaten & Hinton, 2008) visualization of the latent spaces of the IDE-VC model.
As shown [Table] [Table] in the left of Figure 2, style embeddings from the same speaker are well clustered, and style embeddings from different speakers separate in a clean manner.
The clear pattern indicates our style encoder E s can verify the speakers’ identity from the voice samples.
In contrast, the content embeddings (in the right of Figure 2) are indistinguishable for different speakers, which means our content encoder E c successfully eliminates speaker-related information and extracts rich semantic content from the data.
[Table]
We also empirically evaluate the disentanglement, by predicting the speakers’ identity based on only the content embeddings.
A two-layer fully-connected network is trained on the testing set with a content embedding as input, and the corresponding speaker identity as output.
We compare our IDE-VC with AUTOVC and
AdaIN-VC, which also output content embeddings.
The classiﬁcation results are shown in Table 3.
Our IDE-VC reaches the lowest classiﬁcation accuracy, indicating that the content embeddings learned by IDE-VC contains the least speaker-related information.
Therefore, our IDE-VC learns disentangled representations with high quality compared with other baselines.
5. A BLATION S TUDY
[Table]
Moreover, we have considered an ablation study that addresses performance effects from different learning losses in (11), with results shown in Table 4.
We compare our model with two models trained by part of the loss function in (11), while keeping the other training setups unchanged, including the model structure.
From the results, when the model is trained without the style encoder loss term ˆ I 1 , a transferred voice still is generated, but with a large distance to the ground truth.
The veriﬁcation accuracy also signiﬁcantly decreases with no speaker-related information utilized.
When the disentangling term ˆ I 3 is removed, the model still reaches competitive performance, because the style encoder E s and decoder D are well trained by ˆ I 1 and ˆ I 2 .
However, when adding term ˆ I 3 , we disentangle the style and content spaces, and improve the transfer quality with higher veriﬁcation accuracy and less distortion.
The performance without term ˆ I 2 is not reported, because the model cannot even generate ﬂuent speech without the reconstruction loss.
C ONCLUSIONS
We have improved the encoder-decoder voice style transfer framework by disentangled latent representation learning.
To effectively induce the style and content information of speeches into independent embedding latent spaces, we minimize a sample-based mutual information upper bound between style and content embeddings.
The disentanglement of the two embedding spaces ensures the voice transfer accuracy without information revealed from each other.
We have also derived two new multi-group mutual information lower bounds, which are maximized during training to enhance the representativeness of the latent embeddings.
On the real-world VCTK dataset, our model outperforms previous works under both many-to-many and zero-shot voice style transfer setups.
Our model can be naturally extended to other style transfer tasks modeling time-evolving sequences,