L EARNING AND CONTROLLING THE SOURCE - FILTER REPRESENTATION OF SPEECH WITH A VARIATIONAL AUTOENCODER
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding and controlling latent representations in deep generative models is a challenging yet important problem for analyzing, transforming and generating various types of data.
In speech processing, inspiring from the anatomical mechanisms of phonation, the source-ﬁlter model considers that speech signals are produced from a few independent and physically meaningful continuous latent factors, among which the fundamental frequency and the formants are of primary importance.
In this work, we show that the source-ﬁlter model of speech production naturally arises in the latent space of a variational autoencoder (VAE) trained in an unsupervised fashion on a dataset of natural speech signals.
Using only a few seconds of labeled speech signals generated with an artiﬁcial speech synthesizer, we experimentally demonstrate that the fundamental frequency and formant frequencies are encoded in orthogonal subspaces of the VAE latent space and we develop a weakly-supervised method to accurately and independently control these speech factors of variation within the learned latent subspaces.
Without requiring additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms that is conditioned on the fundamental frequency and formant frequencies, and which is applied to the transformation of speech signals.
I NTRODUCTION
High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents their dimensions from varying independently from each other.
This suggests that there exists a latent representation of smaller dimension from which the high-dimensional observed data were generated.
Discovering the hidden properties of complex data is the goal of representation learning, and deep latent-variable generative models have emerged as promising unsupervised approaches (Goodfellow et al., 2014; Kingma & Welling, 2014; Rezende et al., 2014; Chen et al., 2016;
Higgins et al., 2017; Kim & Mnih, 2018;
Chen et al., 2018).
The variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014), which is equipped with both a generative and inference model, can be used not only for data generation but also for analysis and transformation.
As an explicit model of a probability density function (pdf), the VAE can also be used as a learned prior for solving inverse problems such as compressed sensing (Bora et al., 2017), speech enhancement (Bando et al., 2018; Leglaive et al., 2018), or source separation (Kameoka et al., 2019; Jayaram & Thickstun, 2020).
Making sense of the latent representation learned by a VAE and controlling the underlying continuous factors of variation in the data are important challenges to build more expressive and interpretable generative models and probabilistic priors.
Previous works on representation learning with deep generative models, in particular VAEs, have mostly focused on images (Higgins et al., 2017; Kim & Mnih, 2018;
Chen et al., 2018; Locatello et al., 2019; 2020).
Yet, it is not always easy to deﬁne the ground-truth latent factors of variation involved in the generation of natural images.
For speech data, the latent factors of variation can be directly related to the anatomical mechanisms of speech production.
This makes speech data interesting for investigating the disentangled representation learning capabilities of VAEs, complementary to studies dealing with images.
A key concept for characterizing the structure of speech signals is deduced from the source-ﬁlter model proposed by Fant (1970).
This model, described in more detail in Section 2.2, implies that a speech signal is mainly characterized by a few continuous latent factors of variation corresponding to the vibration of the vocal folds (i.e., the source), which deﬁnes the fundamental frequency, and the resonances of the vocal tract (i.e., the ﬁlter), which deﬁne the formants.
The source-ﬁlter model is at the core of various fundamental speech processing techniques such as cepstral representations and linear predictive coding (LPC) (Rabiner & Schafer, 2010).
Valin & Skoglund (2019); Wang et al. and Juvela et al. (2019) have recently shown that the efﬁciency of neural speech vocoders can be largely improved by leveraging the sourceﬁlter model.
Other works investigating the interaction between the source-ﬁlter model and neural networks include Lee et al.
and Choi et al.
All these studies illustrate the interest of combining deep learning techniques with more traditional signal processing models and algorithms.
In this work, we interpret and control the latent space of a VAE from the perspective of the source-ﬁlter model of speech production, which can be beneﬁcial for various applications in speech analysis, transformation, and synthesis.
We ﬁrst train a VAE on a dataset of about 25 hours of unlabeled speech signals.
Then, using only a few seconds of labeled speech signals generated with an artiﬁcial speech synthesizer, we propose a method to analyze and control the fundamental frequency and the formant frequencies in the latent representation of the previously trained VAE.
Our contributions are the following: (i) We experimentally demonstrate that the fundamental frequency and the frequency of the ﬁrst three formants are encoded in orthogonal subspaces of the VAE latent space.
This shows that a vanilla VAE trained in an unsupervised fashion is able to learn a representation that is compliant with the source-ﬁlter model of speech production.
(ii) We develop a weakly-supervised method to precisely and independently control the source-ﬁlter continuous latent factors of speech variation within the learned subspaces.
We put in evidence the orthogonality of these subspaces, which allows us to perform speech transformations in a disentangled manner (i.e., modifying one of the factors does not affect the others).
(iii) Without requiring additional information such as text or human-labeled data, we propose a deep generative model of speech spectrograms conditioned on the fundamental frequency and formant frequencies.
To the best of our knowledge, this is the ﬁrst study showing the link between the classical source-ﬁlter model of speech production and the representation learned in the latent space of a VAE.
Thanks to this link, we propose a principled method to generate speech data controlled with interpretable trajectories (of e.g., fundamental frequency and formant frequencies).
B ACKGROUND 2. V ARIATIONAL AUTOENCODER
Generative modeling consists in learning a probabilistic model of an observable random variable x ∈ X ⊂ R D .
Let D = { x 1 , ..., x N ∈ X} be a dataset of N = # D independent and identically distributed (i.i.d.)
observations of x .
The empirical distribution of x is deﬁned by ˆ p ( x )
= 1 N (cid:80) x n ∈D δ ( x − x n ) , where δ is the Dirac delta function, which is null everywhere except in 0 where it takes the value 1.
The variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) attempts to approximate ˆ p ( x ) with a pdf p θ ( x ) parametrized by θ .
High-dimensional data such as natural images or speech signals exhibit some form of regularity which prevents the D dimensions of x from varying independently from each other.
We can thus assume that there exists a latent variable z ∈ R L , with
L (cid:28) D , from which the observed data were generated.
Accordingly, the model distribution in the VAE is deﬁned by marginalizing the joint distribution of the latent and observed data, p θ ( x ) = (cid:82) p θ ( x | z ) p ( z ) d z .
In this work, the observed data vector x ∈ R D + denotes the power spectrum of a short frame of speech signal (i.e., a column of the short-time Fourier transform (STFT) power spectrogram).
Its entries are non negative and its dimension D equals the number of frequency bins.
We use the
Itakura-Saito VAE (IS-VAE)
(Bando et al., 2018; Leglaive et al., 2018; Girin et al., 2019) deﬁned by
[Equation]
where N and Exp denote the densities of the multivariate Gaussian and univariate exponential distributions, respectively, and [ v ] d denotes the d -th entry of v .
The inverse scale parameters of p θ ( x | z ) are provided by a neural network called the decoder, parametrized by θ and taking z as input.
The marginal likelihood p θ ( x ) and the posterior distribution p θ ( z | x ) are intractable due to the non linearities of the decoder, so it is necessary to introduce an inference model q φ ( z | x )
≈ p
θ ( z | x ) , which in the VAE is usually deﬁned by
[Equation]
where the mean and variance parameters are provided by a neural network called the encoder network, parametrized by φ and taking x as input.
Then, the VAE training consists in maximizing a lower-bound of ln p θ ( x ) , called the evidence lower-bound (ELBO) and deﬁned by
L ( θ, φ ) =
E ˆ p ( x ) (cid:2)
E q φ ( z | x ) [ p θ ( x | z )] − D KL ( q φ ( z | x ) (cid:107) p ( z )) (cid:3) .
During training, the generative and inference model parameters θ and φ are jointly estimated by maximizing the ELBO, using (variants of) stochastic gradient descent with the so-called reparameterization trick (Kingma & Welling, 2014; Rezende et al., 2014).
2. S OURCE - FILTER MODEL OF SPEECH PRODUCTION
The source-ﬁlter model of speech production (Fant, 1970) is at the basis of many speech processing systems.
It considers that the production of speech results from the interaction of a source signal with a linear ﬁlter.
In voiced speech, the source originates from the vibration of the vocal folds, which produces a quasi-periodic glottal sound wave whose fundamental frequency deﬁnes the pitch .
In unvoiced speech, a noise source is produced by a turbulent airﬂow or an acoustic impulse.
The source signal is modiﬁed by the vocal tract, which is assumed to act as a linear ﬁlter.
The cavities of the vocal tract give rise to resonances, which are called the formants and are characterized by their frequency, amplitude and bandwidth.
By moving the speech articulators such as the tongue, lips, and jaw, humans modify the shape of their vocal tract, which results in a change of the acoustic ﬁlter and the associated resonances.
This is how the different elementary speech sounds called phonemes are produced to form syllables, words and sentences.
The power spectra and the spectral envelopes of two French vowels are displayed in Figure 1.
The spectral envelopes show that the formant frequencies are different for the two vowels.
In this example however, the harmonic structure of the spectra shows that the fundamental frequency is the same for the two vowels.
Formant frequencies are important distinctive features of vowels.
In a ﬁrst approximation, they can be related to the opening of the mouth, the front/rear position of the tongue, and the rounding of the lips for the ﬁrst, second, and third formant respectively.
For voiced phonemes, humans are able to control the formants independently of the pitch (i.e., to change the ﬁlter independently of the source (Fant, 1970)) and of each other (MacDonald et al., 2011).
The independence of the source and ﬁlter characteristics makes the speech signals an interesting material for representation learning methods, especially with deep generative latent-variable models.
In the present study, in addition to the pre-trained IS-VAE speech spectrogram model, we also assume the availability of an artiﬁcial speech synthesizer allowing for an accurate and independent control of the fundamental frequency and formants.
In this work, we use Soundgen (Anikin, 2019), a parametric synthesizer based on the source-ﬁlter model of speech production.
For a given speech sound, the voiced component of the source signal is generated by a sum of sine waves, the noise component by a ﬁltered white noise, and both components are then summed and passed through a linear ﬁlter simulating the effect of the human vocal tract.
Importantly, this synthesizer allows us to easily generate artiﬁcial speech data labeled with the fundamental frequency and formant frequency values.
A NALYZING AND CONTROLLING SOURCE - FILTER FACTORS OF SPEECH VARIATION IN THE VAE
In this section, from a VAE trained on natural speech signals and a few artiﬁcially-generated labeled speech signals, we propose (i) a method to learn latent subspaces encoding source-ﬁlter factors of speech variation, (ii) a simple strategy to measure the disentanglement of the learned representation, and (iii) a weakly-supervised approach to control the continuous factors of variation in the learned subspaces and generate corresponding speech signals.
Let f i denote the speech factor of variation (in Hz) corresponding to the fundamental frequency, for i = 0 , and to the formant frequencies, for i ∈ { 1 , 2 , ... } .
Let D i denote a dataset of artiﬁciallygenerated speech vectors (more precisely short-term power spectra) synthesized by varying only f i , [Figure]
[Figure]
) m a g n i t u d e ( d B ) /a/ /i/
[Figure]
all other factors { f j , j (cid:54) = i } being arbitrarily ﬁxed. All examples in D i are labeled with the factors of variation.
It would be very difﬁcult to build such a dataset from existing corpora of unlabeled natural speech.
In contrast, it is an easy task using an artiﬁcial speech synthesizer such as Soundgen, which precisely takes the fundamental frequency and formant parameters as input, and outputs waveforms from which we extract power spectra. 3. L EARNING [Figure]
ENCODING
[Figure] FILTER
[Figure] Let ˆ p ( i ) ( x ) denote the empirical distribution associated with D i , deﬁned similarly as ˆ p ( x ) .
We also introduce the following marginal distribution over the latent vectors:
[Equation]
In the literature, this quantity is referred to as the aggregated posterior (Makhzani et al., 2016).
However, q φ ( z | x ) is usually aggregated over the empirical distribution ˆ p ( x ) such that the aggregated posterior is expected to match with the prior p ( z ) (Chen et al., 2018; Dai & Wipf, 2018).
In contrast, in Equation (3) we aggregate over the “biased” data distribution ˆ p ( i ) ( x ) , where we know only one latent factor varies.
This deﬁnes the explicit inductive bias (Locatello et al., 2019) that we exploit to learn the latent source-ﬁlter representation of speech in the VAE.
In the following of the paper, without loss of generality, we assume that, for each data vector in D i , the associated latent vector z has been centered by subtracting the mean vector
[Equation]
Because only one factor varies in D i , we expect latent vectors drawn from the “biased” aggregated posterior in Equation (3) to live on a lower-dimensional manifold embedded in the original latent space R L .
We assume this manifold to be a subspace characterized by its semi-orthogonal basis matrix
U i ∈
R L × M i , 1 ≤ M i < L .
This matrix is computed by solving the following optimization problem:
[Equation]
The space spanned by the columns of U i is a subspace of the original latent space R L in which the latent vectors associated with the variation of the factor f i in D i are expected to live.
In Appendix A, we show that, similarly to principal component analysis (PCA) (Pearson, 1901), the solution to the optimization problem (5) is given by the M i eigenvectors corresponding to the M i largest eigenvalues of
[Equation]
The dimension M i of the subspace can be chosen such as to retain a certain percentage of the data variance in the latent space.
Note that the only source of supervision used here is the knowledge that only the factor f i varies in the dataset D i .
3. D ISENTANGLEMENT ANALYSIS OF THE LATENT REPRESENTATION
As deﬁned by Higgins et al.
(2018), a representation is disentangled if it is possible to learn orthogonal latent subspaces associated with each factor of variation, whether they are single- or multidimensional.
The approach presented in the previous subsection exactly follows this deﬁnition and offers a natural and straightforward way to objectively measure if the unsupervised VAE managed to learn a disentangled representation of the factors of variation under consideration.
First, by simply looking at the eigenvalues associated with the columns of U i ∈ R L × M i , we can measure the amount of variance that is retained by the projection U
i U (cid:62) i .
If a small number of components M i represents most of the variance, it indicates that only a few intrinsic dimensions of the latent space are dedicated to the factor of variation f i and varying this factor can be done by afﬁne transformations.
Second, if for two different factors of variation f i and f j , with i (cid:54) =
j , the columns of U i are orthogonal to those of U j , this indicates that the two factors are encoded in orthogonal subspaces and therefore disentangled.
It should however be veriﬁed experimentally that applying transformations by moving onto the subspace associated with f i generalizes to values of { f j , j (cid:54) =
i } different than the ones used in D i .
3. C ONTROLLING THE FACTORS OF VARIATION FOR SPEECH TRANSFORMATION AND GENERATION
So far, for each factor f i , we have deﬁned a methodology to learn a latent subspace
U i ∈
R L × M i that encodes its variations in the dataset D i , containing a few examples of speech data generated by an artiﬁcial synthesizer.
Making now use of the labels in D
i , we learn a regression model g η i : R
+ (cid:55)→ R M i from the factor f i , whose value is denoted by y ∈ R + , to the data coordinates in the latent subspace deﬁned by U i .
The parameters η i are deﬁned as the solution of the following optimization problem:
[Equation]
where ˆ q ( i ) φ ( z , y )
= (cid:82) q φ ( z | x )ˆ p ( i ) ( x , y )
d x , ˆ p ( i ) ( x , y ) is the empirical distribution associated with
D i , considering now both the speech data vector x and the label y , and c = denotes equality up to an additive constant w.r.t.
η .
The dataset D i is very small with only a few hundreds examples, and as it is synthetic and labels are not provided by human annotators, the problem can be considered very weakly supervised.
For simplicity and because it revealed efﬁcient for this task, g η i is chosen as a piece-wise linear regression model learned independently for each output coordinate m ∈ { 1 , ..., M i } .
This choice is supported by the fact that the semi-orthogonal matrix U i decorrelates the data (Bengio et al., 2013).
Solving the optimization problem (7) then basically consists in solving a linear system of equations (Jekel & Venter, 2019).
We can now transform a speech spectrogram by analyzing it with the VAE encoder, then linearly moving in the learned subspaces using the above regression model, and ﬁnally resynthesizing it with the VAE decoder.
Given a source latent vector z and a target value y for the factor f i , we apply the following afﬁne transformation:
[Equation]
This transformation consists in (i) subtracting the projection of z onto the subspace associated with the factor of variation f i ; and (ii) adding the target component provided by the regression model g η i mapped from the learned subspace to the original latent space by the matrix
U i .
This operation allows us to move only in the latent subspace associated with the factor f i .
If this subspace is orthogonal to the latent subspaces associated with the other factors { f j , j (cid:54) =
i } , the latter should remain the same between z and ˜ z , only f i should be modiﬁed.
This process can be straightforwardly generalized to multiple factors, by subtracting and adding terms corresponding to each one of them.
Finally, as the prior p ( z ) and inference model q φ ( z | x ) are Gaussian (see Equations (1) and (2)), the transformation in Equation (8) has the following probabilistic formulation (using U (cid:62)
i U i
=
I ):
[Equation] [Equation]
The prior in Equation (9) is now conditioned on the factor f i and can be used to generate speech data given input trajectories of fundamental frequency and formant frequencies.
As we assumed centered latent data, the mean vector µ φ ( D i ) deﬁned in Equation (4) must be added to ˜ z before mapping this vector through the generative model p θ ( x | z ) .
[Figure]
[Figure]
E XPERIMENTS
This section presents qualitative and quantitative experimental results of the proposed method for controlling the fundamental frequency and formant frequencies of speech signals with a VAE.
The
VAE is trained on about 25 hours of multi-speaker speech data from the Wall Street Journal (WSJ0) dataset (Garofolo et al., 1993a).
The data space dimension is 513 and the latent space dimension is 16.
For a given factor of variation, the corresponding latent subspace is learned (see Section 3.1) using short trajectories of speech power spectra (corresponding to a few seconds of speech) generated with Soundgen (Anikin, 2019), all other factors being arbitrarily ﬁxed.
When solving the optimization problem (5), the latent subspace dimension M i of each factor of variation is chosen such that 80% of the data variance is retained.
This leads M
= 4 , M 1 = 1 and M 2 =
M 3 = 3 .
The regression models used to control the speech factors of variation in the latent space (see Section 3.3) are learned on the same trajectories, but using the labels that correspond to the input control parameters of Soundgen (i.e., fundamental frequency and formant frequencies values).
More details on the experimental set-up can be found in Appendix B. Given a generated or transformed spectrogram, we use Waveglow (Prenger et al., 2019) to reconstruct the time-domain signal.
4. Q UALITATIVE RESULTS
In Figure 2a, we illustrate the ability of the proposed method to modify the fundamental frequency and formant frequencies in an accurate and independent manner.
The spectrogram contains ﬁve segments of equal length.
The ﬁrst segment corresponds to the original spectrogram of the steady vowel /a/ uttered by a female speaker.
In the following segments, we vary successively each individual factor f i , for i = 0 to 3 , as indicated by the black lines in the ﬁgure.
Variations of f 0 modify the harmonic structure of the signal while keeping the formant structure unaltered.
Variations of f i , i ∈ { 1 , 2 , 3 } , modify the formant frequencies (i.e., the vocal tract resonances, as indicated by the color map) while keeping the fundamental frequency unaltered.
Figure 2b represents a spectrogram generated by using the conditional prior in equation (9) (generalized to conditioning on multiple factors).
We can see that the characteristics of the generated speech spectrogram match well with the input trajectories represented by the lines in the ﬁgure.
In Figure 2c, from left to right we show the original spectrogram of a speech signal uttered by a female speaker (left), and the transformed spectrograms where the fundamental frequency is set constant over time (middle) and where the pitch has been removed (i.e., the original voiced speech signal is converted into a whispered speech signal) (right). This last spectrogram is simply obtained by subtracting to z its projection onto the latent
[Table]
subspace corresponding to f 0 (i.e., by considering only the two ﬁrst terms in the right-hand side of
Equation (8)).
It results in a spectrogram where the harmonic component is neutralized, while preserving the original formant structure.
This is remarkable considering that the VAE was not trained on whispered speech signals, and it further conﬁrms that the proposed method dissociates the source and the ﬁlter contributions in the VAE latent space.
Other qualitative results can be found in Appendix C (visualization of trajectories in the learned latent subspaces and additional examples of generated and transformed speech spectrograms) and at https://tinyurl.com/iclr (including audio examples).
4. Q UANTITATIVE RESULTS
Due to space limitations, we only provide in this section quantitative results related to the fundamental frequency ( f 0 ) and the ﬁrst two formant frequencies ( f 1 and f 2 ).
Additional results can be found in Appendix D, including the performance for modiﬁcations of the third formant frequency ( f 3 ), the performance with a VAE trained on other datasets (e.g., a dataset of French speech signals), and the performance on the TIMIT dataset (Garofolo et al., 1993b), which is richer than the English vowel one in terms of phonemes but is not labeled with the fundamental and formant frequencies, making the evaluation less reliable.
4.2. O RTHOGONALITY OF THE LATENT SUBSPACES
Following the discussion in Section 3.2, we compute the dot product between all pairs of unit vectors in the matrices U
i ∈
R L × M i , i ∈ { 0 , 1 , 2 } .
Figure 3 shows that the resulting correlation matrix is mainly diagonal.
Except for a correlation value of − 0 .
23 across f 1 and the ﬁrst component of f 2 , all other values are below 0 .
13 (in absolute value), conﬁrming the orthogonality of the learned subspaces and thus the disentanglement of the learned source-ﬁlter representation of speech.
4.2. P ITCH AND FORMANT TRANSFORMATIONS OF E NGLISH VOWELS Experimental set-up
In this experiment, we quantitatively evaluate the performance of the proposed method regarding the modiﬁcation of the fundamental frequency and formant frequencies in speech signals (see Section 3.3).
We use a corpus of 12 English vowels uttered by 50 male andfemale speakers (Hillenbrand et al., 1995).
This dataset is labeled with the fundamental and formant frequencies.
We transform each signal in this dataset by varying one single factor f i at a time, for i ∈ { 0 , 1 , 2 } .
The ranges of transformation for the fundamental frequency, ﬁrst formant frequency, and second formant frequency are respectively [100 , 300] Hz, [300 , 900]
Hz, and [1100 , 2700]
Hz, with a step of 1 Hz, 10 Hz and 20 Hz respectively.
For the modiﬁcation of each factor f i , we measure the performance regarding three aspects: First, in terms of accuracy by comparing the target value for the factor (see Equation (8)) and its estimation computed from the modiﬁed output speech signal.
Second, in terms of disentanglement , by comparing the values of f j for j (cid:54) =
i , before and after modiﬁcation of the factor f i .
Third, in terms of speech naturalness of the transformed signal.
Metrics
Accuracy and disentanglement are measured in terms of relative absolute error (in percent, the lower the better).
For a given factor f i , it is deﬁned by δf i = 100% × | ˆ y − y
| /y where y is the target value of f i and ˆ y its estimation from the output transformed signal.
Let us take the example of a modiﬁcation of f 0 : δf 0 measures the accuracy of the transformation on f 0 while δf 1 and δfare used to assess if the other factors of variation f 1 and f 2 remained unchanged after modifying f 0 .
We use CREPE (Kim et al., 2018) to estimate the fundamental frequency and Parselmouth (Jadoul et al., 2018), which is based on PRAAT, to estimate the formant frequencies.
Regarding speech naturalness, we use the objective measure provided by NISQA (Mittag & M¨oller, 2020).
This metric (the higher the better) was developed in the context of speech transformation algorithms and it was shown to highly correlate with subjective mean opinion scores (MOS) (i.e., human ratings).
As a reference, the score provided by NISQA on the original dataset of English vowels (i.e., without any processing) is equal to 2.60 ± 0.53.
Methods
We compare the proposed approach with several methods from the literature: (i) Timedomain pitch-synchronous overlap-and-add (TD-PSOLA) (Moulines & Charpentier, 1990) performs fundamental frequency modiﬁcations through the decomposition of the signal into pitchsynchronized overlapping frames.
(ii) WORLD (Morise et al., 2016) is a vocoder also used for fundamental-frequency modiﬁcations.
It decomposes the speech signal into three components characterizing the fundamental frequency, the aperiodicity, and the spectral envelope.
(iii)
The method proposed by Hsu et al.
(2017b)
(here referred to as “VAE baseline”) consists in applying translations directly in the latent space of the VAE.
Unlike the proposed approach, this method requires predeﬁned latent attribute representations µ src and µ trgt associated with the source and target values of the factor to be modiﬁed, respectively.
In particular, computing µ src requires analyzing the input speech signal, for instance to estimate the fundamental frequency, which is not the case for the proposed method.
The source and target latent attribute representations are then used to perform the translation ˜ z
= z − µ src + µ trgt , where z and ˜ z are respectively the original and modiﬁed latent vectors.
To ensure fair comparison, we build dictionaries of predeﬁned latent attribute representations using the same artiﬁcially-generated speech data that were used in the weakly-supervised training stage of the proposed method.
All the methods we compare with require a pre-processing of the input speech signal to compute the input trajectory of the factor to be modiﬁed (e.g. the fundamental frequency), which is not the case of the proposed method.
Discussion
Experimental results (mean and standard deviation) are shown in Table 1.
Compared to the VAE baseline, the proposed method obtains better performance in terms of accuracy, disentanglement, and naturalness.
These results conﬁrm the effectiveness of performing the transformations in the learned latent subspaces and not directly in the latent space, as well as the advantage of using regression models instead of predeﬁned latent attribute representations.
Regarding f 0 transformation, WORLD obtains the best performance in terms of disentanglement, which is because the source and ﬁlter contributions are decoupled in the architecture of the vocoder.
In terms of naturalness, WORLD and then TD-PSOLA obtain the best performance.
This may be explained by the fact that these method operate directly in the time domain, therefore they do not suffer from phase reconstruction artifacts, unlike the proposed and VAE baseline methods.
Naturalness is indeed greatly affected by phase reconstruction artiﬁcats, even from an unaltered speech spectrogram (i.e., without transformation).
Phase reconstruction in a multi-speaker setting is still an open problem in speech processing.
We want to emphasize that the objective of this study is not to compete with traditional signal processing methods such as TD-PSOLA and WORLD.
It is rather to advance on the understanding of deep generative modeling of speech signals and to compare honestly with highly-specialized traditional systems.
TD-PSOLA and WORLD exploit signal models speciﬁcally designed for the task at hand, which for instance prevents them to be used for modifying formant frequencies.
In contrast, the proposed method is fully based on learning and the same methodology applies for modifying the fundamental or formant frequencies.
R ELATED WORK
Variants of the VAE have recently led to considerable progress in disentangled representation learning (Kim & Mnih, 2018; Higgins et al., 2017; Chen et al., 2018).
From experimental analyses on image data, these methods suggest that a vanilla VAE cannot learn a disentangled representation.
In the present study, we experimentally show that learning a disentangled source-ﬁlter representation of speech using a simple VAE is possible, complying with the deﬁnition of disentanglement proposed in (Higgins et al., 2018).
Locatello et al.
(2019; 2020) recently showed both theoretically and from a large-scale experimental study that the unsupervised learning of disentangled representations is impossible without inductive biases on both the models and the data.
In the present study, we precisely employ a few examples of artiﬁcially-generated labeled speech data in order to disentangle the latent representation of a simple VAE, in terms of source-ﬁlter factors of speech variation.
Modifying speech signals characteristics such as the pitch or timbre is a highly covered research topic where deep generative modeling has recently emerged as a promising approach.
Voice conversion consists in modifying speaker characteristics while keeping the linguistic information unchanged, e.g. (Kaneko & Kameoka, 2018; Kaneko et al., 2019; Kameoka et al., 2020).
Many voice conversion methods use text information as input (Wang et al., 2018;
Zhang et al., 2019; Kulkarni et al., 2021; Li et al., 2021), which greatly helps for transferring non-verbal vocal attributes from one speech signal to another, but it requires datasets labeled with text.
Qian et al.
(2020) and Webber et al.
(2020) recently proposed speech transformation methods based on latent representation learning without resorting to textual information.
The interpretability of the learned representations is enforced by the design of the model.
Qian et al.
(2020) proposed to use three independent encoder networks to decompose a speech signal into fundamental frequency, timbre and rhythm latent representations.
Webber et al.
(2020) focused on controlling source-ﬁlter parameters in speech signals, where the ability to control a given parameter (e.g. fundamental frequency) is enforced explicitly using labeled data and adversarial learning.
In this approach, each parameter to be controlled requires a dedicated training of the model.
Moreover, these methods are speaker-dependent, as speech generation in Qian et al.
(2020) is conditioned on the speaker identity and Webber et al.
(2020) used a single-speaker training dataset.
This contrasts with the proposed method which is speakerindependent, and in which the source-ﬁlter representation of speech naturally emerges in the latent space of a single unsupervised VAE.
Hsu et al.
(2017a;b) proposed to use VAEs for modifying the speaker identity and the phonemic content of speech signals.
In (Hsu et al., 2017b), this is achieved by translations in the latent space.
This is the baseline VAE approach we presented and compared with in Section 4.
It requires to know predeﬁned values of the latent representations associated with the source/target speech attributes to be modiﬁed.
Conversely, Equation (8) in the proposed method does not require the knowledge of the factor f i associated with source vector z , it only requires the value associated with the target vector ˜ z .
Moreover, Hsu et al.
(2017b) did not address the control of continuous factors of speech variation in the VAE latent space, contrary to the present work.
Several methods have been recently proposed to control continuous factors of variation in deep generative models (Jahanian et al., 2019; Plumerault et al., 2020; Goetschalckx et al., 2019;
H ¨ ark ¨ onen et al., 2020), focusing essentially on generative adversarial networks.
They consist in identifying and then moving onto semantically meaningful directions in the latent space of the model.
The present work is inspired by (Plumerault et al., 2020), which assumes that a factor of variation can be predicted from the projection of the latent vector along a speciﬁc axis, learned from artiﬁcially generated trajectories.
The proposed method is however more generic, thanks to the learning of latent subspaces associated to the latent factors and to the introduction of a general formalism based on the use of “biased” aggregated posteriors.
Moreover, these previous works on controlling deep generative models only allow for moving “blindly” onto semantically meaningful directions in the latent space.
In the present study, we are able to generate data conditioned on a speciﬁc target value for a given factor of variation.
Finally, previous work focused on image data, and to the best of our knowledge, this is the ﬁrst approach to control source-ﬁlter factors of speech variation in a VAE.
C ONCLUSION
In this work, using only a few seconds of artiﬁcially generated labeled speech data, we showed that the fundamental frequency and formant frequencies are encoded in orthogonal latent subspaces of an unsupervised VAE and we proposed a weakly-supervised method to control these attributes within the learned subspaces. The method generalizes well when applied to natural speech signals.
To the best of our knowledge, this is the ﬁrst approach that, with a single methodology, is able to extract, identify and control the source and ﬁlter low-level speech attributes within a VAE latent space.
The proposed method could also be applied to other types of data such as natural or face images, provided that one can create a few synthetic images that capture variations in a single latent factor of interest, independently of others. For example, this would be possible for factors encoding the position, rotation and scale of objects in natural images, or some facial expressions.
R EPRODUCIBILITY S TATEMENT
To ensure reproducibility, the implementation of the method and the data generated with soundgen that are used to learn the latent subspaces and regression models will be made freely available online.
A S OLUTION TO THE LATENT SUBSPACE LEARNING PROBLEM
In this Appendix, we show that the solution to the optimization problem (5) is given by the principal eigenvectors of S
φ ( D i ) in Equation (6).
Without loss of generality
[Equation]
we formulate the problem for a centered version of the latent data
[Equation]
) where µ φ ( D i ) is deﬁned in Equation (4).
This centering also affects the inference model originally deﬁned in Equation (2), as follows:
[Equation]
Using Equation (3)
[Equation]
the fact that U (cid:62) U = I , and Equation (12)
[Equation]
the cost function in the optimization problem (5) can be rewritten as follows:
[Equation]
where S φ ( D i ) is deﬁned in Equation (6).
From this last equality [Equation] we see that the optimization problem (5) is equivalent to
[Equation]
Very similarly to PCA (Pearson, 1901), the solution is given by the M i dominant eigenvectors of
S φ ( D i ) (i.e., associated to the M i largest eigenvalues) (Bishop, 2006, Section 12.1).
B E XPERIMENTAL SETUP B. VAE TRAINING
To train the IS-VAE model (Bando et al., 2018; Leglaive et al., 2018; Girin et al., 2019), we use the Wall Street Journal (WSJ0) dataset (Garofolo et al., 1993a) which contains 25 hours of speech signals sampled at 16 kHz, including 52 female and 49 male speakers.
The time-domain spech signals are converted to power spectrograms using the short-time Fourier transform (STFT) with an analysis window of length 64 ms and an overlap of 75%.
The architecture of the VAE is shown in Figure 4, the encoder and decoder networks each have three dense layers.
A hyperbolic tangent ( tanh ) activation function is used at each layer, except for the output layers of the encoder and decoder where we use the identity function.
The VAE input/output dimension is D = 513 (we only keep the non redundant part of the power spectrum at a given time frame) and the latent vector dimension is set to L = 16 .
We train the model using the Adam optimizer (Kingma & Ba, 2015) with a learning rate equal to 0.001.
B. A NALYZING AND CONTROLLING SOURCE - FILTER FACTORS OF SPEECH VARIATION IN THE VAE
For a given factor of variation, the corresponding latent subspace is learned (see Section 3.1) using trajectories of speech power spectra generated with Soundgen (Anikin, 2019), all other factors being arbitrarily ﬁxed.
For f 0 , the trajectory contains 226 points (which corresponds to 3.6 seconds of speech) evenly spaced in the range [85 , 310] Hz, f 1 , f 2 and f 3 are ﬁxed to 600 Hz, 1500 Hz, and 3200 Hz, respectively.
For f 1 , the trajectory contains 401 points (which corresponds to 6.4 seconds of speech) evenly spaced in the range [200 , 1000]
Hz, f 0 , f 2 and f 3 are ﬁxed to 140 Hz, 1600 Hz, and 3200 Hz, respectively.
For f 2 , the trajectory contains 401 points evenly spaced in the range [800 , 2800]
Hz, f 0 , f 1 and f 3 are ﬁxed to 140 Hz, 500 Hz, and 3200 Hz, respectively.
For f 3 , the trajectory contains 241 points (which corresponds to 3.9 seconds of speech) evenly spaced in the [Figure] range [2000 , 3200]
Hz, f 0 , f 1 and f 2 are ﬁxed to 140 Hz, 500 Hz, and 1200 Hz, respectively.
The amplitude of the formants is set to 30 dB, and the bandwidth is automatically calculated from the fromant frequencies by Soundgen, using a formula derived from human phonetic research (Anikin, 2019).
The regression models used to control the speech factors of variation in the latent space (see Sections 3.3) are learned on the same trajectories, but using the labels that correspond to the input control parameters of Soundgen.
C A DDITIONAL QUALITATIVE RESULTS C. V ISUALIZATION OF THE LEARNED LATENT SUBSPACES
For i = 0 , 1 , 2 and 3 , Figures 5a, 5b, 5c, 5d are respectively obtained by projecting the latent mean vectors µ φ ( x ) ∈ R L , for all data vectors x ∈ D
i , within the latent subspace characterized by
U i ∈
R L × M i (i.e., we perform dimensionality reduction).
In the previously reported experiments, the latent subspace dimension M i of each factor of variation was chosen such that 80% of the data variance was retained in the latent space.
It resulted in M
= 4 , M 1 = 1 and M 2 =
M 3 = 3 .
In this section, for visualization purposes, we set M i = 3 for all i ∈ { 0 , 1 , 2 , 3 } .
However, we can see that the f 1 trajectory (Figure 5b) is mainly concentrated along a single axis.
Regarding f 0 (Figure 5a), setting M
= 3 retained 78 % of the variance of D 0 in the latent space.
An important observation that we make from these ﬁgures is that two data vectors x and x (cid:48) , corresponding to values of a given factor that are close, have projections of µ φ ( x ) and µ φ ( x (cid:48) ) that are also close in the learned latent subspaces.
This can be seen from the color bars which indicate the values of the factors of variation.
It indicates that the learned representation preserves the notion of proximity in terms of fundamental frequency and formant frequencies.
In Figure 5e, we project three different datasets D 1 , deﬁned for three different values of f 2 .
Similarly, in Figure 5f we show the trajectories associated with the projection of three datasets D 2 , deﬁned for three different values of f 1 .
We notice that as expected, the trajectories are almost identical and only differ by a translation.
[Figure] [Figure]
E XAMPLES OF GENERATED SPEECH
[Figure] [Figure]
[Figure]
D A DDITIONAL QUANTITATIVE RESULTS D. T RANSFORMATIONS OF THE THIRD FORMANT FREQUENCY ON THE E NGLISH VOWELS DATASET
Table 2 completes Table 1 by including transformations of the third formant frequency in the range [2200 , 3200]
Hz, with a step of 20 Hz.
The new column δf 3 also measures the performance in terms of disentanglement when modifying other factors
f i (cid:54) =
f 3 .
These additional results do not modify the conclusions drawn in Section 4.
[Table] D. R OBUSTNESS WITH RESPECT TO DIFFERENT TRAINING DATASETS FOR THE VAE This Section investigates the robustness of the proposed method with respect to different datasets used to train the VAE model.
This table presents results for modiﬁcations of the fundamental frequency only, applied to the English vowels dataset.
We considered the following training datasets:
• WSJ: the Wall street Journal (Garofolo et al., 1993a) dataset that was used in the previous experiments.
• SIWIS: the SIWIS French speech synthesis dataset (Honnet et al., 2017), which contains more than ten hours of French speech recordings.
• TESS: the Toronto emotional speech dataset (Dupuis & Pichora-Fuller, 2010), which contains 2800 utterances spoken by two actresses using different emotions (anger, disgust, fear, happiness, pleasant surprise, sadness, and neutral).
• LJspeech: the LJspeech dataset (Ito & Johnson, 2017), which contains 13100 short audio clips of a single speaker reading passages from 7 non-ﬁction books.
The artiﬁcially-generated speech dataset used for learning the latent subspaces and the regression models along with the test dataset of English vowels remain the same.
It can be seen in Tablethat the performance remains quite stable with different VAE training datasets.
WSJ is the largest dataset and therefore obtains the best performance.
Interestingly, the results obtained with the SIWIS dataset of French speech signals remain satisfactory, even if there is a mismatch between the training (French) and testing (English) datasets.
[Table]
D. E VALUATION ON THE TIMIT DATASET
In this Section, we evaluate the proposed method on the TIMIT dataset (Garofolo et al., 1993b), using the VAE trained on the WSJ dataset. TIMIT is a corpus of phonemically and lexically transcribed speech of American English speakers of different sexes and dialects. We used the test corpus containing 1680 utterances. Because we are interested in studying the interaction between modiﬁcations of the fundamental frequency and formant frequencies, we only evaluate the method on the phonemes that are voiced (40 phonemes over a total of 52), which can be identiﬁed using the annotations.
The ranges of transformation for the fundamental frequency, ﬁrst, second and third formant frequencies are respectively [100 , 300]
Hz, [300 , 900] Hz, [1100 , 2700]
Hz, and [2200 , 3200]
Hz with a step of 10 Hz, 50 Hz, 100 Hz and 50 Hz, respectively.
The metrics and evaluated methods remain the same as those described in Section 4.2.2.
TIMIT is phonemically richer than the English vowels dataset previously used, however it is not labeled with the fundamental and formant frequencies.
Therefore, we do not have the ground truth values which makes the evaluation more difﬁcult than with the English vowels dataset.
Instead of the ground truth, we use the formant frequencies and the fundamental frequency computed on the original speech utterances (i
[Table]
e., before transformation) using the same tools as described in
Section 4.2.2.
This makes the evaluation on TIMIT less reliable than on the English vowels dataset, but it allows us to test the methods on a larger variety of phonemes.
Results are presented in Table 4.
They are very consistent with the ones obtained on the English vowels dataset (see Tables 1 and 2), which further strengthens the conclusions drawn in the discussion paragraph of Section 4.2.
[Table]
[Table]