T OWARDS G ENERATIVE L ATENT V ARIABLE M ODELS FOR S PEECH
Anonymous authors
Paper under double-blind review
A BSTRACT
While stochastic latent variable models (LVMs) now achieve state-of-the-art performance on natural image generation, they are still inferior to deterministic models on speech.
On natural images, these models have been parameterised with very deep hierarchies of latent variables, but research shows that these model constructs are not directly applicable to sequence data.
In this paper, we benchmark popular temporal LVMs against state-of-the-art deterministic models on speech.
We report the likelihood, which is a much used metric in the image domain but rarely, and often incomparably, reported for speech models.
This is prerequisite work needed for the research community to improve LVMs on speech.
We adapt Clockwork
VAE, a state-of-the-art temporal LVM for video generation, to the speech domain, similar to how WaveNet adapted PixelCNN from images to speech.
Despite being autoregressive only in latent space, we ﬁnd that the Clockwork VAE outperforms previous LVMs and reduces the gap to deterministic models by using a hierarchy of latent variables.
I NTRODUCTION
With the introduction of the variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) came two rapid extensions for modeling speech data (Chung et al., 2015; Fraccaro et al., 2016).
Since then, temporal LVMs have undergone little development and their autoregressive counterparts, such as the WaveNet (Oord et al., 2016a), remain state-of-the-art.
In the image domain, generative LVMs have recently shown superior performance to the PixelCNN (Oord et al., 2016c;b; Salimans et al., 2017), the model that built the foundation for WaveNet.
The improvements in performance have primarily been driven by altered inference models, including top-down (Sønderby et al., 2016) and bidirectional inference (Maaløe et al., 2019), deeper latent hierarchies and skip connections (Sønderby et al., 2016;
Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021).
To innovate and compare LVMs we need good baselining, similar to the many reported benchmarks within the image domain.
However, research in the speech domain has often omitted reporting a likelihood (Oord et al., 2016a; Hsu et al., 2017; Oord et al., 2018b) or has reported likelihoods that are incomparable due to subtle differences in the model parameterizations (Chung et al., 2015;
Fraccaro et al., 2016; Hsu et al., 2017; Aksan & Hilliges, 2019).
Without a proper comparison standard, it is difﬁcult for the ﬁeld of explicit likelihood models on speech to evolve further.
This research pushes forward the state of the LVM on speech by (i) properly benchmarking previous models, (ii) introducing a high-performing hierarchical temporal LVM architecture, and (iii) analyzing the representations of the latent variables.
We ﬁnd that:
(I)
Previous state-of-the-art LVMs achieve close to identical likelihood performance, still signiﬁcantly inferior to the WaveNet.
Interestingly, we also ﬁnd that the WaveNet performs almost identically to a standard LSTM parameterization (Hochreiter & Schmidhuber, 1997) but surprisingly worse than the lossless compression
algorithm
FLAC.
(II) Similar to conclusions within image modeling (Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021), the LVM expressiveness increases with a deeper hierarchy of stochastic latent variables. In direct comparisons, the introduced model outperforms its deterministic counterparts.
However, due to computational cost, it remains infeasible to run the model on the same setting as a state-of-the-art WaveNet model.
x t p ( x t | x <t )
LSTM d t d t −x t −x t p ( x t | z ≤ t , x <t ) p ( z t | x <t , z <t )
VRNN z t z t −d t d t −x t −x t p ( x t | z ≤ t , x <t ) p ( z t | x <t , z <t )
SRNN z t z t −d t d t −x t −x t p ( x t | z ≤ t ) p ( z t | z <t )
CW-VAE z t z t −d t d t −
Since the CW-VAE is not autoregressively dependent on the observed variable and inference injects information into only the stochastic variables, information cannot ﬂow through such a fully deterministic path. This is possible in the VRNN and SRNN. We provide illustrations of in the appendix
(III)
The introduced model ﬁnds strongly clustered speech-related features in its hierarchy of latent variables building a strong case for utilizing such models for other tasks such as semi-supervised learning.
This shows that LVMs without powerful autoregressive decoders for the observed variable have potential as generative speech models when using expressive hierarchies of latent variables.
L ATENT V ARIABLE M ODELS FOR S PEECH 2. R ELATED WORK
LVMs formulated in context of the VAE framework continue to be of interest due to their ability to learn an approximation to the posterior distribution over the dataset.
The posterior is usually on a signiﬁcantly reduced dimensionality compared to the input data and lies very close to a known prior distribution.
Such approximated posterior provides use cases for other tasks beyond generation such as semi-supervised learning (Kingma et al., 2014) and out-of-distribution detection (Havtorn et al., 2021).
Furthermore, from image modeling research, we know that powerful LVMs can achieve state-of-the-art performance without costly autoregressive dependencies on the observed variable.
In recent years, there have been several complementary ways of improving the expressiveness of the
VAE such as building more expressive priors through methods such as Normalizing Flows (Rezende & Mohamed, 2015) and building a deeper hierarchy of stochastic latent variables such as the Ladder
VAE (Sønderby et al., 2016).
In this research, we choose to focus on the latter due to the recent breakthroughs resulting in state-of-the-art VAEs without costly autoregressive dependencies on the observed variable (Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021).
To date, there are two widely cited, and to our knowledge state-of-the-art, explicit likelihood generative LVMs for speech:
• The Variational Recurrent Neural Network (VRNN) (Chung et al., 2015).
• The Stochastic Recurrent Neural Network (SRNN) (Fraccaro et al., 2016).
Other recent LVM contributions also achieve impressive results.
Among the most noteworthy are the FH-VAE (Hsu et al., 2017), that leverages another stochastic latent variable to capture global latent features in the speech, and the VQ-VAE (Oord et al., 2018b), that introduces a hybrid between an LVM with a quantized latent space and an autoregressive model to generate improved empirical samples.
However, the FH-VAE, with its disjoint latent variables, and the VQ-VAE, with its quantized latent space autoregressive prior ﬁtted after training the encoder/decoder, introduce signiﬁcant changes to the original VAE framework to function. The Stochastic WaveNet (Lai et al., 2018) and
STCN (Aksan & Hilliges, 2019) are fully convolutional models that resemble the VRNN.
They are however only autoregressive in observed space and utilize a hierarchy of latent variables. Building on learnings from the LVM improvements in the image domain, we formulate a novel temporal LVM by introducing a hierarchy of stochastic latent variables through the adaptation of a model recently proposed for video prediction:
• The Clockwork Variational Autoencoder (CW-VAE) (Saxena et al., 2021).
2. T EMPORAL VARIATIONAL AUTOENCODING
The VRNN, SRNN and CW-VAE are all autoencoders and take as input a variable-length sequence x
= ( x 1 , x 2 , . .
, x T x ) with x t ∈ X D x .
In general, x may refer to the original observed variable or a deterministic and temporally downsampled representation of the observed variable.
First, x is encoded to a temporal stochastic latent representation z
= ( z 1 , z 2 , . . .
, z T z ) with z t ∈
Z D z and length T z ≤ T x .
This representation is then used to reconstruct the original input x .
The latent variable is assumed to follow some prior distribution p ( z t |· ) .
The prior distribution may depend on latent and observed variables at previous time steps, z <t and x <t , but not x t .
Here we have introduced the shorthand notation z <t =
( z 0 , z 1 , . .
z t − 1 ) .
The models are trained to maximize a likelihood objective.
The exact likelihood is given by
[Equation]
but is intractable to optimize due to the integration over the latent space.
Instead, the true posterior is variationally approximated by q φ ( z | x ) which yields the well-known evidence lower bound (ELBO) on the exact likelihood given by
[Equation]
with respect to { θ , φ } .
We omit the θ and φ subscripts for the remainder of the paper.
A graphical illustration of the models can be seen in ﬁgure 1.
2. V ARIATIONAL R ECURRENT N EURAL N ETWORK (VRNN)
The VRNN (Chung et al., 2015) is essentially a VAE per timestep t .
Each VAE is conditioned on the hidden state of an RNN d t − 1 ∈ R D d , with state transition d t = f
([ x t − 1 , z t − 1 ] , d t − 1 ) where [ · , · ] denotes concatenation.
The joint distribution over observed and latent variables factorizes over time and the latent variables are autoregressive in both the observed and latent space:
[Equation]
The approximate posterior distribution similarly factorizes over time:
[Equation]
The ELBO then becomes
[Equation]
x t x t + x t + x t
+ ...
z 1 t z 1 t + z 1 t
+ z 1 t + ...
z 2 t z 2 t + ... q
( z | x ) x t x t + x t + x t
+ ...
z 1 t z 1 t + z 1 t
+ z 1 t + ...
z 2 t z 2 t
+ ... p ( x , z )
[Equation]
where (cid:2) µ z,t , σ 2 z,t (cid:3) =
ϕ encvrnn
[Equation]
x t
[Equation]
d t
[Equation] [Equation]
6) p (
[Equation]
t | [Equation] <t , [Equation]
t
[Equation]
N (cid:0)
[Equation]
0 ,t , diag ( σ 20 ,t ) (cid:1) , where (cid:2) µ 0 ,t , σ 20 ,t (cid:3) =
ϕ priorvrnn ( d t ) , (7) p ( x
[Equation]
z ≤
[Equation] [Equation]
<
[Equation] [Equation] ) , where ρ x,t = ϕ decvrnn ( z t , d t ) .
The recurrent transition function f is parameterized by a Gated Recurrent Unit (GRU) (Cho et al., 2014).
At timestep zero, d 0 is chosen as the zero vector.
D ( ρ x,t ) denotes any output distribution parameterized by a set of parameters
ρ x,t .
We note that since the decoder is dependent on d t , the transition function f allows the VRNN to learn to ignore parts of or the entire latent variable and establish a purely deterministic transition from x t − 1 to d t similar to a regular GRU (see ﬁgure 1) which well-know weakness of VAEs with powerful decoders (Bowman et al., 2016;
Sønderby et al., 2016).
2. S TOCHASTIC R ECURRENT N EURAL N ETWORK (SRNN)
The SRNN (Fraccaro et al., 2016) is similar to the VRNN but differs by separating the stochastic latent variables from the deterministic representations entirely (see ﬁgure 1).
In generation, this is done by having a GRU run forwards in time over the observed variable to form a deterministic representation d t from x <t .
The latent variable is then sampled from the prior p ( z t | x <t , z <t ) which is conditioned directly on the previous latent variable.
The SRNN also uses a more intricate inference network which essentially learns to solve a smoothing problem rather than a ﬁltering problem by also running backwards in time.
Speciﬁcally, in the smoothing conﬁguration, the inference model q ( z t | x ≤ t , z <t ) includes an additional deterministic variable computed from d t and x t by a GRU running backwards in time i.e. a t = f rev ([ x t , d t ] , a t +1 ) .
In the ﬁltering conﬁguration, this is replaced with an MLP, a t = f rev ([ x t , d t ]) .
The encoding distribution q ( z t | x ≤ t , z <t ) is then conditioned on a t , ϕ encsrnn ( x t , a t ) .
In our experiments we run the SRNN in the smoothing conﬁguration.
2. C LOCKWORK V ARIATIONAL A UTOENCODER (
CW-VAE)
The CW-VAE (Saxena et al., 2021) is a hierarchical latent variable model recently introduced for video generation.
Contrary to the VRNN and SRNN, it is designed to make use of a hierarchy of latent variables and uses top-down inference (Sønderby et al., 2016) which enables learning a covariance between the latent variables in the hierarchy.
As illustrated in ﬁgure 1, it is autoregressive in the latent space but not in the observed space.
Since inference only injects information into the stochastic variables, there is no deterministic path connecting the previously observed variables to the next.
This forces the information to ﬂow through the stochastic latent variables (ﬁgure 1).
Hence, the VRNN and SRNN can collapse to a deterministic model while the CW-VAE cannot.
We denote the latent variable at timestep t and layer l ∈ [1 , L ] by z lt .
In CW-VAE each latent layer is updated only every s l timesteps, where s l is a layer-dependent integer, or stride, deﬁned in a way such that s l is largest for larger values of l .
This imposes the inductive bias that latent variables exist at different temporal resolutions with z l changing over longer time scales than z l − 1 .
In speech, phonetic variation ( 10 − 400 ms ), morphological and semantic features at the word level and speaker-related variation at the global scale make this a reasonable assumption.
The timesteps at which a layer updates its latent state are given by T l ≡ { t ∈ [1 , T ] | t mod s l = 1 } .
In practice and equivalently, we represent this by having references to the unique states copied over time, z lt ≡
[Equation]
The joint distribution factorizes over time and over the latent hierarchy.
[Equation]
The inference model similarly factorizes over time and over the layers of the latent hierarchy with the posterior conditioned on a span of the observed variable x t : t + s l dependent on the layer stride s l .
[Equation]
Similar to the VRNN and SRNN, an encoder, ϕ enc ,l cwvae , is used to parameterize the approximate posterior q ( z | x ) which is taken to be an isotropic Gaussian, as is the prior.
This encoder may be the same for all stochastic layers l , be layer-speciﬁc or use a ladder-network as in the LadderVAE
(Sønderby et al., 2016).
A graphical representation of the inference and generative models for a two-layered
CW-VAE can be seen in ﬁgure 2.
2. M ODELING SPEECH WITH C LOCKWORK VAE S
In the original paper, s l (cid:44) k l − 1 for some constant k making it exponentially dependent on the layer index l with s 1 = 1 .
This is feasible in the video domain with typical frame rates ofor 60 Hz .
In the speech domain however, the observed variable can have sample rates of tens of thousands of frames per second.
This results in sequence lengths that are not generally feasible to model with recurrent architectures.
For this reason, we chose s 1 (cid:29) 1 to achieve an initial temporal downsampling and let s l (cid:44) c l − 1 s 1 for l > 1 and some constant c .
We design the encoder as a ladder-network similar to Sønderby et al.
(2016), as this provides some beneﬁts compared to alternatives such as a standalone encoder per latent variable.
Speciﬁcally, a ladder-network leverages parameter sharing across the latent hierarchy and importantly processes the full observed sequence only once and shares the resulting representations for all latent variables.
This yields a more computationally efﬁcient encoder and a higher activity in latent variables towards the top of the hierarchy.
Finally, we parameterize the encoder/decoder networks using 1D convolutions that operate on the raw waveform.
2. O UTPUT DISTRIBUTION
The choice of output distribution D ( ρ x,t ) is generally data dependent. At the most fundamental level, we select either continuous and discrete distributions depending on the nature of x , the likelihoods of which are not comparable.
In the original VRNN and SRNN, D ( ρ x,t ) was taken to be the continuous isotropic Gaussian or a mixture of it.
This choice generally results in an ill-posed problem with a likelihood that is unbounded from above (Mattei & Frellsen, 2018). As a result, reported likelihoods can be very sensitive to hyperparameter settings and hard to compare.
Additionally, in audio modeling, x ∈ { 0 , 1 , . . . , 2 b − 1 } denotes a discrete-valued waveform sampled with some bit-depth b .
Continuous distributions, and especially mixtures of continuous distributions, are well-known to be able to yield arbitrarily high likelihoods when used to model discrete data (Bishop, 2006).
In the VRNN and SRNN, x was scaled to take values between -1 and 1, as is usual, yielding a gap between the discrete values [-of
-]of 1 2 b − 1 .
This alleviates the issue with discrete values since x becomes approximately continuous as b becomes large.
However, the bit-depth of audio is rarely above 24 meaning that the gap between unique discrete values remains much larger than e.g. the gap between 32 bit ﬂoating point numbers.
This ill-posed likelihood has also been used in other works (Hsu et al., 2017; Lai et al., 2018; Aksan & Hilliges, 2019; Zhu et al., 2020).
To correctly model discrete x with a continuous distribution we must dequantize x to be continuous by e.g. adding uniform noise or using a variational approach (Ho et al., 2019).
The continuous likelihood obtained via dequantization has been shown to be a lower bound on the likelihood that could have been obtained with a discrete distribution (Theis et al., 2016).
Dequantization was common in the image domain (Dinh et al., 2015;
Sønderby et al., 2016) until the introduction of the discretized mixture of logistics (DMoL) (Salimans et al., 2017).
The DMoL was introduced for use in autoregressive models (Salimans et al., 2017) but has become the de facto standard output distribution for generative modeling of natural images.
Notably, this is also the case for latent variable models that are not autoregressive in the observed variable (Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021).
Recently, it was applied to generative speech modeling of raw waveforms (Oord et al., 2018a).
As opposed to e.g. a categorical distribution, the
DMoL induces ordinality on the observed space such that values that are numerically close are also considered close in the probabilistic sense.
This is a sensible inductive bias for images as well as audio where individual samples represent the amplitude of light and pressure, respectively.
S PEECH MODELING EXPERIMENTS
s Model
Conﬁguration
L [bpf]
Uniform
[Table] [Table]
DMoL
Optimal 15.
FLAC8.
WaveNet
D c =10.
LSTM
D d =11.
VRNN
D z =≤ 11.
SRNN
D z =≤ 11.
WaveNet
D c =13.
LSTM
D d =13.
VRNN
D z =≤ 12.
SRNN
D z =≤ 12.
CW-VAE
D z = 96 , L =≤ 12.
CW-VAE
D z = 96 , L =≤ 12.
CW-VAE
D z = 96 , L =≤ 12.
WaveNet
D c =14.
LSTM D d =14. VRNN D z =≤ 13.
SRNN D z =≤ 13. CW-VAE D z = 96 , L =≤ 13.
CW-VAE
D z = 96 , L =≤ 12.
CW-VAE D z = 96 , L =≤ 12. Data We train models on TIMIT (Garofolo, 1993), LibriSpeech (Panayotov et al., 2015) and LibriLight (Kahn et al., 2020).
We provide more details on the datasets in the appendix.
We represent the audio as µ -law encoded PCM standardized to (discrete) values in [ − 1 , 1] with discretization gap of 1 2 b − 1 .
We use the original bit depth of 16 bits and sample rate of 16 000 Hz .
We use this representation both as the input and the reconstruction target.
Likelihood We report likelihoods in units of bits per frame (bpf) as this yields a more intuitive, interpretable and comparable version of likelihood than total likelihood in nats.
It also has direct connections with information theory and compression (Shannon, 1948; Townsend et al., 2019).
In units of bits per frame, lower is better.
The obtained likelihoods can be seen in tables 1 and 2.
For LVMs, we report the onesample ELBO. We describe how to convert likelihood to bpf in the appendix.
Models Architecture and training details are sketched below, while the full details are in the appendix along with additional results for some alternative model conﬁgurations.
We select model conﬁgurations that can be trained on GPUs with a maximum of 12GB of RAM and train all models until convergence.
We use a DMoL with 10 components for the output distribution of all models and model all datasets at their full bit depth of 16 bits . We supply some additional results with a
Gaussian output distribution in the appendix.
[Table] We supply three elementary baselines that form approximate upper and lower bounds on the likelihood.
Speciﬁcally, we evaluate an uninformed discrete uniform distribution and a two-component
DMoL distribution ﬁtted to the training set to benchmark worst case performance.
We also report the compression achieved by the lossless compression algorithm, FLAC (Coalson & Castro Lopo, 2019), which constitutes an approximate lower bound for all considered models.
We conﬁgure the WaveNet baseline as in the original paper using ten layers per block and ﬁve blocks,
L = 10 , B = 5 .
We select the number of channels in the convolutions throughout the model, D c , to allow running it on a 12GB GPU.
We evaluate WaveNet on single frames s = 1 as well as stacks of s = 64 and 256 frames.
We also provide an LSTM baseline (Hochreiter & Schmidhuber, 1997) which runs on stacked waveforms with stack size s .
Hence, every d t is computed from x t : t + s .
It uses encoders and decoders similar to the VRNN and SRNN models and differs only in the design of the recurrent cell where the LSTM is fully deterministic.
We report on LSTM models with hidden state size D d = 256 here.
Other sizes yielded as good or inferior results and can be found in the appendix.
The conﬁguration of the VRNN and SRNN models is with waveform stacks of size s = 1 , s =and 256 .
The stack size of s = 1 is computationally demanding and hence we train this on short randomly sampled segments for each training example and only train them on TIMIT.
For both models we set the latent variable equal in size to the hidden units and run them with D z = 256 .
The CW-VAE is conﬁgured similarly to the VRNN and SRNN models with a convolutional encoder/decoder using strides of s 1 = 64 or 256 and using c = 8 .
This yields s 2 = 512 or 2048 and s 3 = 4096 or 16384 , respectively.
We run the Clockwork VAE with L = 1 , 2 and 3 layers of latent variables.
The number of convolutional channels is set equal to D z .
3. R ESULTS
For temporal resolutions of s = 1 , the deterministic autoregressive models yield the best likelihoods with WaveNet achieving 10 . 88 bpf on TIMIT as seen in table 1.
Somewhat surprisingly, the LSTM baseline almost matches WaveNet with a likelihood of 11 .
11 bpf at s = 1 .
Unsurprisingly, due to being autoregressive also in training, the LSTM trains considerably slower than the parallel dilated convolutional architecture of WaveNet.
Notably, the VRNN and SRNN models achieve likelihoods close to that of WaveNet and the LSTM at around 11 .
09 bpf . For s = 64 , the LSTM trains more efﬁciently but this comes at the cost of higher bpf.
Notably, the LSTM at s = 64 yields a considerably worse likelihood than do VRNN, SRNN and CW-VAE at the same stride being separated by close to 1 bit .
The CW-VAE outperforms both LSTM, VRNN and SRNN when conﬁgured with a hierarchy of latent variables reaching 12 .
15 bits with L = 3 .
With a single layer of latent variables, the missing autoregression in the observed space results in
− − −   10 LDA 1 ( z 1 ) − −
L D A 2 ( z 1 ) bcl dcl gcl kcl tcl d t k dx q s sh z
dh m n l r iy ih ey ae aa ah ao ix     25 Number of neighbors
P h
o
n e m e a cc u r a c
y
[ % ] z 1 z
Mel a likelihood of 12 . 44 bpf which is inferior to both SRNN and VRNN but notably still better than the LSTM.
These observations also carry to s = 256 , where a multilayered CW-VAE outperforms LSTM, VRNN and SRNN.
For strides s > 1 , previous work has attributed the inferior performance of autoregressive sequence models without latent variables, such as WaveNet and the LSTM, to the ability of LVMs to model intra-step correlations (Lai et al., 2019).
For the CW-VAE, VRNN and SRNN, decreasing the stack size s improves the likelihood, as for deterministic models.
This seems to indicate that LVMs may be able to outperform autoregressive models as the stack size approaches s = 1 .
However, due to computational cost of scaling up the CW-VAE for s = 1 , it was infeasible to perform this experiment at present.
For LibriSpeech (table 2), we see similar results with the CW-VAE improving upon the LSTM with s = 64 and the CW-VAE performing better with an additional layer of latent variables.
WaveNet performs similar on LibriSpeech compared to TIMIT
This shows that state-of-the-art likelihoods can be achieved for speech generation with autoregression purely in the latent space as long as the latent space is expressive enough.
This is similar to the ﬁndings on state-of-the-art LVMs for image modeling (Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021).
Finally, a connection can be made between the likelihoods achieved by the considered models and the compression rates of lossless audio compression algorithms.
Whereas lossy compression algorithms such as MP3 exploit the dynamic range of human hearing to achieve 70-95% reduction in bit rate (Brandenburg et al., 1998), state-of-the-art lossless compression algorithms such as FLAC achieve 50-70% (Coalson & Castro Lopo, 2019) independently of audio content.
The LibriSpeech dataset achieves an overall compression to 56.2% of the original bit rate.
Although both the deterministic autoregressive models and the LVMs are lossy, the objective they are trained towards minimize the amount of incurred loss which arguably makes them comparable to lossless compression algorithms.
The best likelihoods achieved by the models considered above roughly correspond to a 30% reduction in bit rate which seems to indicate that there are still solid gains in likelihood to be made in speech modeling.
3. E VALUATION OF LATENT REPRESENTATIONS
A potential beneﬁt of LVMs over purely autoregressive models is that they learn a distilled representation of the data in the form of latent variables.
These should capture the high-level information in the data, and may serve as building blocks for downstream tasks.
Here we evaluate the quality of this learned latent space.
Phonemes Phonemes are a fundamental unit of speech that relate to how certain parts of a word are pronounced.
The TIMIT dataset is annotated with temporally aligned phoneme classes which allows us to analyze how they are organized in the CW-VAE latent space.
Phonemes have durations in the milliseconds which for the TIMIT dataset ranges between 10 and form of recognition [Figure]
[Figure]
[Figure]
e n
[Figure]
e r a cc u r a c
y
[Figure]
H e i g h t a cc u r a c y [ % ] z 1 z [Figure]
Mel
We perform the evaluation with a two-layered CW-VAE with s 1 = 64 and s 2 = 512 .
These strides correspond to temporal resolutions of 4 ms and 32 ms both of which are on the order of phonemes.
We therefore expect both latent variables to capture information related to phonemes.
We infer the latent representations of all utterances of a single speaker from the TIMIT test set.
We take 100 Monte Carlo samples to estimate the mean representation per time step.
We then compute the average latent representation over the duration of each phoneme.
This approximately marginalizes out other sources of variation within the duration of the phoneme.
We obtain a low-dimensional linear subspace of the latent space using a linear discriminant analysis (LDA).
By considering a single speaker, we exclude inter-speaker variation in phoneme pronunciation.
In ﬁgure 3 (left) we visualize the linear subspace of z 1 and the resulting phoneme clusters along with the average accuracy of a leave-one-out k -nearest-neighbor (KNN) classiﬁer on the single left-out latent representation reduced with a 5-dimensional LDA.
We compare to the time-averaged frequency-bin representation of a Mel-spectrogram, also LDA reduced, with hop length set to 64, equal to s 1 for the CW-VAE, window size 256 and 80 Mel bins.
This is a very common preprocessing step in speech applications.
We note that most phonemes are separable in the subspace and that related phonemes such as “s” and “sh” are close 1 .
We show that both latent spaces yield signiﬁcantly better KNN accuracies than the Mel features indicating their usefulness for downstream tasks.
Exploration of latent variable hierarchy In ﬁgure 4 we similarly compute a leave-one-out KNN classiﬁcation on the time-averaged latent representations and Mel-features for the gender and height of the speaker.
We divide the height into three classes: below 175 cm , above 185 cm and in-between.
Compared to phonemes, these are global attributes that can affect the qualities of speech.
Here we again see improved performance from using the learned latent space over Mel-features.
Notably, z 2 is outperformed by the Mel-features for gender identiﬁcation which may indicate that this latent variable learns to ignore this attribute compared to z 1 .
C ONCLUSION /D ISCUSSION
In this paper we have presented benchmarks for speech generation using stochastic latent variable models in comparison to deterministic autoregressive models.
We have adapted the state-of-the-art video generation model, Clockwork VAE, to the speech domain, similar to how WaveNet adapted the PixelCNN.
The Clockwork VAE with a hierarchy of stochastic latent variables outperformed other latent variable models that are autoregressive on both the observed and latent variables.
This is in itself impressive since the Clockwork VAE is autoregressive only in the latent space.
Finally, the Clockwork VAE outperformed a high-performing deterministic autoregressive model showing comparable results to WaveNet.
However, we are still to see a latent variable model outperform WaveNet in a direct comparison.
The Clockwork VAE current remains too computationally expensive for this to happen.
This research serves as a ﬁrst step in that direction.
R EPRODUCIBILITY STATEMENT
The source code used for the work presented in this paper will be made available before the conference.
This code provides all details, practical and otherwise, needed to reproduce the results in this paper including data preprocessing, model training, model likelihood and latent space evaluation.
The source code also includes scripts for downloading and preparing the LibriSpeech, LibriLight and
TIMIT datasets.
The LibriSpeech and LibriLight datasets are open source and can be downloaded with the preparation scripts.
They are also available at https://www.openslr.org/12 and https://github.com/facebookresearch/libri-light , respectively.
The TIMIT dataset is commercial and must be purchased and downloaded from https://catalog.ldc.
upenn.edu/LDC93S1 before running the preparation script.
The stochastic latent variable models considered in this work do not provide an exact likelihood estimate nor an exact latent space representation.
For the likelihood, they provide a stochastic lower bound and some variation in the reproduced likelihoods as well as latent representations must be expected between otherwise completely identical forward passes.
This variance is fairly small in practice when averaging over large datasets such as those considered in this work.
We seed our experiments to reduce the randomness to a minimum, but parts of the algorithms underlying the
CUDA framework are stochastic for efﬁciency.
To retain computational feasibility, we do not run experiments with a deterministic CUDA backend.
E THICS STATEMENT
The work presented here fundamentally deals with automated perception of speech and generation of speech.
These applications of machine learning potentially raise a number of ethical concerns.
For instance, the these models might see possibly adverse use in automated surveillance and generation of deep fakes. To counter some of these effects, this work has focused on openness by using publicly available datasets for model development and benchmarking. Additionally, the work will open source the source code used to create these results.
Ensuring the net positive effect of the development of these technologies is and must continue to be an ongoing effort.
We do not associate any signiﬁcant ethical concerns with the datasets used in this work.
However, one might note that the TIMIT dataset has somewhat skewed distributions in terms of gender and race diversity.
Speciﬁcally, the male to female ratio is about two to one while the vast majority of speakers are Caucasian.
Such statistics might have an effect of some ethical concern on downstream applications derived from such a dataset as also highlighted in recent research (Koenecke et al., 2020).
In LibriSpeech, there is an approximately equal number of female and male speakers while the diversity in race is unknown to the authors
A D ATASETS
TIMIT TIMIT
(Garofolo, 1993) is a speech dataset which contains 16 kHz recordings ofspeakers of eight major dialects of American English, each reading ten phonetically rich sentences.
It amounts to 6300 total recordings splits approximately in 3.94 hours of audio for training and 1.
hours of audio for testing.
No speakers or sentences in the test set are in the training set.
TIMIT includes temporally aligned annotations of phonemes and words as well as speaker metadata such as gender, height, age, race, education level and dialect region (Garofolo, 1993).
LibriSpeech and LibriLight
The LibriSpeech dataset (Panayotov et al., 2015) consists of readings of public domain audio books amounting to approximately 1000 h of audio.
The data is derived from the LibriVox project.
LibriLight (Kahn et al., 2020) is a subset of LibriSpeech created as an automatic speech transcription (ASR) benchmark with limited or no supervision.
We speciﬁcally train on the 100 h train-clean-100 subset of LibriSpeech and the 10 h subset of LibriLight.
Both datasets store the audio as 16 bit pulse code modulation (PCM). B M ODEL ARCHITECTURES
VRNN We implement the VRNN as described in the original work (Chung et al., 2015) but replace the Gaussian output distribution with the DMoL.
SRNN We implement the VRNN as described in the original work (Fraccaro et al., 2016) but replace the Gaussian output distribution with the DMoL. CW-VAE We implement the CW-VAE based on the original work (Saxena et al., 2021) but with some modiﬁcations also brieﬂy described in section 2.6.
We replace the encoder/decoder model architectures of the original work with architectures designed for waveform modelling.
Speciﬁcally, the encoder and decoder are based on the Conv-TasNet (Luo & Mesgarani, 2019) and uses similar residual block structure.
However, contrary to the Conv-TasNet, we require downsampling factors larger than two.
In order to achieve this we
LSTM
The LSTM baseline uses an MLP encoder to embed the waveform subsegment x t : t + s −to a feature vector before feeding it to the LSTM cell.
The encoder is similar to the parameterization of φ encvrnn for the VRNN described above.
The LSTM cell produces the hidden state d t from x t : t + s −and passes it to a decoder.
Like the encoder, the decoder is parameterized like φ decvrnn of the VRNN.
It outputs the waveform predictions x t + s : t +2 s − 1 from the hidden state d t .
WaveNet We implement WaveNet as described in the original work (Oord et al., 2016a) but use a discretized mixture of logistics as the output distribution as also done in other work (Oord et al., 2018a).
Our WaveNet is not conditioned on any signal other than the raw waveform.
The model applies the causal convolution directly to the raw waveform frames (i.e. one input channel).
An alternative option that we did not examine is to replace the initial convolution with an embedding lookup with a learnable vector for each waveform frame value.
During training, a single WaveNet output is independent from inputs outside of the receptive ﬁeld.
For this reason, we can generally train WaveNet on random subsequences of training examples that are shorter than the full examples.
This reduces memory requirements but does not bias the gradient.
The subsequences are chosen to be of length 16000 for models with s = 1 which is larger than the receptive ﬁeld of 5117 and corresponds to one second of audio in TIMIT and LibriSpeech.
For models with s = 64 and s = 256 we train on the full example lengths since the receptive ﬁeld is effectively s times larger.
We implement all models and training scripts in PyTorch 1.9 (Paszke et al., 2017).
For both datasets we use the Adam optimizer (Kingma & Ba, 2015) with default parameters as
We use learning rate 3 e − 4 and no learning rate schedule.
We use PyTorch automatic mixed preceision (AMP) for WaveNet to yield signiﬁcant reductions in memory consumption.
We did not observe any signiﬁcant difference in ﬁnal model performance compared to full ( 32 bit ) precision when trained with a small number of channels.
We split TIMIT as done in other work (Chung et al., 2015; Fraccaro et al., 2016; Aksan & Hilliges, 2019).
We train and test on the full examples, padding batches with zeros when examples are not of equal length.
We sample batches such that they consist of examples that are approximately the same length to minimize the amount of computation wasted on padding.
We train on the 10h and 100h hour subsets of Librispeech (Panayotov et al., 2015).
The 10h subset is as deﬁned in the LibriLight dataset (Kahn et al., 2020) (a subset of Librispeech).
The 100h subset corresponds to the Librispeech train-clean-100 split.
In all cases we evaluate on all the test splits dev-clean, dev-other, test-clean, test-other.
Since the average sequence length in Librispeech is about 4 times longer than that of TIMIT, we do not train on the full sequences.
Instead, in each epoch we randomly sample a segment of length L s frames from each training example.
For WaveNet we set L s = 16000 which, as required, is larger than the receptive ﬁeld and train with batch size 2.
For CW-VAE we set L s = 128000 as the model is autoregressive in latent space and as such can beneﬁt from longer sequence lengths and train with batch size 4. This length approximately corresponds to the length of the longest examples in TIMIT.
In testing, we evaluate on the full sequences. For LibriSpeech we need to split the test examples into segments, again due to memory constraints.
Hence, we do several forward passes per test example, carrying along the internal state for models that are autoregressive in training (LSTM, VRNN, SRNN, CW-VAE) and deﬁne segments to overlap according to model architecture. D C ONVERTING THE LIKELIHOOD TO UNITS OF BITS PER FRAME Here we brieﬂy describe how to compute a likelihood in units of bits per frame (bpf).
In the main text, we use log to mean log e , but here we will be explicit.
In general, conversion from nats to bits (i.e., from log e to log 2 ) is achieved by log 2 ( x ) = log e ( x ) / log 2 ( e ) .
Remember that log 2 p ( x ) factorizes as (cid:80) t log 2 p ( x t ) . In contrast to computing bits per dimension in the image domain, it is important to remember that each example x i must be weighted differently according the sequence length of each speciﬁc example. Thus, we compute the likelihood in bits per frame over the entire dataset as: [Equation]
where i denotes the example index, T i is the length of that example and t is the time index.
E A DDITIONAL LATENT SPACE CLUSTERING
We provide some additional latent space clustering of speaker gender in ﬁgure 5 and of speaker height in ﬁgure 6.
F A DDITIONAL LIKELIHOOD RESULTS
TIMIT, µ -law, DMoL
We provide additional results on TIMIT with audio represented as µ -law encoded PCM in table 3.
TIMIT, linear, Gaussian We also provide some results on TIMIT with the audio instead represented as linear PCM (linearly encoded) and using Gaussian output distributions as has been done previously in the literature (Chung et al., 2015;
Fraccaro et al., 2016; Lai et al., 2018; Aksan &
Hilliges, 2019).
We provide the results in table 4 and include likelihoods reported in the literature for reference.
For our models, we use the same architectures as before but replace the discretized mixture of logistics with either a Gaussian distribution or a mixture of Gaussian distributions.
We constrain the variance of the Gaussians used with our models to be at least σ 2 min = 0 .
01 2 in order to avoid the variance going to zero, the likelihood going to inﬁnity and optimization becoming unstable.
From table 4 we note that the performance of the CW-VAE with Gaussian output distribution when modeling linear PCM (i.e. not µ -law encoded) does not compare as favorably to the other baselines as it did with the discretized mixture of logistics distribution.
We hypothesize that this has to do with using a Gaussian output distribution in latent variable models which, as has been reported elsewhere (Mattei & Frellsen, 2018), leads to a likelihood function that is unbounded above and can grow arbitrarily high.
We discuss this phenomenon in further detail in section G.
We speciﬁcally hypothesize that models that are autoregressive in the observed variable (VRNN,
SRNN, Stochastic WaveNet, STCN) are well-equipped to utilize local smoothness to put very high density on the correct next value and that this in turn leads to a high degree of exploitation of the unboundedness of the likelihood.
Not being autoregressive in the observed variable, the CW-VAE cannot exploit this local smoothness in the same way.
Instead, the reconstruction is conditioned on a stochastic latent variable, p ( x t | z 1 t ) , which introduces uncertainty and likely larger reconstruction variances.
G A DDITIONAL DISCUSSION ON G AUSSIAN LIKELIHOODS IN LVM S
As noted in section F, we constrain the variance of the output distribution of our models to be σ 2 min = 0 .
01 2 for the additional results on TIMIT with Gaussian outputs.
This limits the maximum value attainable by the prediction/reconstruction density of a single waveform frame x t .
Speciﬁcally, we can see that since
[Equation]
12) the best prediction/reconstruction density is achieved when σ 2 ≤ σ 2 min and µ = x t .
Here · indicates any variables we might condition on such as the previous input frame x t − 1 or some latent variables.
We can evaluate this best case scenario for σ 2 min = 0 .
[Equation]
Hence, with perfect prediction/reconstruction and the minimal variance ( 0 . 01 2 ), a waveform frame contributes to the likelihood with 3 . 686 nats .
With an average test set example length of 49 367 .
3 frames frames this leads to a best-case likelihood of 181967.
We provide a list of maximally attainable Gaussian likelihoods on TIMIT for different minimal variances in table 5.
One can note that the maximal likelihood at σ 2 min = 0 .
1 2 is lower than the likelihoods achieved by some models in table 4.
This indicates that the models learn to use very small variances in order to increase the likelihood.
H A DDITIONAL DISCUSSION ON THE CHOICE OF OUTPUT DISTRIBUTION
The DMoL uses a discretization of the continuous logistic distribution to deﬁne a mixture model over a discrete random variable.
This allows it to parameterize multimodal distributions which can express ambiguity about the value of x t .
The model can learn to maximize likelihood by assigning a bit of probability mass to multiple potential values of x t .
While this is well-suited for autoregressive modeling, for which the distribution was developed, the potential multimodality poses a challenge for non-autoregressive latent variable models which independently sample multiple neighboring observations at the output.
In fact, if multiple neighboring outputs deﬁned by the subsequence x t 1 : t 2 have multimodal p ( x t |· ) , we risk sampling a subsequence where each neighboring value expresses different potential realities, independently. s Model Conﬁguration
L [bpf]
Wavenet D C =11. Wavenet D C =11.
Wavenet
D C =11.
Wavenet
D C =10.
Wavenet
D C =10.
Wavenet
D C =10.
LSTM
D d = 128 , L =11.
LSTM
D d = 256 , L =11.
VRNN
D z =≤ 11.
SRNN
D z =≤ 11.
LSTM
D d = 256 , L =11.
LSTM
D d = 256 , L =12.
LSTM
D d = 256 , L =12.
LSTM
D d = 256 , L =12.
WaveNet
D c =13.
LSTM
D d = 96 , L =13.
LSTM
D d = 96 , L =13.
LSTM
D d = 96 , L =13.
LSTM
D d = 256 , L =13.
LSTM
D d = 256 , L =13.
LSTM
D d = 256 , L =13.
LSTM
D d = 512 , L =13.
LSTM
D d = 512 , L =13.
LSTM
D d = 512 , L =13.
VRNN
D z =≤ 12.
VRNN
D z =≤ 12.
SRNN
D z =≤ 12.
SRNN
D z =≤ 12.
CW-VAE
D z =
[Table]
Interestingly
[Table]
most work on latent variable models with non-autoregressive output distributions seem to ignore this fact and simply employ the mixture distribution with 10 mixture components (Maaløe et al., 2019; Vahdat & Kautz, 2020; Child, 2021).
However, given the empirically good results [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
by equation 12 with different values of the minimum variance σ 2 min .
of latent variable models for image generation, this seems to have posed only a minor problem in practice.
We speculate that this is due to the high degree of similarity between neighbouring pixels in images.
I.e. if the neighboring pixels are nuances of red, then, in all likelihood, so is the central pixel.
In the audio domain, however, neighbouring waveform frames can take wildly different values, especially at low sample rates.
Furthermore, waveforms exhibit a natural symmetry between positive and negative amplitudes.
Hence, it seems plausible that multimodality may pose a larger problem in non-autoregressive speech generation by causing locally incoherent samples than it seems to do in image modelling.
Finally, one can note that for continuous variables, the change of variables formula can in many cases convert the likelihood of a random variable following one distribution into the likelihood of another variable following a different distribution.
However, the change of variables formula does not apply to conversions where one or both random variables are discrete.
I A DDITIONAL GRAPHICAL MODELS
In ﬁgure 7 we show the unrolled graphical model of a three-layered CW-VAE with k 1 = 1 and c =yielding k 2 = 2 and k 3 = 4 .
We show both the generative and inference models and highlight in blue the parameter sharing between the two models due to top-down inference.
In ﬁgure 8 we show the graphical model of the recurrent cell of the CW-VAE for a single time step.
As noted in (Saxena et al., 2021), this cell is very similar to the one of the Recurrent State Space
Model (RSSM) (Hafner et al., 2019).
In ﬁgure 9 we illustrate the unrolled graphical models of the inference and generative models of the VRNN (Chung et al., 2015).
We include the deterministic variable d t in order to illustrate the difference to other latent variable models.
Likewise, in ﬁgure 10 we illustrate the unrolled graphical models the SRNN (Fraccaro et al., 2016).
J D ISTRIBUTION OF PHONEME DURATION IN TIMIT
In ﬁgure 11 we plot a boxplots of the duration of each phoneme in the TIMIT dataset.
We do this globally as well as for a single speaker to show that phoneme duration can vary between individual speakers. K CW-VAE SAMPLES FROM THE PRIOR
For the two-layered CW-VAE trained on TIMIT, we provide samples from the prior at the following
URL: https://doi.org/10.5281/zenodo.
[Figure] [Figure] [Figure]
[Figure] [Equation] [Figure] [Equation]
[Figure]
x t x t + x t + z t z t + z t + d t d t + d t
+ p θ ( z , x ) (a) x t x t + x t + z t z t + z t + a t a t + a t + d t d t + d t + q φ
( z | x )
[Figure]
b c l d c l g c l p c l k c l t c l b d g p t k d x q j h s s h
z f v d h m n n g l r w y hh
h v i
y
i
h e h e y a e aa a w a y a h a
o o y
o w uh u w u x e r a x
i x a x r a x - h
[Figure]
D u r a t
i o n o f p
r
o n o un c
i a t i o n ( s p e a k e r
D R W 0 )
[ m s ]
[Figure]
b c l d c l g c l p c l k c l t c l b d g p t k d x q j h c h s s h
z z h f t h v d h m n n g e m e n e n g n x l r w y hh h
v e l i
y
i
h e h e y a e aa a w a y a h a
o o y
o w uh u w u x e r a x
i x a x r a x - h
[Figure]