RAVE: A variational autoencoder for fast and high-quality neural audio synthesis
Anonymous authors
Paper under double-blind review
Abstract
Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks.
However, as raw waveform modelling remains an inherently diﬃcult task, audio generative models are either computationally intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals.
Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suﬀer from low synthesis quality.
In this paper, we introduce a Realtime Audio
Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis.
We introduce a novel two-stage training procedure, namely representation learning and adversarial ﬁne-tuning .
We show that using a post-training analysis of the latent space allows a direct control between the reconstruction ﬁdelity and the representation compactness.
By leveraging a multi-band decomposition of the raw waveform, we show that our model is the ﬁrst able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU.
We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models.
Finally, we present applications of our model for timbre transfer and signal compression .
All of our source code and audio examples are publicly available.
Introduction
Deep learning applied to audio signals proposes exciting new ways to perform speech generation, musical composition and sound design.
Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation (Chung et al., 2015; Fraccaro et al., 2016;
Oord et al., 2016; Vasquez & Lewis, 2019; Dhariwal et al., 2020) or timbre transfer between instruments (Mor et al., 2018).
However, these approaches remain computationally intensive, as modeling audio raw waveforms requires dealing with extremely large temporal dimensionality.
To cope with this issue, previous approaches usually rely on very low sampling rates (16 to 24kHz), which can be suﬃcient for speech signals, but is considered as low-quality for musical applications.
Furthermore, the auto-regressive sampling procedure used by most models (Engel et al., 2017) is prohibitively long.
This precludes real-time application which are pervasive in musical creation, while parallel models (D´efossez et al., 2018) can only allow fast generation at the cost of a lower sound quality.
More recently, Engel et al.
(2019) and Wang et al.
(2019) proposed to leverage classical synthesis techniques to address these limitations, by relying on pre-computed audio descriptors as an extraneous information to condition the models.
While these approaches achieve state-of-the-art results in terms of audio quality, naturalness and computational eﬃciency, the extensive use of audio descriptors highly restricts the type of signals that can be generated.
A possible solution to alleviate this issue would be to rely on Variational Autoencoders (Kingma & Welling, 2014), as they provide a form of trainable analysis-synthesis framework (Esling et al., 2018), without explicit restrictions on the type of features learned.
However, estimating the dimensionality of the latent representation associated with a given dataset prior to model training is far from trivial.
Indeed, a wrong estimation of the latent dimensionality may result in either poor reconstruction or uninformative latent dimensions, which makes latent exploration and manipulation diﬃcult.
In this paper, we overcome the limitations outlined above by proposing a VAE model built speciﬁcally for fast and high-quality audio synthesis.
To do so, we introduce a speciﬁc twostage training procedure where the model is ﬁrst trained as a regular VAE for representation learning , then ﬁne-tuned with an adversarial generation objective in order to achieve high quality audio synthesis.
We combine a multi-band decomposition of the raw waveform alongside classical synthesis blocks inspired by Engel et al.
(2019), allowing to achieve highquality audio synthesis with sampling rates going up to 48kHz without a major increase in computational complexity.
We show that our model is able to converge on complex datasets using a low number of parameters, achieving state-of-the-art results in terms of naturalness and audio quality, while being usable in real-time on a standard laptop CPU.
We compare our model with several state-of-the-art models and show its superiority in unsupervised audio modeling.
In order to address the dimensionality of the learned representation, we introduce a novel method to split the latent space between informative and uninformative parts using a singular value decomposition, and show that replacing the latter part with random noise does not aﬀect the reconstruction quality.
This procedure allows easier exploration and manipulation of latent trajectories, since we only need to operate on a subset of informative latent dimensions.
Finally, we discuss the application of our model in signal compression and timbre style transfer .
Our key contributions are:
• A two-stage training procedure where the model is ﬁrst trained as a regular VAE, then ﬁne-tuned with an adversarial generation objective, as depicted in ﬁgure• A post-training analysis of the latent space providing a way to balance between reconstruction ﬁdelity and representation compactness • High-quality audio synthesis models with sampling rates going up to 48kHz • 20 times faster than realtime synthesis on a standard laptop CPU Audio samples and
supplementary ﬁgures are provided in the accompanying website 1 .
We highly encourage readers to listen to accompanying samples while reading the paper.
[Figure]
State-of-art 2. Variational autoencoders
Generative models aim to understand a given dataset x ∈ R d x by modelling its underlying distribution p ( x ).
To simplify this problem, we can consider that the generation of x is conditioned by latent variables z ∈ R d z , responsible for most of the variations present in x .
Therefore, the complete model is deﬁned by the joint distribution p ( x , z )
=
p ( x | z ) p ( z ), which is usually not analytically solvable given the complexity of real-world data.
Variational autoencoders address this problem by introducing an inference model q φ ( z | x ), optimized to minimize its Kullback-Leibler (KL) divergence with the true posterior distribution p ( z | x )
[Equation]
which can be rearranged to obtain the ﬁnal objective used to train a VAE, called the
Evidence Lower BOund (ELBO), as shown by Kingma & Welling (2014)
[Equation]
The ELBO minimizes the reconstruction error of the model through the likelihood of the data given a latent log p θ ( x | z ), while regularizing the posterior distribution q φ ( z | x ) to match a predeﬁned prior p ( z ).
Both posterior distributions q φ and p θ are parametrized by neural networks respectively called encoder and decoder .
Higgins et al.
(2016) proposes to weight the KL divergence in equation (2) with a parameter β to control the trade-oﬀ between accurate reconstruction and strong latent regularization.
They show that increasing β >leads to less entangled latent dimensions, at the detriment of the reconstruction quality.
2. Autoencoding raw waveform
One of the ﬁrst approaches addressing the raw waveform modelling task are WaveNet (Oord et al., 2016) and SampleRNN (Mehri et al., 2017), where the probability of a waveform x is factorized as a product of conditional probabilities
[Equation]
Those models require a large amount of data and parameters to properly converge.
Furthermore, the autoregressive nature of the synthesis process makes it prohibitively slow, and prone to accumulate errors.
WaveNet has also been adapted by Engel et al.
(2017) for their NSynth model, addressing the representation learning task.
Unlike equation (2), they do not regularize the learned representation, and rather encode the raw waveform determinisitically to its latent counterpart.
It implies the absence of a prior distribution p ( z ) and, therefore, prevents sampling from the latent space.
This restricts the applications of the model to simple reconstructions and interpolations.
As a way to speed-up the synthesis process, D´efossez et al.
(2018) proposed an autoencoder with feed-forward convolutional networks parametrizing both the encoder and the decoder.
They use a perceptually-motivated distance between waveforms called spectral distance as the reconstruction objective
[Equation]
where STFT is the Short-Term Fourier Transform.
Since they use a squared STFT, the phase component is discarded making the loss permissive to inaudible phase variations.
They show that their model is 2500 times faster that NSynth during synthesis, at the expense of a degraded sound quality.
Following the recent advances in generative adversarial modelling (Goodfellow et al., 2014),
Kumar et al.
(2019) proposed to use an adversarial objective to address the parallel audio modelling task.
The discriminator is trained to diﬀerentiate true samples from generated ones, while the generator is optimized to produce samples that are classiﬁed as true by the discriminator.
A feature matching loss is added to the adversarial loss, minimizing the L distance between the discriminator feature maps of real and synthesized audio.
This feature matching mechanism can be seen as a learned metric to evaluate the distance between two samples, and has been successfully applied to the conditional waveform modelling task (e.g spectrogram to waveform or replacement of the decoder in a pretrained autoencoder model).
Method 3. Two-stage training procedure
Ideally, the representation learned by a variational autoencoder should contain high-level attributes of the dataset.
However, two perceptually similar audio signals may contain subtle phase variations that produce dramatically diﬀerent waveforms.
Hence, estimating the reconstruction term in equation (2) using the raw waveform penalizes the model if those subtle variations are not included in the learned representation.
This might both hamper the learning process and include in the latent space those low-level variations about audio signal that are not relevant perceptually.
To address this problem, we split the training process in two stages, namely representation learning and adversarial ﬁne-tuning .
3.1. Stage 1: Representation learning
The ﬁrst stage of our procedure aims to perform representation learning .
We leverage the multiscale spectral distance S ( · , · ) proposed by Engel et al.
in order to estimate the distance between real and synthesized waveforms, deﬁned as
[Equation]
where N is a set of scales, STFT n is the amplitude of the Short-Term Fourier Transform with window size n and hop size n/ 4, and (cid:107) · (cid:107) F , (cid:107) · (cid:107) 1 are respectively the Frobenius norm and L 1 norm.
Using an amplitude spectrum-based distance does not penalize the model for inaccurately reconstructed phase, but encompasses important perceptual features about the signal.
We train the encoder and decoder with the following loss derived from the ELBO
[Equation]
We start by training the model solely with L vae , and once this loss converges, we switch to the next training phase.
3.1. Stage 2: Adversarial fine-tuning
The second training stage aims at improving the synthesized audio quality and naturalness.
As we consider that the learned representation has reached a satisfactory state at this point, we freeze the encoder and only train the decoder using an adversarial objective.
GANs are implicit generative models allowing to sample from a complex distribution by transforming a simpler one, called the base distribution .
Here, we use the learned latent space in the ﬁrst stage as the base distribution, and train the decoder to produce synthesized signals similar to the real ones by relying on a discriminator D .
We use the hinge loss version of the GAN objective, deﬁned as
[Equation]
In order to ensure that the synthesized signal ˆx does not diverge too much from the ground truth x , we keep minimizing the spectral distance deﬁned in equation (5), but also add the feature matching loss L FM proposed by Kumar et al.
Altogether, this yields the following objective for the decoder
[Equation] [Equation]
3. Latent representation compactness
The loss proposed in equation (6) contains two terms, a reconstruction and regularisation term.
Those two terms are somewhat conﬂicting, since the reconstruction term maximises the mutual information between the latent representation and the data distribution, while the regularisation term guides the posterior distribution towards independence with the data (potentially causing posterior collapse ).
In practice, the pressure applied by the regularisation term to the encoder during training encourages it to learn a compact representation, where informative latents have the highest KL divergence from the prior, while uninformative latents have a KL divergence close to 0 (Higgins et al., 2016).
Here, we address the task of identifying the most informative parts of the latent space in order to restrict the dimensionality of the learned representation to the strict minimum required to reconstruct a signal.
To do so, we adapt the method for range and null space estimation (see appendix A) to this problem.
Let Z ∈ R
b
×
d be a matrix composed of b samples z ∈ R d , where z ∼ q φ ( z | x ).
Using a Singular Value Decomposition (SVD) directly on Z to solve the problem of ﬁnding informative parts of the latent space would not be relevant given the high variance present in the collapsed parts of Z .
In order to adapt this to our problem, we ﬁrst remove the variance from Z , by considering the matrix Z (cid:
) ∈
R b × d that veriﬁes
[Equation]
Hence, dimensions of the posterior distribution q φ ( z | x ) that have collapsed to the prior p ( z ) will result in a constant value in Z (cid:48) , that we set to 0 by removing the average of Z (cid:48) across the ﬁrst dimension.
The only dimensions of Z (cid:48) with non-zero values are therefore correlated with the input, which constitute the informative part of the latent space.
Applying a SVD on this centered matrix, we can obtain the matrix Σ containing the singular values of Z (cid:48) , by computing
[Equation]
As detailed in appendix A, the rank r of Z (cid:48) is equal to the number of non-zero singular values in Σ .
Given the high variation that exists in real-world data, it is unlikely that the vanishing singular values of Z (cid:48) are equal to 0.
Therefore, instead of tracking the exact rank r of Z (cid:48) , we deﬁne a ﬁdelity parameter f ∈ [0 − 1], with the associated rank r f deﬁned as the smallest integer verifying
[Equation]
Given the ﬁdelity value f , and a latent representation z ∼ q φ ( z | x ), we reduce the dimensionality of z by projecting it on the basis deﬁned by V T and keep only the r f ﬁrst dimensions.
We obtain a low-rank representation z f , whose dimensionality depends on both the dataset and f .
Before providing z f to the decoder, we concatenate it with noise sampled from the prior distribution, and project it back on its original basis using V .
We demonstrate in section 5.3 the inﬂuence of f on the reconstruction.
Experiments 4. RAVE
Here, we introduce our Realtime Audio Variational autoEncoder (RAVE) built for highquality audio representation learning and faster than realtime synthesis.
Since we target the modelling of 48kHz audio signals, we leverage a multiband decomposition of the raw waveform (see appendix B) as a way to decrease the temporal dimensionality of the data (Yang et al., 2020).
This allows us to expand the temporal receptive ﬁeld of our model without a major increase in computational complexity.
We demonstrate the inﬂuence of the multiband decomposition on the synthesis speed in section 5.2.
Using a 16-band decomposition, we successfully model 48kHz audio signals while producing a compact latent representation using the post-training analysis presented in section 3.2.
Encoder.
We deﬁne our encoder as the combination of a multiband decomposition followed by a simple convolutional neural network, transforming the raw waveform into a 128-dimensional latent representation.
A detailed description of the architecture of the encoder is given in annex C.1.
Decoder.
Our decoder is a modiﬁed version of the generator proposed by Kumar et al.
We use the same alternation of upsampling layers and residual networks, but instead of directly outputting the raw waveform we feed the last hidden layer to three sub-networks.
The ﬁrst sub-network ( waveform ) synthesizes a multiband audio signal (with tanh activation), which is multiplied by the output of the second sub-network ( loudness ), generating an amplitude envelope (with sigmoid activation).
The last sub-network is a noise synthesizer as proposed in Engel et al.
(2019), and produces a multiband ﬁltered noise added to the previous signal.
We found that using an explicit amplitude envelope helps reducing artifacts in the silent parts of the signal, while the noise synthesizer slightly increases the reconstruction naturalness of noisy signals.
See annex C.2 for more details about the architecture of the decoder.
Discriminator.
We use the exact same discriminator as in Kumar et al.
(2019), which is a strided convolutional network applied on diﬀerent scales of the audio signal to prevent artifacts.
We also use the same feature matching loss as in the original paper.
Training.
We follow the training procedure proposed in 3.1, and train RAVE for 3M steps, speciﬁcally 1 .
5M steps for each stage, summing to a total of approximately 6 days on a single TITAN V GPU.
We use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 10 − 4 , β = (0 . 5 , 0 . 9), and batch size 8.
We use dequantization, random crop and allpass ﬁlters with random coeﬃcients as our data augmentation strategy.
Baselines.
We evaluate our model in the context of unsupervised representation learning.
We compare it against two state-of-the-art models: an unsupervised NSynth (Engel et al., 2017) and the autoencoder from SING (D´efossez et al., 2018).
We use the oﬃcial implementations for both baselines with default parameters.
We do not to evaluate the
DDSP (Engel et al., 2019) and NSF (Wang et al., 2019) approaches since we target the unsupervised modelling of any type of audio signals.
Notably, our proposal is able to model both monophonic and polyphonic signals, while the aforementioned methods are restricted to monophonic signals.
4. Datasets
Strings.
Since our main target is the modelling of musical audio signals, we use an internal dataset composed of approximately 30 hours of raw recordings of strings in various conﬁgurations (monophonic solos and polyphonic group performances, with diﬀerent styles and recording conﬁgurations) sampled at 48kHz.
We employ a 90/10 train/test split.
We downsample this dataset to 16kHz when using NSynth and SING.
VCTK.
The Voice Conversion ToolKit (Yamagishi et al., 2019) is a speech dataset composed of approximately 44 hours of raw audio sampled at 48kHz, produced by 110 diﬀerent speakers with various accents.
We use it to evaluate the performances of RAVE when addressing the speech modelling task in an unsupervised fashion.
We also employ a 90/ train/test split and downsample to 16 kHz for other methods.
Results
To encourage reproducibility, we release the source code of our model alongside pretrained weights used to produce the presented results.
Audio samples corresponding to the diﬀerent parts of this section are available on our accompanying website.
5. Synthesis quality
First, we performed a qualitative experiment where participants were asked to rate audio samples on a scale of 1 to 5.
The test consisted of 15 trials, with each presenting an audio sample of the strings dataset never seen during training, alongside its reconstruction by the three models.
The order in which the trials and samples are presented was randomized.
A total of 33 participants took the test, with most being audio professionals.
We report the results of this study in table 1.
[Table]
As we can see, RAVE outperforms both NSynth and SING in terms of audio quality without relying on autoregressive generation.
Furthermore, it achieves high-quality audio synthesis with at least 3.5 times less parameters.
There is still a gap in the evaluation between the ground truth and the other models.
This might come from the diﬃculty of modeling the variety of room acoustics conditions present in the original dataset (as discussed in Engel et al.
(2019)), sometimes making it obvious that the evaluated samples are synthesized.
5. Synthesis speed
Several approaches using normalizing ﬂows (Prenger et al., 2019) or adversarial models (Kumar et al., 2019) can achieve faster than realtime synthesis, but only by relying on
GPUs.
Other approaches make strong assumptions about the signal in order to simplify the generation process (Engel et al., 2019;
Wang et al., 2019), allowing real-time synthesis on CPU, while restricting the range of audio signals that can be modeled.
In table 2, we evaluate the synthesis speed of all models on both CPU and GPU.
The synthesis speed is calculated as the average number of audio samples generated per second for 100 trials. [Table]
Being the only model relying on autoregressive synthesis, NSynth is also the slowest, peaking at 57Hz during generation.
As expected, the parallel nature of SING and RAVE makes them several orders of magnitude faster than NSynth.
The addition of the multiband decomposition speeds up RAVE by a factor of 25, allowing our model to outperform SING on both CPU and GPU.
Overall, we obtain audio synthesis at 48kHz with a 20 × faster than realtime factor on CPU, and up to 240 × on GPU.
5. Balancing compactness and fidelity
Having a compact learned representation has several beneﬁts, since it is easier to analyse, manipulate and understand.
However, it usually comes at the expense of trading oﬀ the reconstruction quality.
Instead of determining the latent space dimensionality prior to the training, we rely on the ﬁdelity parameter f
to estimate it post-training and accordingly crop the learned representation, as explained in section 3.2.
In ﬁgure 2, we compute the relationship between f and the estimated number of dimensions r f for both the strings and vctk datasets.
We depict the inﬂuence of f on the reconstruction quality by measuring the spectral distance (see equation 5) between the original and reconstructed samples.
[Figure]
By setting f = 0 . 99, the dimensionality of the learned representation drops from 128 to just 24 on the strings dataset and 16 on the vctk dataset.
The downsampling factor of the encoder is 2048, resulting in a latent representation sampled at ∼ 23Hz.
Further decreasing f results in a higher spectral distance as shown in ﬁgure 2 (right), but signiﬁcantly reduces the learned representation size.
To exemplify this, we reconstruct a sample from the vctk dataset using diﬀerent ﬁdelity values, and display the resulting melspectrograms in ﬁgure 3. [Figure] [Equation] [Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Figure]
f [Equation] [Figure] As the ﬁdelity parameter decreases, reconstructed samples get less accurate, loosing parts of their attributes such as phonemes or speaker identity.
5. Timbre transfer
We demonstrate that RAVE can be used to perform domain transfer even if it has not been speciﬁcally trained to address this particular task.
We perform domain transfer by simply providing to a pretrained RAVE model audio samples coming from outside of its original training distribution (e.g violin samples are reconstructed with a model trained on speech, see section F for more details).
5. Signal compression
Since the representation learned by RAVE is signiﬁcantly more compact than the raw audio waveform, it can be used as a data-driven compression system for transmission and storage purposes, or to produce the base signal for a higher-level model to work on.
Applying our model provides a compression ratio of 2048, producing latent signals sampled at ∼ 23Hz, which can be used as a simpler representation for further learning tasks.
To illustrate this, we train a WaveNet-inspired model at generating latent signals in an autoregressive fashion for performing audio synthesis with RAVE.
Combining this autoregressive model with the decoder from RAVE, we still obtain synthesis at a 5 times faster than realtime factor.
Furthermore, since latent signals are sampled at a much slower rate than the raw waveform, we obtain a 3-second-long receptive ﬁeld with as few as 9M parameters, corresponding to only 10 layers of the original WaveNet architecture.
Examples of unconditional generation are available on our accompanying website.  Related work
The combination of VAE and GAN has already been studied as a way to build a representation learning model with high quality generation abilities.
Larsen et al.
(2015) propose to replace the reconstruction loss with a learned metric similar to the feature matching loss we use in equation 8.
However, contrary to our two-stage training, they train their encoder to minimize their adversarial reconstruction loss.
Applying this method to RAVE yields a larger estimated latent space dimensionality, hence making the learned representation harder to analyse and interact with (see annex E). Previous approaches combine perceptual losses and auxiliary losses to achieve high quality audio synthesis. Yamamoto et al. (2020) leverage a multiscale spectral loss together with an adversarial objective to address the mel-scale spectrogram inversion task, while Ping et al.
(2018) use a spectral loss as a guide to stabilize their distillation procedure.
Contrary to our two-stage procedure, they mainly use this combination as a warmup mechanism to help their model converge, whereas we use both losses with two separate tasks in mind: representation learning encompassing spectral attributes of the signal, and high-quality audio synthesis based on an adversarial objective, using a ﬁxed learned latent space.
The use of a multiband decomposition of the raw waveform has been successfully applied by Yu et al.
(2019) and Yang et al.
(2020) to the audio modelling task,
and they both show that it helps producing higher quality results while reducing the training and synthesis time.
Modelling 48kHz audio is challenging in term of temporal complexity and memory requirements, and building up from the aforementioned work we show that using a 16 band decomposition allows RAVE to address this task.
Conclusion
In this paper, we introduced RAVE: a Realtime Audio Variational autoEncoder for fast and high-quality neural audio synthesis.
First, we proposed a two-stage training procedure, which ensures that the latent representation is adequate, prior to performing adversarial ﬁnetuning for generating high-quality audio signals.
We showed that our method outperforms the performances of previous approaches in both quantitative and qualitative analyses.
By leveraging a multiband decomposition of the raw waveform, we are able to achieve 20 times faster than realtime synthesis on a standard laptop CPU.
Finally, we proposed a method to control the trade-oﬀ between reconstruction quality and representation compactness, easing the analysis and control of the learned representation.
We provide the open-source code of
RAVE and hope that this will spark contributions and creative uses of our model.
A Range and null space estimation
Let Z be a b × d real-valued matrix with (cid:80) i
Z ij = 0.
We consider the singular value decomposition of Z
[Equation]
where U and V are orthogonal real-valued matrices of respective dimensions b × b and d × d , and Σ is a rectangular diagonal matrix with non-negative values on the diagonal called singular values , sorted by decreasing value.
The number of non-zero singular values gives the rank r of the matrix, i.e the dimension of the vector space spanned by its columns.
The range and null space associated with Z are respectively spanned by the ﬁrst r and last d − r columns of V T .
A low rank version of Z can be obtained by keeping only the r ﬁrst columns of Σ and V , such that
[Equation]
where ˜Σ and ˜V T are respectively of dimension b × r and r × d .
Note that the equality in equation (13) only holds if the last d − r singular values are equal to 0.
Rearranging equation (13), we get the formula to perform a Principal Component Analysis (PCA) on Z with r components
[Equation]
Since V is orthogonal, we can get Z back by multiplying equation (14) by V T .
B Multiband decomposition
The main use of a multiband decomposition is to represent an audio signal sampled at a given sampling rate (e.g 48kHz) as a combination of several downsampled sub-signals (e.g 3kHz), where each sub-signal covers a particular range of frequencies.
This decomposition is useful for compression purposes such as mp3 encoding/decoding, since it allows the use of diﬀerent bit rates on speciﬁc areas of the spectrum.
Multiband decompositions of the raw waveform have already been repeatedly applied in the context of audio generative modelling.
(Yu et al., 2019) speciﬁcally use Pseudo Quadrature Mirror ﬁlters (PQMF), deﬁned by Nguyen (1994).
PQMF are M − band ﬁlter-banks that split a signal into M sub-signals decimated by a factor M .
The M ﬁlters are cosine modulations of a prototype low-pass ﬁlter with cutoﬀ frequency
[Equation]
carefully designed to avoid aliasing in the reconstructed signal.
In practice, it is impossible to create a perfect prototype ﬁlter (i.e perfectly rejected band), but we can instead design a ﬁlter that allows the cancellation of aliasing between neighbouring sub-bands, using methods such as the optimisation of a non-linear objective function as described by M. Rossi et al.
(1996), or using a simple Kaiser window whose parameters are optimised in an analysissynthesis pipeline, as proposed by Lin & Vaidyanathan (1998).
Since the ﬁlter-bank is an orthogonal basis of the signal due to the cosine modulations implied during its creation, the re-synthesis process can be performed by reapplying a temporally ﬂipped version of the ﬁlter-bank to the sub-bands.
C Model architecture C. Encoder
The encoder of RAVE is a convolutional neural network with leaky ReLU activation and batch normalization (see ﬁgure 4).
For all of our experiments we use N = 4, with hidden sizes [64, 128, 256, 512] and strides [4, 4, 4, 2].
The full latent space has 128 dimensions
[Figure]
C. Decoder
We detail the diﬀerent blocks composing the decoder [Figure]
[Figure]
D Latent component collapse
The regularisation term in equation 6 applies a pressure on the encoder to produce a posterior distribution that is close to the prior, and therefore not correlated with the input data.
Since this objective is contrary to the reconstruction objective, it has been empirically shown by Higgins et al.
that some latents are more informative than others, as it can be seen in ﬁgure 7 in the case of RAVE.
[Figure]
d i v e r g e n c e f r o m p
r
i o r
[Figure]
For both datasets, most of the latents have a low KL divergence (lower than 0.01), and only a few (respectively 16 and 9 for the strings and VCTK datasets) have a KL divergence higher than 0.1, which is consistent with the dimensionality estimated by the method proposed in section 3.2 for a ﬁdelity parameter f = 0 .
E Two stage training and latent space compactness
Compared to previous approaches combining VAE and GANs (Larsen et al., 2015), we stop training the encoder once the adversarial stage starts.
As shown in ﬁgure 8, training the encoder during the second stage results in a dramatically increased estimated latent space dimensionality, which goes against our objective of building a compact representation.
Iteration number 1e
L a t e n t
d
i m e n s i o n a li t y f=0.
f=0.
f=0.
f=0.
(a) Frozen encoder 0.
Iteration number 1e
L a t e n t
d
i m e n s i o n a li t y f=0.
f=0.
f=0.
f=0.
[Figure]
encoder
F Out of domain latent representation
We demonstrate in ﬁgure 9 that RAVE can be used to address the timbre transfer task (see section 5.4).
[Figure]
High-level attributes such as the overall loudness and the fundamental frequency of the harmonic components are kept after performing domain transfer.
Other audio attributes such as formants are absent in the examples coming from the strings dataset, and are added by the model in the speech-transfered version.
However, there are no guarantees that the encoder will encode signals from an out-of-domain distribution into a latent representation that matches the prior.
We empirically evaluate the KL divergence of two trained RAVE models for in and out-of-domain data distributions, and report the results in table 3.
[Table]
Unsurprisingly, the smallest KL divergences are observed when encoding data sampled from the training distribution, while encoded out-of-domain signals roughly double the KL divergence.
Stronger transfer abilities may be achieved by integrating a domain adaptation technique such as the one proposed by Lample et al.
(2017) or Mor et al.