Problems using deep generative models for probabilistic audio source separation
Abstract
Recent advancements in deep generative modeling make it possible to learn prior distributions from complex data that subsequently can be used for Bayesian inference.
However, we ﬁnd that distributions learned by deep generative models for audio signals do not exhibit the right properties that are necessary for tasks like audio source separation using a probabilistic approach.
We observe that the learned prior distributions are either discriminative and extremely peaked or smooth and non-discriminative.
We quantify this behavior for two types of deep generative models on two audio datasets
Introduction: Langevin dynamics for source separation
Our initial goal was to use Langevin dynamics [11] in combination with deep generative priors to perform source separation of audio mixes.
Our approach closely follows the work of [3], where mixed images are successfully separated.
The biggest advantage of the approach used by [3] is that it does not rely on pairs of source signals and mixes as required by SOTA audio source separation models [10, 4].
Last, in contrast to [9], we are interested in performing the source separation in the time domain, which has multiple advantages like decreased computational complexity as well as the preservation of the phase of the signals.
As common in the source separation literature, we assume that the mix mmm ∈ X is a linear combination of N source signals sss 1 , . . .
,sss N ∈ X mmm = [Equation]
[Equation]
[Equation]
As seen in [3], we take a probabilistic approach in order to solve the source separation problem.
According to Bayes rule we can compute the posterior distribution using
[Equation]
where we use a Gaussian approximation p ( mmm )
=
N ( g ( sss ) , γ 2 I ) with noise parameter γ .
Stochastic
Gradient Langevin Dynamics (SGLD)
[21] enables us sample from the posterior distribution p ( sss | mmm ) without the need for evaluating p ( mmm ) .
A new sample of the source signals can be generated by
[Equation]
where (cid:15) = N (0 , 1) .
Last, we assume that the prior of the source signals factorizes as follows
[Equation]
We choose to parameterize the priors p ( sss 1 ) . . .
p ( sss N ) with deep generative models.
This allows us to easily compute ∇ sss log p ( sss t ) using the automatic differentiation tools from [12].
As noted in [3], in order to successfully recover the original source signals using Equation 3, we require the priors p ( sss 1 ) . . .
p ( sss N ) to be discriminative as well as sufﬁciently smooth.
After our initial experiments failed, we observed that the learned prior distributions are either discriminative and extremely peaked or smooth and non-discriminative.
In the following, we quantify this behavior for two types of deep generative models on two audio dataset.
Method: Modeling raw audio with generative models
Deep learning models as used for image applications are unsuitable for raw audio signals (signals in the time-domain).
Digital audio is sampled at high sample rates, commonly 16kHz up to 44kHz.
The features of interest lie at scales of strongly different magnitudes.
Therefore, generative models need to model the complete range of frequencies containing high-frequency features like timbre and slow frequency features like song structure.
We will be using two types of generative models to learn the likelihood p ( sss ) of audio signals, namely, WaveNet [18] and FloWaveNet [5].
WaveNet The WaveNet is an autoregressive generative model for raw audio.
The generation of a new sample x t is conditioned on all previous samples x 1 , . . .
, x t − 1 .
The likelihood of the entire signal is given by
[Equation]
After training, starting from a single sample the model is able to iteratively generate a coherent time signal.
The distribution p ( x t | x 1 , . . .
, x t − 1 ) is modeled as a multinomial logistic regression, therefore the continuous signal x is discretized using a so-called µ -law encoding [18], resulting in 256 classes.
Due to the autoregressive nature of the Wavenet the genrative process is difﬁult to parallalize and generally slow.
The WaveNet adapts the PixelCNN [19] architecture to the audio domain.
It is a fully-convolutional network where dilated causal convolutions [23] are used.
Using a stack of dilated convolutions increases the receptive ﬁeld of the deep features without increasing the computational complexity.
Further, the convolutions are gated [2] and the output is constructed as the sum of skip connections from each layer.
The skip connections fuse information from multiple time-scales.
FloWaveNet An alternative to using an autoregressive model to model p ( xxx ) are normalizing ﬂows [15].
Normalizing ﬂows are a class of exact likelihood models, which are amenable to gradientbased optimisation and efﬁcient in inference and sampling.
A standard normalizing ﬂow in continuous space, is based on the simple change of variables formula.
Given an observed data variable xxx ∈ X ; a prior probability distribution p Z ( · ) on a latent variable zzz ∈ Z , and a differentiable, bijective function zzz =
f ( xxx ) , we can model a probability distribution p X ( xxx ) as
[Equation]
[Equation]
Since Equation 6 requires the computation of a determinant, a special architecture is necessary to reduce the computational cost.
One such architecture was introduced in [1].
[1] is using so-called coupling layers in which an input xxx is masked into two equally sized parts xxx a and xxx b .
One part is fed into a function to provide the weights of an afﬁne transformation ( s ( · ) , t ( · ) ) of the other
[Equation]
The resulting Jacobian is a triangular matrix, whose determinant can be easily computed.
There exist multiple approaches to combine the dilated convolutions used in the WaveNet and normalizing ﬂows.
In the case of FloWaveNet [5] a WaveNet encoder is used to predict the weights of the afﬁne transformation within every coupling layer.
This architecture enables fast, parallel generation of audio samples.
[Figure]
Experiments 3.1 Datasets
Toy data We create a toy-like dataset consisting of four distinct waveforms, as shown in Figure 1: a sine, a sawtooth, a square and a triangle wave.
We generate the waveforms using a sampling frequency of 16kHz .
For each waveform we sample a random frequency f ∈ [27Hz , 4186Hz] , a random amplitude
A ∈ [0 . 8 , 1 . 0] and a random phase φ ∈ [0 , 2 π ] .
[Equation]
The mix mmm is equal to the mean of the four source signals.
We create 5000 mixes of one second length for training and a testset of 1500 [-mixes.musdb
-]mixes.
musdb18 The musdb18 [13] dataset, created for the 2018 Signal Separation Evaluation Campaign [16], is a benchmark dataset used to evaluate audio source separation algorithms.
The dataset consists of 150 songs from various artists and genres, split into train and test sets sized 100 and 50, respectively.
For each song, the full mix mmm and four separate sources sss 1 ,sss 2 ,sss 3 ,sss 4 are given: drums , bass , vocals and others .
The others source contains any instruments not contained in the ﬁrst three
Note that the mix mmm does not strictly follow Equation 8 since it involves audio effects like compression.
The song ﬁles are encoded with a sampling rate of 44 . 1kHz which we down-sample to 16kHz .
We extract 150 ﬁxed-length frames of one second from each song.
3.2 Discrimative power of the priors
In the following, we train a separate WaveNet and FloWaveNet model for each signal source type of both datasets, in total eight separate generative models for each dataset.
Details about architecture choices and training schedules can be found in the Appendix 5.
Using Langevin dynamics for separation we optimize the separated source frames under each prior model.
During training of the deep generative priors, they explicitly contract the density for positive, in-class examples.
During separation, the priors encounter negative out-of-distribution samples for the ﬁrst time.
To be useful for separation, the priors have to give a low likelihood to samples from the other possible sources.
FloWaveNet
WaveNet − 10 0 − 10 − 3 010 − 3 10− 1.598 − 1.198 − 0.799 − 0.399 − 0.
(a) σ = 0 .−
10 1 − 10 − 1 − 10 − 3 010 − 3 10 −− 2.718 − 2.038 − 1.359 − 0.679 − 0.
(b) σ = 0 .−
10 1 − 10 − 1 − 10 − 3 010 − 3 10 −− 10 1 − 10 0 − 10 − 1 − 10 − 2 − 10 −(c) σ = 0 .−
10 1 − 10 − 1 − 10 − 3 010 − 3 10 −− 1.153 − 0.864 − 0.576 − 0.288 − 0.
(d) σ = 0 .− 14.04 − 9.20 − 4.36 0.48 5.
− 1.678 − 1.258 − 0.839 − 0.419 − 0.
(e) σ = 0 .− 14.04 − 9.20 − 4.36 0.48 5.
− 0.7947 − 0.5960 − 0.3974 − 0.1987 − 0.
(f) σ = 0 .
Following the results in [8], we test the out-of-distribution detection performance of the deep generative priors by evaluating the mean log-likelihood of the test data of each source under each source prior.
In Figure 3 we show that only for the FloWaveNet model trained with the Toy data behaves as anticipated.
The in-class samples have a high likelihood while for all prior models the out-of-class samples have a low likelihood.
For the musdb18 dataset, neither the WaveNet- nor the FloWaveNet-based priors can discriminate between in-class and out-of-class samples.
We hypothesize that this stems from the fact that the real musical data is severely more complicated compared than the Toy data.
The in-class variability of the real sources is that high, that the models are not able to learn a distribution that would be discriminative.
Note that the source other in musdb18 contains an undeﬁned set of instruments making a model of those sounds in general impractical.
But even if we are ignoring this subset neither prior model can discriminate the remaining source types.
As a result, for the following experiments we focus on the Toy data dataset.
3.3 Smoothness of the learned distribution
In the case of the Toy dataset, only the FloWaveNet priors can distinguish between in-class and out-of-class signals.
However, when we tried to use those priors for source separation as described in Section 1, we failed.
We argue that one possible explanation is the peakedness of the learned prior distributions.
The probability mass learned by the model is peaked at true samples but quickly decays with more disturbance of the input.
The reason for this behavior is that all models are trained with noise-free samples of their respective signal sources (sine, square, saw, triangle).
As proposed in [3], we now approximate the noisy distribution log p σ ( x ) , which is the convolution of the noiseless distribution with a Gaussian with variance σ : log p σ ( x ) ∗ N (0 , σ ) by adding Gaussian noise with the same variance to the input.
Figuratively speaking, the Gaussian noise in data space translates to Gaussian smoothing of the peak in the probability distribution of the data.
Instead of retraining the deep generative priors we ﬁne-tune [22] the noise-free models used in
Section 3.2.
We follow [3] in evaluating the noise-conditional model at different levels of noise σ ∈ { 0 . 01 , 0 . 027 , 0 . 077 , 0 .
In Figure 4 we show the cross mean log-likelihood for increasing noise-conditionals for the Toy data dataset.
We ﬁnd that even with small levels of conditional noise added the discriminative power of the learned generative models decays signiﬁcantly.
While being smooth the noise-conditional distributions cannot be used for source separation as intended.
3.4 Random and constant inputs
Previous works [17][20][8] have pointed out that generative models tend to assign high likelihood values to constant inputs.
We ﬁnd the same holds true for generative priors trained on the Toy data
[Table]
dataset. Table 1 shows that for the noise-free model a constant zero input is highly likely, except under the square wave prior, which we assume stems from the square wave never having the value 0.0.
When ﬁne-tuning the model with a noise-conditioning of σ = 0 .
359 the constant zero input becomes less likely for the sources sine and triangle but more likely for saw and square .
To test whether the noise-conditioning results in simple constant inputs being unlikely but pure noise input in return becoming likely we evaluate the likelihood of noise drawn from a zero-centered Gaussian with reasonable wide variance for both the noise-free and noise-conditional model.
In Table 1, we see that for the noise-free prior model, a high variance input noise sampled from N (0 , 0 . 5) is highly unlikely.
Evaluating the same input noise under the wider noise-conditioned prior model the input becomes more likely.
For the sine and saw waveform the noise input is even as likely as a normal in-class input.
We read these results to support the previous interpretation that even a small amount of noise ﬁne-tuning can have severe effects on the estimated density.
The noise-free prior models have sharp likelihood peaks around true data, in which even small amounts of added noise are highly unlikely.
The noise-conditioning of the ﬂow models ﬂattens these peaks in so far that noise and out-of-distribution samples become highly likely, even at small levels of noise-conditioning.
Conclusion
In this work, we show that contemporary generative models for modeling of audio signals also exhibit strong problems with out-of-distribution data as similarly described in [8] for models of image data.
Our experiments reinforce a suspicion that was also experimentally found in prior work on image data.
Current deep generative models do not learn a density that is discriminative against out-of-distribution samples.
We show that in our case the models lose their ability to detect out-of-distribution samples when trained with additive noise which is added to smooth the learned densities.
Therefore our work further demonstrates to be cautious when applying current ﬂow-based models to data outside close bounds of their training distribution.
Appendix 5.1 Source separation with SGLD
For better understanding of the source separation approach we had in mind using the generative models as prior we give the implementation in Algorithm 1.
Algorithm
The Langevin sampling procedure for source separation is fairly straight forward.
For a ﬁxed number of steps T we sample we take a step into the direction of the gradient under the priors and the gradient of the mixing constraint while adding Gaussian noise (cid:15) t . [Table]
[Table]
5.2 Model and training details
We construct the ﬂow models closely following the architecture of FloWaveNet [5] which we show in Figure 5.
It combines the afﬁne coupling layer proposed in RealNVP [1] with the Activation
Normalization proposed in Glow [7] but does not learn the channel mixing function as in Glow and apply the ﬁxed checkerboard masking over the channel dimension.
The WaveNets are constructed as described in the original WaveNet work [14].
As in the original work the outputs of the model at each time-point are modeled with a multinomial distribution with a size of 256 and therefore uses a cross-entropy loss for optimization.
The quantization of the wave data is done with standard µ -law encoding.
The hyperparameters for all for model architectures are listed in Table 2.
The models are trained with the Adam optimizer [6].
As all models are fully convolutional the input size is in no way regimented by the architecture, only in so far that we are avoiding padding in the lower layers nevertheless we ﬁx the size of all frames to 2 14 = 16384 .
The initial learning rate is set to 1 e − 4 and decreased with γ = 0 .
6 in a ﬁxed ﬁve-step decrease schedule.
The toy model is trained with a batch size of 5 and the musdb18 model with a batch size of 2.
We train the two unconditional ﬂows and the WaveNets are trained for each 150.000 steps.
The ﬁne-tuning with the added noise is each trained until convergence which in practice was achieved in 20.000 to 40.000 steps.
[Figure]