BDDM: B ILATERAL D ENOISING D IFFUSION M ODELS FOR F AST AND H IGH -Q UALITY S PEECH S YNTHESIS
A BSTRACT
Diffusion probabilistic models (DPMs) and their extensions have emerged as competitive generative models yet confront challenges of efﬁcient sampling.
We propose a new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with a novel bilateral modeling objective.
We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than a conventional surrogate.
We also ﬁnd that BDDM allows inheriting pre-trained score network parameters from any DPMs and consequently enables speedy and stable learning of the schedule network and optimization of a noise schedule for sampling.
Our experiments demonstrate that BDDMs can generate high-ﬁdelity audio samples with as few as three sampling steps.
Moreover, compared to other state-of-the-art diffusion-based neural vocoders, BDDMs produce comparable or higher quality samples indistinguishable from human speech, notably with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).
I NTRODUCTION
Deep generative models have shown a tremendous advancement in speech synthesis (van den Oord et al., 2016; Kalchbrenner et al., 2018;
Prenger et al., 2019; Kumar et al., 2019a; Kong et al., 2020b;
Chen et al., 2020; Kong et al., 2021).
Successful generative models can be mainly divided into two categories: generative adversarial network (GAN) (Goodfellow et al., 2014) based and likelihoodbased.
The former is based on adversarial learning, where the objective is to generate data indistinguishable from the training data. Yet, the training GANs can be very unstable, and the relevant training objectives are not suitable to compare against different GANs. The latter uses log-likelihood or surrogate objectives for training, but they also have intrinsic limitations regarding generation speed or quality.
For example, the autoregressive models (van den Oord et al., 2016; Kalchbrenner et al., 2018), while being capable of generating high-ﬁdelity data, are limited by their inherently slow sampling process and the poor scaling properties on high-dimensional data.
Likewise, the ﬂowbased models (Dinh et al., 2016; Kingma & Dhariwal, 2018; Chen et al., 2018; Papamakarios et al., 2021) rely on specialized architectures to build a normalized probability model, whose training is less parameter-efﬁcient.
Other prior works use surrogate objectives, such as the evidence lower bound in variational auto-encoders (Kingma & Welling, 2013; Rezende et al., 2014; Maaløe et al., 2019) and the contrastive divergence in energy-based models (Hinton, 2002; Carreira-Perpinan & Hinton, 2005).
These models, despite showing improved speed, typically only work well for lowdimensional data, and, in general, the sample qualities are not competitive to the GAN-based and the autoregressive models (Bond-Taylor et al., 2021).
An up-and-coming class of likelihood-based models is the diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015), which introduces the idea of using a forward diffusion process to sequentially corrupt a given distribution and learning the reversal of such diffusion process to restore the data distribution for sampling.
From a similar perspective, Song & Ermon (2019) proposed the score-based generative models by applying the score matching technique (Hyvarinen & Dayan, 2005) to train a neural network such that samples can be generated via Langevin dynamics.
Along these two lines of research, Ho et al.
(2020) proposed the denoising diffusion probabilistic models (DDPMs) for high-quality image syntheses. Dhariwal & Nichol (2021) demonstrated that improved
DDPMs Nichol & Dhariwal (2021) are capable of generating high-quality images of comparable or even superior quality to the state-of-the-art (SOTA) GAN-based models.
For speech syntheses, DDPMs were also applied in Wavegrad (Chen et al., 2020) and DiffWave (Kong et al., 2021) to produce higher-ﬁdelity audio samples than the conventional non-autoregressive models (Yamamoto et al., 2020; Kumar et al., 2019b;
Yang et al., 2020; Bi´nkowski et al., 2020) and matched the quality of the SOTA autoregressive methods (Chen et al., 2020).
Despite the compelling results, the diffusion generative models are two to three orders of magnitude slower than other generative models such as GANs and VAEs.
Their primary limitation is that they require up to thousands of diffusion steps during training to learn the target distribution.
Therefore a large number of reverse steps are often required at sampling time.
Recently, extensive investigations have been conducted to reduce the sampling steps for efﬁciently generating high-quality samples, which we will discuss in the related work in Section 2.
Distinctively, we conceived that we might train a neural network to efﬁciently and adaptively estimate a much shorter noise schedule for sampling while achieving generation performances comparable or superior to the conventional DPMs.
With such an incentive, after introducing the conventional DPMs as our background in Section 3, we propose in Section 4 bilateral denoising diffusion models (BDDMs), named after a bilateral modeling perspective – parameterizing the forward and reverse processes with a schedule network and a score network, respectively.
We theoretically derive that the schedule network should be trained after the score network is optimized.
For training the schedule network, we propose a novel objective to minimize the gap between a newly derived lower bound and the log marginal likelihood.
We describe the training algorithm as well as the fast and high-quality sampling algorithm in Section 5.
The training of the schedule network converges very fast using our newly derived objective, and its training only adds negligible overhead to DDPM’s.
In Section 6, our neural vocoding experiments demonstrated that BDDMs could generate high-ﬁdelity samples with as few as three sampling steps.
Moreover, our method can produce speech samples indistinguishable from human speech with only seven sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).
R ELATED W ORK
Prior works showed that noise scheduling is crucial for efﬁcient and high-ﬁdelity sample generation in diffusion probabilistic models.
DDPMs (Ho et al., 2020) shared a linear noise schedule for both training and sampling, which, however, requires thousands of sampling iterations to obtain competitive results.
To speed up the sampling process, one class of related work, including (Chen et al., 2020; Kong et al., 2021; Nichol & Dhariwal, 2021), attempts to use a different, shorter noise schedule for sampling.
For clarity, we thereafter denote the training noise schedule as β ∈ R T and the sampling noise schedule as ˆ β ∈ R N with N < T .
In particular, Chen et al.
(2020) applied a grid search (GS) algorithm to select ˆ β .
Unfortunately, GS becomes prohibitively slow when N grows large, e.g., N = 6 took more than a day on a single NVIDIA Tesla P40 GPU.
This is because the time costs of GS algorithm grow exponentially with N , i.e., O (9 N ) with 9 bins as the default setting in Chen et al.
Instead of searching, Kong et al.
(2021) devised a fast sampling (FS) algorithm based on an expert-deﬁned 6-step noise schedule for their score network.
However, this speciﬁcally tuned noise schedule is hard to generalize to other score networks, tasks, or datasets.
Another class of noise scheduling methods searches for a subsequence of time indices of the training noise schedule, which we call the time schedule .
DDIMs (Song et al., 2021a) introduced an accelerated reverse process that relies on a pre-speciﬁed time schedule.
A linear and a quadratic time schedule were used in DDIMs and showed superior generation quality over DDPMs withinto 100 sampling steps.
Nichol & Dhariwal (2021) proposed a re-scaled noise schedule for fast sampling, but this also requires pre-specifying the time schedule and the training noise schedule.
Nichol & Dhariwal (2021) also proposed learning variances for the reverse processes, whereas the variances of the forward processes, i.e., the noise schedule, which affected both the means and variances of the reverse processes, were not learnable.
According to the results of (Song et al., 2021a;
Nichol &
Dhariwal, 2021), using a linear or quadratic time schedule resulted in quite different performances in different datasets, implying that the optimal choice of schedule varies with the datasets.
So, there remains a challenge in ﬁnding a short and effective schedule for fast sampling on different datasets.
Notably, Kong & Ping (2021) proposed a method to map a noise schedule to a time schedule for fast sampling.
In this sense, searching for a time schedule becomes a sub-set of the noise scheduling problem, which resembles the above category of methods.
Although DPMs (Sohl-Dickstein et al., 2015) and DDPMs (Ho et al., 2019) mentioned that the noise schedule could be learned by re-parameterization, the approach was not investigated in their works.
Closely related works that learn a noise schedule emerged until very recently.
San-Roman et al.
(2021) proposed a noise estimation (NE) method, which trained a neural net with a regression loss to estimate the noise scale from the noisy sample at each time point, and then predicted the next noise scale.
However, NE requires a prior assumption of the noise schedule following a linear or
Fibonacci rule.
Most recently, a concurrent work to ours by Kingma et al.
(2021) jointly trained a neural net to predict the signal-to-noise ratio (SNR) by maximizing the variational lower bound.
The
SNR was then used for noise scheduling.
Different from ours, this scheduling neural net only took t as input and is independent of the noisy sample generated during the loop of sampling process.
Intrinsically, with limited information about the sampled data, the predicted SNR could deviate from the actual SNR of the noisy data during sampling.
B ACKGROUND 3. D IFFUSION PROBABILISTIC MODELS (DPM S )
Given i.i.d.
samples { x 0 ∈ R D } from an unknown data distribution p data ( x 0 ) , diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015) deﬁne a forward process q ( x 1: T | x 0 ) =
(cid:81)
Tt =1 q ( x t | x t − 1 ) that converts any complex data distribution into a simple, tractable distribution after T steps of diffusion.
A reverse process p θ ( x t − 1 | x t ) parameterized by θ is used to model the data distribution: p θ ( x 0 )
= (cid:82) π
( x T ) (cid:81)
Tt =1 p θ ( x t − 1 | x t ) d x 1: T , where π ( x T ) is the prior distribution for starting the reverse process.
Then, the variational parameters θ can be learned by maximizing the standard log evidence lower bound (ELBO):
[Equation]
3. D ENOISING D IFFUSION P ROBABILISTIC M ODELS (DDPM S )
As an extension to DPMs, denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020) applied the score matching technique (Hyvarinen & Dayan, 2005; Song & Ermon, 2019) to deﬁne the reverse process.
In particular, DDPMs considered a Gaussian diffusion process parameterized by a noise schedule β ∈ R T with 0 < β 1 , . .
. , β T < 1 : [Equation] T
[Equation]
t = q β t ( x t | x t − 1 ) , where q β t ( x t | x t − 1 ) :=
N ( (cid:112) 1 − β t x t − 1 , β t I ) .
Based on the nice property of isotropic Gaussians, one can express x t directly conditioned on x 0 : [Equation]
I [Equation] where α t = t (cid:89)
i [Equation] (cid:112) [Equation]
To revert this forward process, DDPMs employ a score network 1 (cid:15) θ ( x t , α t ) to deﬁne p θ ( [Equation] |
[Equation]
where Σ t is the co-variance matrix deﬁned for the reverse process.
Ho et al.
(2020) showed that setting Σ t =
˜ β t I = 1 − α 2 t − 1 1 − α 2 t β t
I is optimal for a deterministic x 0 , while setting Σ t =
β t
I is optimal for a white noise x 0 ∼ N ( 0 , I ) .
Alternatively, Nichol & Dhariwal (2021) proposed learnable variances by interpolating the two optimals with a jointly trained neural network, i.e., Σ t,θ ( x )
= diag (exp( v θ ( x ) log β t
+ (1 − v θ ( x )) log ˜ β t )) ,
where v θ ( x ) ∈ R D is a trainable network.
Note that the calculation of the complete ELBO in Eq.
(1) requires T forward passes of the score network, which would make the training computationally prohibitive for a large T .
To feasibly train the score network, instead of computing the complete ELBO, Ho et al.
(2020) proposed an efﬁcient training mechanism by sampling from a discrete uniform distribution: t ∼ U{ 1 , ..., T } , x 0 ∼ p data ( x 0 ) , (cid:15) t ∼ N ( 0 , I ) at each training iteration to compute the training loss:
[Equation]
, (5) which is a re-weighted form of D KL ( q β ( x t − 1 | x t , x 0 ) ||
p θ ( x t − 1 | x t )) .
Ho et al.
(2020) reported that the re-weighting worked effectively for learning θ .
Yet, we demonstrate it is deﬁcient for learning the noise schedule β in our ablation experiment in Section 6.2.
B ILATERAL DENOISING DIFFUSION MODELS (BDDM S ) 4. P ROBLEM FORMULATION
For fast sampling with DPMs, we strive for a noise schedule ˆ β for sampling that is much shorter than the noise schedule β for training.
As shown in Fig.
1, we deﬁne two separate diffusion processes corresponding to the noise schedules, β and ˆ β , respectively.
The upper diffusion process parameterized by β is the same as in Eq.
(2), whereas the lower process is deﬁned as q ˆ β (ˆ x 1: N | ˆ x 0 ) =
(cid:81)
Nn =1 q ˆ β n (ˆ x n | ˆ x n − 1 ) with much fewer diffusion steps ( N (cid:28) T ).
In our problem formulation, β is given, but ˆ β is unknown.
The goal is to ﬁnd a ˆ β for the reverse process p θ (ˆ x n − 1 | ˆ x n ; ˆ β n ) such that ˆ x 0 can be effectively recovered from ˆ x N with N reverse steps.
4. M ODEL DESCRIPTION
Although many prior arts (Ho et al., 2020; Chen et al., 2020; Song et al., 2021a; San-Roman et al., 2021) directly applied a shortened linear or Fibonacci noise schedule to the reverse process, we argue that these are sub-optimal solutions.
Theoretically, the diffusion process speciﬁed by a new shortened noise schedule is essentially different from the one used to train the score network θ .
Therefore, θ is not guaranteed suitable for reverting the shortened diffusion process.
This issue motivated a novel modeling perspective to establish a link between the shortened schedule ˆ β and the score network θ , i.e., to have ˆ β optimized according to θ .
As a starting point, we consider an N = (cid:98) T/τ (cid:99) , where 1 ≤ τ < T is a hyperparameter controlling the step size such that each diffusion step between two consecutive variables in the shorter diffusion process corresponds to τ diffusion steps in the longer one.
Based on Eq.
(2), we deﬁne the following:
[Equation]
where x t is an intermediate diffused variable we introduced to link the two differently indexed diffusion sequences.
We call it a junctional variable, which can be easily generated given x 0 and β during training:
[Equation]
Unfortunately, for the reverse process when x 0 is not given, the junctional variable is intractable.
However, our key observation is that while using the score by a score network θ ∗ trained for the long β -parameterized diffusion process, a short noise schedule ˆ β ( φ ) can be optimized accordingly by introducing a schedule network φ .
We provide its mathematical derivations in Appendix A.3.
Next, we present a formal deﬁnition of BDDM and derive its training objectives, L ( n ) score ( θ ) and
L ( n ) step ( φ ; θ ∗ ) , for the score network and the schedule network, respectively, in more detail.
4. S CORE N ETWORK
Recall that a DDPM starts the reverse process with a white noise x T ∼ N ( 0 , I ) and takes T steps to recover the data distribution:
[Equation]
A BDDM, in contrast, starts from the junctional variable x t , and reverts a shorter sequence of diffusion random variables with only n steps:
[Equation]
where q ˆ β (ˆ x n − 1 ; x t , (cid:15) n ) is deﬁned as a re-parameterization on the posterior:
[Equation]
where ˆ α n = (cid:81) ni =1 (cid:113) 1 −
ˆ β
i , x t = α t x 0 + (cid:112) 1 − α 2 t (cid:15) n is the junctional variable that maps x t to ˆ x n given an approximate index t ∼ U{ ( n − 1) τ, ..., nτ − 1 , nτ } and a sampled white noise (cid:15) n ∼ N ( 0 , I ) .
Detailed derivation from Eq.
(9) to (10) is provided in Appendix A.2.
4.3. T RAINING OBJECTIVE FOR SCORE NETWORK
With the above deﬁnition, a new form of lower bound to the log marginal likelihood can be derived such that log p θ
[Equation]
ˆ x
[Equation]
x 0 , x t ) , where
[Equation]
See detailed derivation in Proposition 1 in Appendix A.2.
In the following Proposition 2, we prove that via the junctional variable x t , the solution θ ∗ for optimizing the objective L ( t ) ddpm ( θ ) , ∀ t ∈ { 1 , ..., T } is also the solution for optimizing L ( n ) score ( θ ) , ∀ n ∈ { 2 , ..., N } .
Thereby, we show that the score network θ can be trained with L ( t ) ddpm ( θ ) and re-used for reverting the short diffusion process over ˆ x N :0 .
Although the newly derived lower bound result in the same objective as the conventional score network, it for the ﬁrst time establishes a link between the score network θ and ˆ x N :0 .
The connection is essential for learning ˆ β , which we will describe next.
4. SCHEDULE NETWORK
In BDDMs, a schedule network is introduced to the forward process by re-parameterizing ˆ β n as ˆ β n ( φ ) =
f φ (cid:16) x t ; ˆ β n +1 (cid:17) , and recall that during training, we can use x t = α t x 0 + (cid:112) 1 − α 2 t (cid:15) n and ˆ β n +
=
− α 2 t
+ τ α 2 t .
Through the re-parameterization, the task of noise scheduling, i.e., searching for ˆ β , can now be reformulated as training a schedule network f φ that ancestrally estimates datadependent variances.
The schedule network learns to predict ˆ β n based on the current noisy sample x t – this makes our method fundamentally different from existing and concurrent work, including Kingma et al.
(2021) – as we reveal that, aside from ˆ β n +1 , t , or n that reﬂects diffusion step information, x t is also essential for noise scheduling from a reverse direction at inference time.
Speciﬁcally, we adopt the ancestral step information ( ˆ β n +1 ) to derive an upper bound for the current step while leaving the schedule network only to take the current noisy sample x t as input to predict a relative change of noise scales against the ancestral step.
First, we derive an upper bound of ˆ β n by proving 0 < ˆ β n < min (cid:110) 1 − ˆ α 2 n +1 1 − ˆ β n +1 , ˆ β n +1 (cid:111) in Appendix A.1.
Then, by multiplying the upper bound by a ratio estimated by a neural network σ φ : R D (cid:55)→ (0 , 1) , we deﬁne
[Equation] [Equation]
where the network parameter set φ is learned to estimate the ratio between two consecutive noise scales ( ˆ β n and ˆ β n +1 ) from the current noisy input x t .
Finally, at inference time for noise scheduling, starting from a maximum reverse steps ( N ) and two hyperparameters (ˆ α N , ˆ β N ) , we ancestrally predict the noise scale ˆ β n ( φ ) =
f φ (cid:16) ˆ x n ; ˆ β n +1 (cid:17) , for n from N to 1 , and cumulatively update the product ˆ α n =
[Equation]
4.4. T RAINING OBJECTIVE FOR SCHEDULE NETWORK
Here we describe how to learn the network parameters φ effectively.
First, we demonstrated that φ should be trained after θ is well-optimized, referring to Proposition 3 in Appendix A.3.
The
Proposition also shows that we are minimizing the gap between the lower bound F ( n ) score ( θ ∗ ) and log p θ ∗ (ˆ x 0 ) , i.e., log p θ ∗ (ˆ x 0 ) − F ( n ) score ( θ ∗ ) , by minimizing the following objective
[Equation]
14) which is deﬁned as a KL divergence to directly compare p θ ∗ (ˆ x n − 1 | ˆ x n = x t ) against the reparameterized forward process posteriors, which are tractable when conditioned on the junctional noise scale α t and x 0 .
The detailed derivation of Eq.
(14) is also provided in the proof of Proposition 3 to get its concrete formulas as shown in Step (8-10) in Alg. 2.
A LGORITHMS : TRAINING , NOISE SCHEDULING , AND SAMPLING 5. T RAINING SCORE AND SCHEDULE NETWORKS
Following the theoretical result in Appendix A.3, θ should be optimized before learning φ .
Thereby ﬁrst, to train the score network (cid:15) θ , we refer to the settings in (Ho et al., 2020;
Chen et al., 2020; Song et al., 2021a) to deﬁne β as a linear noise schedule: β t = β start + tT
( β end − β start ) , for 1 ≤ t ≤ T, where β start and β end are two hyperparameter that speciﬁes the start value and the end value.
This results in Algorithm 1, which resembles the training algorithm in (Ho et al., 2020).
Next, based on the converged score network θ ∗ , we train the schedule network φ .
We draw an n ∼ U{ 2 , . . .
, N } at each training step, and then draw a t ∼ U{ ( n − 1) τ, ..., nτ } .
These together can be re-formulated as directly drawing t ∼ U{ τ, ..., T − τ } for a ﬁner-scale time step.
Then, we
[Table] [Table]
[Table] [Table]
sequentially compute the variables needed for calculating L ( n ) step ( φ ; θ ∗ ) , as presented in Algorithm 2.
We observed that, although a linear schedule is used to deﬁne β , the noise schedule of ˆ β predicted by f φ is not limited to but rather different from a linear one.
5. N OISE SCHEDULING FOR FAST AND HIGH - QUALITY SAMPLING
After the score network and the schedule network are trained, the inference procedure can divide into two phases: (1) the noise scheduling phase and (2) the sampling phase.
First, we run the noise scheduling process similarly to a sampling process with N iterations maximum.
Different from training, where α t is forward-computed, ˆ α n is instead a backward-computed variable (from N to 1 ) that may deviate from the forward one because { ˆ β i } n − 1 i are unknown in the noise scheduling phase during inference.
To start noise scheduling, we ﬁrst set two hyperparameters: ˆ α N and ˆ β N .
We use β 1 , the smallest noise scale seen in training, as a threshold to early stop the noise scheduling process so that we can ignore small noise scales ( < β 1 ) that were never seen by the score network.
Overall, the noise scheduling process presents in Algorithm 3.
In practice, we apply a grid search algorithm of M bins to Algorithm 3, which takes O ( M 2 ) time, to ﬁnd proper values for (ˆ α N , ˆ β N ) .
We used M = 9 as in (Chen et al., 2020).
The grid search for our noise scheduling algorithm can be evaluated on a small subset of the training samples.
Empirically, even as few as 1 sample for evaluation works well in our algorithm.
Finally, given the predicted noise schedule ˆ β ∈ R N s , we generate samples with N s sampling steps, as shown in Algorithm 4.
E XPERIMENTS
We conducted a series of experiments on neural vocoding tasks to evaluate the proposed BDDMs.
First, we compared BDDMs against several strongest models that have been published: the mixture of logistics (MoL) WaveNet
(Oord et al., 2018) implemented in (Yamamoto, 2020), the WaveGlow (Prenger et al., 2019) implemented in (Valle, 2020), the MelGAN (Kumar et al., 2019a) implemented in (Kumar, 2019), the HiFi-GAN (Kong et al., 2020b) implemented in (Kong et al., 2020a) and the two most recently proposed diffusion-based vocoders, i.e., WaveGrad (Chen et al., 2020) and
DiffWave (Kong et al., 2021), both re-implemented in our code.
The hyperparameter settings of
BDDMs and all these models are detailed in Appendix B.
In addition, we also compared BDDMs to a variety of scheduling and acceleration techniques applicable to DDPMs, including the grid search (GS) approach in WaveGrad, the fast sampling (FS)
[Table] [Table]
approach based on a user-deﬁned 6-step schedule in DiffWave, the DDIMs (Song et al., 2021a) and a noise estimation (NE) approach (San-Roman et al., 2021).
For fair and reproducible comparison with other models and approaches, we used the LJSpeech dataset (Ito & Johnson, 2017), which consists of 13,100 22kHz audio clips of a female speaker.
All diffusion models were trained on the same training split as in (Chen et al., 2020).
We also replicated the comparative experiment of neural vocoding using a multi-speaker VCTK dataset (Yamagishi et al., 2019) as presented in Appendix C and obtained a result consistent with that obtained from the LJSpeech dataset.
6. S AMPLING QUALITY IN OBJECTIVE AND SUBJECTIVE METRICS
To assess the quality of each generated audio sample, we used both objective and subjective measures for comparing different neural vocoders given the same ground-truth spectrogram s as the condition, i.e., (cid:15) θ
( x , s , α t ) .
Speciﬁcally, we used two scale-invariant metrics: the perceptual evaluation of speech quality (PESQ) (Rix et al., 2001) and the short-time objective intelligibility (STOI) (Taal et al., 2010) to measure the noisiness and the distortion of the generated speech relative to the reference speech.
Mean opinion score (MOS) was also used as a subjective metric for evaluating the naturalness of the generated speech.
The assessment scheme of MOS is included in Appendix B.
In Table 1, we compared BDDMs against the state-of-the-art (SOTA) vocoders.
To predict noise schedules with different sampling steps (3, 7, and 12), we set three pairs of { ˆ α N , ˆ β N } for BDDMs by running on Algorithm 3 a quick hyperparameter grid search, which is detailed in Appendix B.
Among the 9 evaluated vocoders, only our proposed BDDMs with 7 and 12 steps and DiffWave with 200 steps showed no statistic-signiﬁcant difference from the ground-truth in terms of MOS.
Moreover, BDDMs signiﬁcantly outspeeded DiffWave in terms of RTFs.
Notably, previous diffusion- based vocoders achieved high MOS scores at the cost of an unacceptable RTF for industrial deployment.
In contrast, BDDMs managed to achieve a high standard of generation quality with onlysampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).
In Table 2, we evaluated BDDMs and alternative accelerated sampling methods, which used the same score network for a pair-to-pair comparison.
The GS method performed stably when the step number was small (i.e., N ≤ 6 ) but not scalable to more step numbers, which were therefore bypassed in the comparisons of 7 and 12 steps.
The FS method by Song et al.
(2021a) was linearly interpolated to 3 and 7 steps for a fair comparison.
Comparing its 7-step and 3-step results, we observed that the FS performance degraded drastically.
Both the DDIM and the NE methods were stable across all the steps but were not performing competitively enough.
In comparison, BDDMs consistently attained the leading scores across all the steps.
This evaluation conﬁrmed that BDDM was superior to other acceleration methods for DPMs in terms of both stability and quality.
6. A BLATION S TUDY AND A NALYSIS
We attribute the primary advantage of BDDMs to the newly derived objective L ( n ) step for learning φ .
To better reason about this, we performed an ablation study, where we substituted the proposed loss with the standard negative ELBO for learning φ as mentioned by Sohl-Dickstein et al. (2015).
We plotted the network outputs with different training losses in Fig.
It turned out that, when using
L ( n ) elbo to learn φ , the network output rapidly collapsed to zero within several training steps; whereas, the network trained with L ( n ) step produced ﬂuctuating outputs.
The ﬂuctuation is a desirable property showing the network properly predicts t -dependent noise scales, as t is a random time step drawn from a uniform distribution in training.
By setting ˆ β = β , we empirically validated that F ( t ) bddm := F ( t ) score + L ( t ) step ≥ F ( t ) elbo with their respective values at t ∈ [20 , 180] using the same optimized θ ∗ .
Each value is provided with 95% conﬁdence intervals, as shown in Fig.
In this experiment, we used the LJ speech dataset and set T = 200 and τ = 20 .
Notably, we dropped their common entropy term R
θ ( ˆ x 0 , x t ) < 0 to mainly compare their
KL divergences.
This explains those positive lower bound values in the plot.
The graph shows that our proposed bound F ( t ) bddm is always a tighter lower bound than the standard one across all examined t .
Moreover, we found that F ( t ) bddm attained low values with a relatively much lower variance for t ≤ 50 , where F ( t ) elbo was highly volatile.
This implies that F ( t ) bddm better tackles the difﬁcult training part, i.e., when the score becomes more challenging to estimate as t → 0 .
C ONCLUSIONS
BDDMs parameterize the forward and reverse processes with a schedule network and a score network, of which the former’s optimization is tied with the latter by introducing a junctional variable.
We derived a new lower bound that leads to the same training loss for the score network as in DDPMs (Ho et al., 2020), which thus enables inheriting any pre-trained score networks in DDPMs.
We also showed that training the schedule network after a well-optimized score network can be viewed as tightening the lower bound.
Followed from the theoretical results, an efﬁcient training algorithm and a noise scheduling algorithm were respectively designed for BDDMs.
Finally, in our experiments,
BDDMs showed a clear edge over the previous diffusion-based vocoders.
A T HEORETICAL DERIVATIONS FOR BDDM S
In this section, we provide the theoretical supports for the following:
• The derivation for upper bounding ˆ β n (see Appendix A.1).
• The score network θ trained with L ( t ) ddpm ( θ ) for the reverse process p θ ( x t − 1 | x t ) can be re-used for the reverse process p θ (ˆ x n − 1 | ˆ x n ) (see Appendix A.2).
• The schedule network φ can be trained with L ( n ) step ( φ ; θ ∗ ) after the score network θ is optimized.
(see Appendix A.3).
A. D ERIVING AN UPPER BOUND FOR NOISE SCALE
Since monotonic noise schedules have been successfully applied to in many prior arts including
DPMs (Ho et al., 2020; Kingma et al., 2021) and score-based methods (Song et al., 2020; Song &
Ermon, 2020), we also follow the monotonic assumption and derive an upper bound for ˆ β n as below:
Remark 1.
Suppose the noise schedule for sampling is monotonic, i.e., 0 < ˆ β 1 < . . .
<
ˆ β N < 1 , then, for 1 ≤ n < N , ˆ β n satisﬁes the following inequality:
[Equation] [Equation]
Proof.
By the general deﬁnition of noise schedule, we know that 0 < ˆ β 1 , . . .
, ˆ β N < 1 (Note: no inequality sign in between).
Given that ˆ α n = (cid:81) ni =1 (cid:113) 1 − ˆ β
i , we also have 0 <
ˆ α 1 , . .
, ˆ α t < 1 .
First, we show that ˆ β n < 1 − ˆ α 2 n +1 1 − ˆ β n +1 :
[Equation] [Equation]
Next, we show that ˆ β n < 1 − ˆ α n +1 :
[Equation]
Now, we have ˆ β n < min (cid:110) 1 − ˆ α 2 n +1 1 − ˆ β n +1 , 1 − ˆ α n +1 (cid:111) .
When 1 − ˆ α n +1 < 1 − ˆ α 2 n +1 1 − ˆ β n +1 , we can show that ˆ β n +
<
1 − ˆ
[Equation]
n
[Equation] [Equation]
[Equation]
By the assumption of monotonic sequence, we also have ˆ β n < ˆ β n +1 .
Knowing that ˆ β n +1 < 1 − ˆ α n +1 is always true, we obtain a tighter bound for ˆ β n : 0 < ˆ β n < min (cid:110)
− ˆ α 2 n +1 1 − ˆ β n +1 , ˆ β n +1 (cid:111) .
A. D ERIVING THE TRAINING OBJECTIVE FOR SCORE NETWORK
First, followed from the data distribution modeling of BDDMs as proposed in Eq.
[Equation]
we can derive a new lower bound to the log marginal likelihood as follows:
Proposition 1.
Given x t ∼ q β ( x t | x 0 )
[Equation]
the following lower bound holds for n ∈ { 2 , . . .
, N } :
[Equation]
where
[Equation]
Proof
[Equation]
[Jensen’s Inequality] ≥
[Equation]
Next
[Equation]
we show that the score network θ trained with L ( t ) ddpm ( θ ) can be re-used in BDDMs.
We ﬁrst provide the derivation for Eq.
We have
[Equation]
Proposition 2.
Suppose x t ∼ q β ( x t | x 0 ) , then any solution satisfying θ ∗ =
argmin θ L ( t ) ddpm ( θ ) , ∀ t ∈ { 1 , ..., T } , also satisﬁes θ ∗ =
argmin θ L ( n ) score ( θ ) , ∀ n ∈ { 2 , ..., N } .
Proof.
By the deﬁnition in Eq.
(4), we have
[Equation]
Here, from the training objective in Eq.
(5), since x t = α t x 0 + (cid:112) 1 − α 2 t (cid:15) n , the noise scale argument for the score network is known to be α t .
Therefore, we can use (cid:15) θ ( x t , α t ) instead of (cid:15) θ ( x t , ˆ α n ) for expanding L ( n ) score ( θ ) .
Since p θ (ˆ x n − 1 | ˆ x n = x t ) and q ˆ β (ˆ x n − 1 ; x t , (cid:15) n ) are two isotropic Gaussians with the same variance, the KL divergence is a scaled (cid:96) 2 -norm of their means’ difference:
[Equation] [Equation]
which is proportional to
[Equation]
2 as deﬁned in Eq.
[Equation] [Equation]
Next
[Equation]
we can simplify R θ (ˆ x 0 , x t ) to a reconstruction loss for ˆ x 0 :
[Equation]
[Equation] [Equation]
where p θ (ˆ x 1 | ˆ x n = x t ) can be efﬁciently sampled using the reverse process in (Song et al., 2021a).
Yet, in practice, similar to the training in (Song et al., 2021a; Chen et al., 2020; Kong et al., 2021), we dropped R θ (ˆ x 0 , x t ) when training θ .
In theory, we know that R θ (ˆ x 0 , x t ) achieves its optimal value at θ ∗ = argmin θ (cid:107) (cid:15) θ
(ˆ x 1 , ˆ α 1 ) − (cid:15) 1 (cid:107) 22 , which shares a similar objective as L ( t ) ddpm .
By minimizing
L ( t ) ddpm , we train a score network θ ∗ that best minimizes ∆ (cid:15) t := (cid:107) (cid:15) t − (cid:15) θ ∗ ( α t x 0 + (cid:112) 1 − α 2 t (cid:15) t , α t ) (cid:107)for all 1 ≤ t ≤ T .
Since the ﬁrst diffusion step has the smallest effect on corrupting ˆ x 0 (i.e., β 1 ≈ 0 ), it sufﬁces to consider a ˆ α 1 = √ 1 − β
= α 1 , in which case we can jointly minimize
R θ (ˆ x 0 , x t ) by minimizing L (1) ddpm .
In this sense, during training, given x t ∼ q β ( x t | x 0 ) , we can train the score network with the same training objective as in DDPMs and DDIMs.
Practically, it is beneﬁcial for BDDMs as we can re-use the score network θ of any well-trained DDPM or DDIM.
A. D ERIVING THE TRAINING OBJECTIVE FOR SCHEDULE NETWORK
Given that θ can be trained to maximize the log evidence with the pre-speciﬁed noise schedule β for training, the consequent question of interest in BDDMs is how to ﬁnd a fast and good enough noise schedule ˆ β ∈ R N for sampling given an optimized θ ∗ .
In BDDMs, this problem is reduced to how to effectively learn the network parameters φ .
Proposition 3.
Suppose θ has been optimized and hypothetically converged to the optimal θ ∗ , where by optimal it means that with θ ∗ we have p θ ∗ (ˆ x n −
|
ˆ x n = x t ) =
q ˆ β
(ˆ x n − 1 ; x t , (cid:15) n ) given x t ∼ q β ( x t | x 0 ) .
When ˆ β is unknown but we have x 0 =
ˆ x 0 and ˆ α n = α t , we can minimize the gap between the optimal lower bound F ( n ) score ( θ ∗ ) and log p θ ∗ (ˆ x 0 ) , i.e, log p θ ∗ (ˆ x 0 ) − F ( n ) score ( θ ∗ ) , by minimizing the following objective with respect to ˆ β n :
[Equation]
[Equation] [Equation]
where
[Equation]
Proof.
Note that ˆ x 0 = x 0 , ˆ α n = α t , x t = α t x 0 + (cid:112) 1 − α 2 t (cid:15) n and p θ ∗ (ˆ x n − 1 |
ˆ x n = x t ) =
q ˆ β
(ˆ x n − 1 ; x t , (cid:15) n ) .
When x 0 is given to p θ ∗ , we can express the probability as follows:
[Equation]
[See Eq.
(2) in (Frey, 1999)]
[Equation]
where, different from p θ ∗ (ˆ x n − 1 | ˆ x n = x t ) , from Eq.
(49) to Eq.
(50), instead of conditioning on a speciﬁc x t , when x 0 is given x t can be generated using any z ∼ N ( 0 , I ) .
From this
[Equation]
we can express the gap between log p θ ∗ (ˆ x 0 ) and F ( n ) score ( θ ∗ ) in the following form:
[Equation]
Next
[Equation]
we evaluate the above KL divergence term.
By deﬁnition
[Equation]
we have
[Equation] [Equation]
Together with Eq.
(54), we have
[Equation]
[Equation] [Equation]
[Equation] [Equation]
where
[Equation] [Equation]
As we use a schedule network φ to estimate ˆ β n from (ˆ α n +1 , ˆ β n +1 ) as deﬁned in Eq.
(13), we obtain the ﬁnal step loss for learning φ :
[Equation]
This proposed objective for training the schedule network can be interpreted as to better model the data distribution (i.e., maximizing log p
θ (ˆ x 0 ) )
by correcting the gradient scale ˆ β n for the next reverse step (from ˆ x n to ˆ x n − 1 ) given the gradient vector (cid:15) θ ∗ estimated by the score network θ ∗ .
B E XPERIMENTAL DETAILS B. C ONVENTIONAL GRID SEARCH ALGORITHM FOR DDPM S
We reproduced the grid search algorithm in (Chen et al., 2020), in which a 6-step noise schedule was searched.
In our paper, we generalized the grid search algorithm by similarly sweeping the N -step noise schedule over the following possibilities with a bin width M = 9 :
[Equation]
where ⊗ denotes the cartesian product applied on two sets.
LS-MSE was used as a metric to select the solution during the search.
When N = 6 , we resemble the GS algorithm in (Chen et al., 2020).
Note that above searching method normally does not scale up to N > 6 steps for its exponential computational cost O (9 N ) .
B. H YPERPARAMETER SETTING IN BDDM S
Algorithm 2 took a skip factor τ to control the stride for training the schedule network.
The value of τ would affect the coverage of step sizes when training the schedule network, hence affecting the predicted number of steps N for inference – the higher τ is, the shorter the predicted inference schedule tends to be.
We set τ = 66 for training the BDDM vocoders in this paper.
For initializing Algorithm 3 for noise scheduling, we could take as few as 1 training sample for validation, perform a grid search on the hyperparameters { (ˆ α N = 0 .
1 α T i, ˆ β N = 0 .
1 j ) } for i, j = 1 , ..., 9 , i.e., 81 possibilities in total, and use the PESQ measure as the selection metric.
Then, the predicted noise schedule corresponding to the maximum PESQ was stored and applied to the online inference afterward, as shown in Algorithm 4.
Note that this searching has a complexity of only
O ( M 2 ) (e.g., M = 9 in this case), which is much more efﬁcient than O ( M N ) in the conventional grid search algorithm in (Chen et al., 2020), as discussed in Section B.1.
B. I MPLEMENTATION DETAILS
Our proposed BDDMs and the baseline methods were all implemented with the Pytorch library.
The score networks for the LJ and VCTK speech datasets were trained from scratch on a single NVIDIA
Tesla P40 GPU with batch size 32 for about 1M steps, which took about 3 days.
For the model architecture, we used the same architecture as in DiffWave (Kong et al., 2021) for the score network with 128 residual channels; we adopted a lightweight GALR network (Lam et al., 2021) for the schedule network.
GALR was originally proposed for speech enhancement, so we considered it well suited for predicting the noise scales.
For the conﬁguration of the GALR network, we used a window length of 8 samples for encoding, a segment size of 64 for segmentation and only two
GALR blocks of 128 hidden dimensions, and other settings were inherited from (Lam et al., 2021).
To make the schedule network output with a proper range and dimension, we applied a sigmoid function to the last block’s output of the GALR network.
Then the result was averaged over the segments and the feature dimensions to obtain the predicted ratio: σ φ ( x ) =
AvgPool2D ( σ ( GALR ( x ))) , where GALR ( · ) denotes the GALR network, AvgPool2D ( · ) denotes the average pooling operation applied to the segments and the feature dimensions, and σ ( x ) :
= 1 / (1 + e − x ) .
The same network architecture was used for the NE approach for estimating α 2 t and was shown better than the
ConvTASNet used in the original paper (San-Roman et al., 2021).
It is also notable that the computational cost of a schedule network is indeed fractional compared to the cost of a score network, as predicting a noise scalar variable is intrinsically a relatively much easier task.
Our GALR-based schedule network, while being able to produce stable and reliable results, was about 3.6 times faster than the score network.
The training of schedule networks for BDDMs took only 10k steps to converge, which consumed no more than an hour on a single GPU.
[Table] [Table]
Regarding the image generation task, to demonstrate the generalizability of our method, we directly adopted a score network pre-trained on the CIFAR-10 dataset implemented by a third-party opensource repository.
Regarding the schedule network, to demonstrate that it does not have to use specialized architecture, we replaced GALR by the VGG11 (Simonyan & Zisserman, 2014), which was also used by as a noise estimator in (San-Roman et al., 2021).
The output dimension (number of classes) of VGG11 was set to 1 .
Similar to the setting for GALR in speech synthesis, we added a sigmoid activation to the last layer to ensure a [0 , 1] output.
Similar to the training in speech domain, we trained the VGG11-based schedule networks while freezing the score networks for 10k steps, which normally can be ﬁnished in about two hours.
Our code for the speech vocoding and the image generation experiments will be uploaded to Github after the ﬁnal decision of ICLR is released.
B. C ROWD - SOURCED SUBJECTIVE EVALUATION
All our Mean Opinion Score (MOS) tests were crowd-sourced.
We refer to the MOS scores in (Protasio Ribeiro et al., 2011), and the scoring criteria have been included in Table 3 for completeness.
The samples were presented and rated one at a time by the testers.
C A DDITIONAL EXPERIMENTS A
demonstration page at https://bilateral-denoising-diffusion-model.
github.io shows some samples generated by BDDMs trained on LJ speech and VCTK datasets.
C. M ULTI - SPEAKER SPEECH SYNTHESIS
In addition to the single-speaker speech synthesis, we evaluated BDDMs on the multi-speaker speech synthesis benchmark VCTK (Yamagishi et al., 2019).
VCTK consists of utterances sampled at 48 KHz by 108 native English speakers with various accents.
We split the VCTK dataset for training and testing: 100 speakers were used for training the multi-speaker model and 8 speakers for testing.
We trained on a 44257-utterance subset (40 hours) and evaluated on a held-out 100utterance subset.
For the score network, we used the Wavegrad architecture (Chen et al., 2020) so as to examine whether the superiority of BDDMs remains in a different dataset and with a different score network architecture.
Results are presented in Table 4.
For this multi-speaker VCTK dataset, we obtained consistent observations with that for the single-speaker LJ dataset presented in the main paper.
Again, the proposed BDDM with only 16 or 21 steps outperformed the DDPM with 1,000 steps.
To the best of our knowledge, ours was the ﬁrst work that reported this degree of superior.
When reducing to 8 steps, BDDM obtained performance on par with (except for a worse PESQ)
the costly gridsearched 8 steps (which were unscalable to more steps) in DDPM.
For NE, we could again observe a degradation from its 16 steps to 21 steps, indicating the instability of NE for the VCTK dataset likewise.
In contrast, BDDM gave continuously improved performance while increasing the step number.
C. C OMPARING DIFFERENT REVERSE PROCESSES FOR BDDM S
This section demonstrates that BDDMs do not restrict the sampling procedure to a specialized reverse process in Algorithm 4.
In particular, we evaluated different reverse processes, including that of DDPMs as shown in Eq.
(4) and DDIMs (Song et al., 2021a), for BDDMs and compared the objective scores on the generated samples.
DDIMs (Song et al., 2021a) formulate a non-Markovian generative process that accelerates the inference while keeping the same training procedure as DDPMs.
The original generative process in Eq.
(4) in DDPMs is modiﬁed into
[Equation]
where γ is a sub-sequence of length N of [1 , ..., T ] with γ N = T , and ¯ γ := { 1 , ..., T } \ γ is deﬁned as its complement; Therefore, only part of the models are used in the sampling process.
To achieve the above, DDIMs deﬁned a prediction function f ( t ) θ ( x t ) that depends on (cid:15) θ to predict the observation x 0 given x t directly:
[Equation]
By leveraging this prediction function, the conditionals in Eq.
(71) are formulated as
[Equation]
where the detailed derivation of σ t and ς can be referred to (Song et al., 2021a).
In the original
DDIMs, the accelerated reverse process produces samples over the subsequence of β indexed by γ : ˆ β = { β n | n ∈ γ } .
In BDDMs, to apply the DDIM reverse process, we use the ˆ β predicted by the schedule network in place of a subsequence of the training schedule β .
Finally.
the objective scores are given in Table 5.
Note that the subjective evaluation (MOS) is omitted here since the other assessments above have shown that the MOS scores are highly correlated with the objective measures, including STOI and PESQ.
They indicate that applying BDDMs to either DDPM or DDIM reverse process leads to comparable and competitive results.
Meanwhile, the results show some subtle differences: BDDMs over a DDPM reverse process gave slightly better samples in terms of signal error and consistency metrics (i.e., LS-MSE and MCD), while BDDM over a DDIM reverse process tended to generate better samples in terms of intelligibility and perceptual metrics (i.e., STOI and PESQ).
[Table] [Table]
C. U NCONDITIONAL IMAGE GENERATION
For the unconditional image generation task, we evaluated the proposed BDDMs on the benchmark
CIFAR-10 (32 × 32) dataset.
The score functions, including those initially proposed in DDPMs (Ho et al., 2020) or DDIMs (Song et al., 2021a) and those pre-trained in the above third-party implementations, are all conditioned on a discrete step-index.
We estimated the noise schedule ˆ β in continuous space using the VGG11 schedule network and then mapped it to discrete time schedule using the approximation method in (Kong & Ping, 2021).
Table 6 shows the performances of different sampling methods for DDPMs in CIFAR-10.
By setting the maximum number of sampling steps ( N ) for noise scheduling, we can fairly compare the improvements achieved by BDDMs against related methods in the literature in terms of FID.
Remarkably, BDDMs with 100 sampling steps not only surpassed the 1000-step DDPM baseline, but also produced the SOTA FID performance amongst all generative models using less than or equal to 100 sampling steps.
[Equation]