BDDM: B ILATERAL D ENOISING D IFFUSION M ODELS FOR F AST AND H IGH -Q UALITY S PEECH S YNTHESIS
Anonymous authors
Paper under double-blind review
A BSTRACT
Diffusion probabilistic models (DPMs) and their extensions have emerged as competitive generative models yet confront challenges of efﬁcient sampling.
We propose a new bilateral denoising diffusion model (BDDM) that parameterizes both the forward and reverse processes with a schedule network and a score network, which can train with a novel bilateral modeling objective.
We show that the new surrogate objective can achieve a lower bound of the log marginal likelihood tighter than a conventional surrogate.
We also ﬁnd that BDDM allows inheriting pre-trained score network parameters from any DPMs and consequently enables speedy and stable learning of the schedule network and optimization of a noise schedule for sampling.
Our experiments demonstrate that BDDMs can generate high-ﬁdelity audio samples with as few as 3 sampling steps.
Moreover, comparing to other state-of-the-art diffusion-based neural vocoders, BDDMs produce comparable or higher quality samples indistinguishable from human speech, notably with only 7 sampling steps (143x faster than Wdifferent aspect of WaveGrad and 28.6x faster than DiffWave).
I NTRODUCTION
Deep generative models have shown a tremendous advancement in speech synthesis (van den Oord et al., 2016; Kalchbrenner et al., 2018;
Prenger et al., 2019; Kumar et al., 2019a; Kong et al., 2020b;
Chen et al., 2020; Kong et al., 2021).
Successful generative models can be mainly divided into two branches – the GAN (Goodfellow et al., 2014) based approach and the likelihood-based approach.
The former uses an adversarial training procedure, but the training is sometimes unstable, and the relevant training objectives are not suitable to compare different GAN models; the latter uses loglikelihood or surrogate objectives for training, but they also have intrinsic limitations.
For example, the autoregressive models (van den Oord et al., 2016; Kalchbrenner et al., 2018) have exceedingly slow sampling speed, and poor scaling properties on high-dimensional data, as its sampling is inherently a sequential process.
Likewise, the ﬂow-based models (Dinh et al., 2016; Kingma & Dhariwal, 2018; Chen et al., 2018; Papamakarios et al., 2021) rely on specialized architectures to build a normalized probability model so that the model training is less parameter-efﬁcient and typically more difﬁcult than other likelihood-based generative models (Bond-Taylor et al., 2021).
On the other hand, for those using the surrogate objectives, such as the evidence lower bound in variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014; Maaløe et al., 2019) and the contrastive divergence in energy-based models (Hinton, 2002; Carreira-Perpinan & Hinton, 2005), they improve inference speed but often yield lower-quality samples than autoregressive and GAN-based models and typically only work well for low-dimensional data (Bond-Taylor et al., 2021).
An up-and-coming class of likelihood-based models is the diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015), which shed light on the generation of high-quality samples comparable or even superior (Dhariwal & Nichol, 2021) to the state-of-the-art (SOTA) GAN-based models.
Sohl-Dickstein et al.
(2015) introduced the idea of using an iterative forward process for destroying the structure of a given distribution while learning the reverse process for restoring the structure in the data.
With a different modeling objective, the score-based generative models (Song & Ermon, 2019) used a neural network trained with the score matching (Hyvarinen & Dayan, 2005) to produce samples via Langevin dynamics.
As a seminal work of DPMs, Ho et al.
(2020) proposed the denoising diffusion probabilistic models (DDPMs), which exploited a connection (Vincent, 2011) between the above two ideas that the diffusion framework can be viewed as a speciﬁc approach to provide support to score matching.
For the ﬁrst time, DDPMs present high-quality image synthesis results on par with GAN models, especially on several higher-resolution image benchmarks.
Subsequently, score matching and denoising diffusion models were applied to speech syntheses, such as Wavegrad (Chen et al., 2020) and a parallel contribution Diffwave (Kong et al., 2021), and demonstrated the capability of generating high ﬁdelity audio samples outperforming non-autoregressive GAN-based models (Yamamoto et al., 2020; Kumar et al., 2019b;
Yang et al., 2020; Bi´nkowski et al., 2020) meanwhile matching the quality of SOTA autoregressive methods (Chen et al., 2020).
Despite the compelling results, the diffusion generative models are two to three orders of magnitude slower than other generative models such as GANs and VAEs.
Their major limitation is that they require up to thousands of diffusion steps during training to learn the target distribution, and therefore a large number of reverse steps are often required at sampling time.
Recently, extensive investigations have been conducted to reduce the sampling steps for efﬁciently generating high-quality samples, which we will discuss in the related work in Section 2.
Distinctively, we conceived that we may train a neural network to efﬁciently and adaptively estimate a much shorter noise schedule for sampling that achieves generation performances comparable or even superior to the conventional DPMs.
With such an incentive, after introducing the conventional DPMs as our background in Section 3, we propose in Section 4 bilateral denoising diffusion models (BDDMs), named after a bilateral modeling perspective – parameterizing the forward and reverse processes with a schedule network and a score network, respectively.
We theoretically derive that the schedule network should be trained after the score network is optimized. For training the schedule network, we propose a novel objective to minimize the gap between a newly derived lower bound and the log marginal likelihood.
We describe the training algorithm as well as the fast and high-quality sampling algorithm in Section 5.
The training of the schedule network converges very fast using our newly derived objective, and its training only adds negligible overhead to DDPM’s.
In Section 6, our neural vocoding experiments demonstrated that BDDMs can generate high-ﬁdelity samples with as few as 3 sampling steps, and produce speech samples indistinguishable from human speech with only 7 sampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).  R ELATED W ORK
Prior works showed that noise scheduling is crucial for efﬁcient and high-ﬁdelity sample generation in diffusion probabilistic models. DDPMs (Ho et al., 2020) shared a linear noise schedule for both training and sampling, which, however, requires thousands of sampling iterations to obtain competitive results. To speed up the sampling process, one class of related work, including (Chen et al., 2020; Kong et al., 2021; Nichol & Dhariwal, 2021), attempts to use a separate, shorter noise schedule for sampling. For clarity, we thereafter denote the training noise schedule as β ∈ R T and the sampling noise schedule as ˆ β ∈ R N with N < T .
In particular, Chen et al.
(2020) applied a grid search (GS) algorithm to select ˆ β .
Unfortunately, GS becomes prohibitively slow when N grows large, e.g., N = 6 took more than a day on a single NVIDIA Tesla P40 GPU.
This is because the time costs of GS algorithm grow exponentially with N , i.e., O (9 N ) with 9 bins as the default setting in Chen et al.
Instead of searching, Kong et al.
(2021) devised a fast sampling (FS) algorithm based on an expert-deﬁned 6-step noise schedule for their score network.
However, this speciﬁcally tuned noise schedule is hard to generalize to other score networks, tasks, or datasets.
Another class of noise scheduling methods searches for a subsequence of time indices of the training noise schedule, which we called the time schedule .
DDIMs (Song et al., 2021a) introduced an accelerated reverse process that relies on a pre-speciﬁed time schedule.
A linear and a quadratic time schedule were used in DDIMs and showed superior generation quality over DDPMs within 10 tosampling steps.
Nichol & Dhariwal (2021) proposed a re-scaled noise schedule for fast sampling, but this also requires pre-specifying the time schedule and the training noise schedule.
Nichol &
Dhariwal (2021) also proposed learning variances for the reverse processes, whereas the variances of the forward processes, i.e., the noise schedule, which affected both the means and variances of the reverse processes, were not learnable.
According to the results in (Song et al., 2021a;
Nichol &
Dhariwal, 2021), using a linear or quadratic time schedule resulted in quite different performances in different datasets, implying that the optimal choice of schedule varies with the datasets.
So, there remains a challenge in ﬁnding a short and effective schedule for fast sampling on different datasets.
Notably, Kong & Ping (2021) proposed a method to map a noise schedule to a time schedule for fast sampling.
In this sense, searching for a time schedule becomes a sub-set of the noise scheduling problem, which resembles the above category of methods.
Although DPMs (Sohl-Dickstein et al., 2015) and DDPMs (Ho et al., 2019) mentioned that the noise schedule could possibly be learned by re-parameterization, the approach was not investigated in their works.
Closely related works that learn a noise schedule emerged until very recently.
San-Roman et al.
(2021) proposed a noise estimation (NE) method, which trained a neural net with a regression loss to estimate the noise scale from the noisy sample at each time point, and then predicted the next noise scale.
However, NE requires a prior assumption of the noise schedule following a linear or Fibonacci rule.
Most recently, a concurrent work to ours by Kingma et al.
(2021) jointly trained a neural net to predict the signal-to-noise ratio (SNR) by maximizing the variational lower bound.
The SNR was then used for noise scheduling.
Different from ours, this scheduling neural net only took t as input and is independent to the noisy sample generated during the loop of sampling process.
Intrinsically, with limited information about the sampled data, the predicted SNR could deviate from the actual SNR of the noisy data during sampling.
B ACKGROUND 3. D IFFUSION PROBABILISTIC MODELS (DPM S )
Given i.i.d.
samples { x 0 ∈ R D } from an unknown data distribution p data ( x 0 ) , diffusion probabilistic models (DPMs) (Sohl-Dickstein et al., 2015) deﬁne a forward process q ( x 1: T | x 0 ) =
(cid:81)
Tt =1 q ( x t | x t − 1 ) that converts any complex data distribution into a simple, tractable distribution after T steps of diffusion.
A reverse process p θ ( x t − 1 | x t ) parameterized by θ is used to model the data distribution: p θ ( x 0 )
= (cid:82) π
( x T ) (cid:81)
Tt =1 p θ ( x t − 1 | x t ) d x 1: T , where π ( x T ) is the prior distribution for starting the reverse process.
Then, the variational parameters θ can be learned by maximizing the standard log evidence lower bound (ELBO):
[Equation]
3. D ENOISING D IFFUSION P ROBABILISTIC M ODELS (DDPM S )
As an extension to DPMs, de-noising diffusion probabilistic models (DDPMs) (Ho et al., 2020) applied the score matching technique (Hyvarinen & Dayan, 2005; Song & Ermon, 2019) to deﬁne the reverse process.
In particular, DDPMs considered a Gaussian diffusion process parameterized by a noise schedule β ∈ R T with 0 < β 1 , . .
. , β T < 1 :
[Equation]
T (cid:89) t = q β t ( x t | x t − 1 ) , where q β t ( x t | x t − 1 ) :
=
N ( (cid:112) 1 − β t x t − 1 , β t I ) .
Based on the nice property of isotropic Gaussians, one can express x t directly conditioned on x 0 : q β ( x t | x 0 ) =
N ( α t x 0 , (1 − α 2 t ) I ) , where α t = t (cid:89) i = (cid:112) 1 − β i .
To revert this forward process, DDPMs employ a score network 1 (cid:15) θ ( x t , α t ) to deﬁne p θ ( x t − 1 | x t ) :
= N (cid:32) 1 √ 1 − β t (cid:32) x t − β t
[Equation]
where Σ t is the co-variance matrix deﬁned for the reverse process.
Ho et al.
(2020) showed that setting Σ t = ˜ β t I = 1 − α 2 t − 1 1 − α 2 t β t
I is optimal for a deterministic x 0 , while setting Σ t =
β t
I is optimal for a white noise x 0 ∼ N ( 0 , I ) .
Alternatively, Nichol & Dhariwal (2021) proposed learnable variances by interpolating the two optimals with a jointly trained neural network, i.e., Σ t,θ ( x ) :=
diag (exp( v θ ( x ) log β t + (1 − v θ ( x )) log ˜ β t )) , where v θ ( x ) ∈ R D is a trainable network.
Note that the calculation of the complete ELBO in Eq.
(1) requires T forward passes of the score network, which would make the training computationally prohibitive for a large T .
To feasibly train the score network, instead of computing the complete ELBO, Ho et al.
(2020) proposed an efﬁcient training mechanism by sampling from a discrete uniform distribution: t ∼ U{ 1 , ..., T } , x 0 ∼ p data ( x 0 ) , (cid:15) t ∼ N ( 0 , I ) at each training iteration to compute the training loss: L
[Equation]
(5) which is a re-weighted form of D KL ( q β ( x t − 1 | x t , x 0 )
||
p θ ( x t − 1 | x t )) .
Ho et al.
(2020) reported that the re-weighting effectively worked for learning θ , yet we demonstrate it is deﬁcient for learning the noise schedule β in our ablation experiment in Section 6.2.
B ILATERAL DENOISING DIFFUSION MODELS (BDDM S ) 4. P ROBLEM FORMULATION
For fast sampling with DPMs, we strive for a much shorter noise schedule ˆ β for sampling than the noise schedule β for training.
As shown in Fig.
1, we speciﬁcally deﬁne two separate diffusion processes corresponding to the two noise schedules, β and ˆ β , respectively.
The upper diffusion process parameterized by β is the same as the one deﬁned in Eq. 2, whereas the lower process is deﬁned as q ˆ β
(ˆ x 1: N | ˆ x 0 ) = (cid:81)
Nn =1 q ˆ β (ˆ x n | ˆ x n − 1 ) with much fewer diffusion steps ( N (cid:28) T ). In our considered problem, β is given while ˆ β is unknown.
The goal is to ﬁnd a ˆ β for the reverse process p θ (ˆ x n − 1 | ˆ x n ; ˆ β ) such that ˆ x 0 can be effectively recovered from ˆ x N with N reverse steps.
Although many prior arts (Ho et al., 2020; Chen et al., 2020; Song et al., 2021a; San-Roman et al., 2021) directly applied the shortened linear noise schedule to the reverse process, we argue that these are sub-optimal solutions. Theoretically, the diffusion process speciﬁed by a new shortened noise schedule ˆ β is fundamentally different from the one we used to train the score network θ .
Therefore, we are not guaranteed whether θ still works in reverting the new diffusion process. The above issue motivated a novel modeling perspective to establish a link between ˆ β and the score network θ . As a starting point, we consider an N = (cid:98) T/τ (cid:99) , where 1 ≤ τ < T is a hyperparameter controlling the step size such that each diffusion step between two consecutive variables in the lower diffusion process corresponds to τ diffusion steps in the upper one. Based on Eq.
(2), we deﬁne the following: [Equation] where ˆ β n +1 = 1 − α 2 t + τ α 2 t , and x t is an intermediate diffused variable we introduced to link the two differently indexed diffusion sequences.
We call it a junctional variable, which can be easily generated during training as we have x 0 and β given.
Unfortunately, for the reverse process where x 0 is not given, the junctional variable is intractable. Then, the proposed bilateral denoising diffusion model (BDDM) introduces a schedule network φ . We will demonstrate that if we can ideally estimate the score for the β -parameterized diffusion process using the score network θ ∗ , the schedule network φ can be trained to learn an optimal noise schedule ˆ β ( φ ) , in which, by optimality, we mean to maximize the data likelihood of generated samples given the score estimated by θ ∗ , i.e., p θ ∗ (ˆ x 0 ) . Next, we present a formal deﬁnition of a
BDDM and derive the relevant training losses in more detail. 4.2. S CORE N ETWORK We recall that a DDPM starts the reverse process with a white noise x T ∼ N ( 0 , I ) and takes T steps to recover the data distribution: [Equation]
a BDDM, in contrast, reverts a shorter sequence of diffusion random variables with only n steps, starting from a junctional variable x t : [Equation] where 2 ≤ n ≤ N , and q ˆ β (ˆ x n − 1 | x t ; (cid:15) n ) comes from a reparameterization deﬁned as the following: [Equation]
x n − 1 | x t ; (cid:15) n ) [Equation] = q
[Equation] where ˆ α n = (cid:81) ni =1 (cid:113) 1 − ˆ β i , (cid:15) n = x t − ˆ α n x 0 √ 1 − ˆ α 2 n , and x t ∼ q β ( x t | x 0 ) is the junctional variable that maps x t to ˆ x n with an approximate index t ∼ U{ ( n − 1) τ, ..., nτ } .
With this deﬁnition, we show in Appendix A.2 that a new form of lower bound to the log marginal likelihood can be derived: log p θ (ˆ x 0 ) ≥ [Equation]
[Equation]
θ ) − R θ (ˆ x 0 , x t ) , where
[Equation]
Then, we show that the score network θ for reverting the diffusion process over ˆ x 0: N can also be trained with L ( t ) ddpm ( θ ) even without knowing ˆ β .
Essentially, although the newly derived lower bound does not modify the loss for training the score network, it explains the relationship between the score network θ and ˆ x 0: N . This can be further exploited for learning ˆ β , as presented below.
4.2. SCHEDULE NETWORK
In BDDMs, a schedule network is introduced to the forward process by re-parameterizing ˆ β n as ˆ β n ( φ ) = f φ (cid:16) x t , ˆ β n +1 (cid:17) such that Eq. (9) becomes as follows:
[Equation]
where at each training step, by sampling t ∼ U{ ( n − 1) τ, ..., nτ } , we have x t ∼ q β ( x t | x 0 ) and ˆ β n +1 = 1 − α 2 t
+ τ α 2 t .
Through the re-parameterization, the task of noise scheduling, i.e., searching for ˆ β , can now be reformulated as training a schedule network f φ that recursively estimates datadependent variances. The schedule network lets ˆ β n be fully dependent on the current noisy sample x t and the ˆ β n +1 , which is beneﬁcial for noise scheduling from a reverse direction at inference time.
We expect f φ to be able not only to estimate the absolute noise scale of the inputted sample x t , but also to predict the relative change of noisiness comparable to τ steps of diffusion speciﬁed by β in the reverse direction. These challenge a proper deﬁnition of f φ and an effective training objective for learning φ , which will both be discussed in the following.
Monotonic noise schedules have been successfully applied to in prior arts including DPMs (Ho et al., 2020; Kingma et al., 2021) and score-based methods (Song et al., 2020; Song & Ermon, 2020).
We also follow the monotonic assumption and derive an upper bound for ˆ β n as below:
Remark 1.
Suppose the noise schedule for sampling is monotonic, i.e., 0 < ˆ β 1 < . . .
<
ˆ β N < 1 , then, for 1 ≤ n < N , ˆ β n satisﬁes the following inequality:
[Equation]
The proof is presented in Appendix A.1.
By multiplying the upper bound by a ratio estimated by a network σ φ : R D (cid:55)→ (0 , 1) , we then deﬁne f
[Equation]
[Equation]
, where the network parameter set φ is learned to estimate the ratio between two consecutive noise scales ( ˆ β n and ˆ β n +1 ) from the current noisy observation x t .
Consequently, at inference time for noise scheduling, starting from a maximum reverse steps ( N ) and two hyperparameters (ˆ α N , ˆ β N ) , we ancestrally, for n from N to 1 , predict the noise scale ˆ β n = f φ (cid:16) ˆ x n , ˆ β n +1 (cid:17) and cumulatively update the product ˆ α n =
[Equation]
4.2. T RAINING OBJECTIVE
Since we show, in Appendix A.2, that the score network parameters θ can be trained similarly to
DDPMs with L ( t ) ddpm ( θ ) , the remaining challenge would be the training of the schedule network f φ .
In Appendix A.3, the theoretical results suggest that φ should be trained after θ is well-optimized.
In a realistic setting, we assume the optimized θ ∗ to be closed to the hypothetically optimal values.
Given a well-optimized θ ∗ , we optimize the following loss w.r.t φ :
[Equation]
where δ t =
(cid:112) 1 − α 2 t . For training, we use ˆ β n ( φ ) = f φ (cid:16) α t x 0 + (cid:112) 1 − α
[Equation]
n , 1 − α
[Equation]
+ τ α 2 t (cid:17) , and draw t ∈ U{ ( n − 1) τ, . . .
, nτ } . The derivation for L ( n ) step ( φ ; θ ∗ ) is described in Appendix A.3.
A LGORITHMS : TRAINING , NOISE SCHEDULING , AND SAMPLING 5. T RAINING SCORE AND SCHEDULE NETWORKS
Following the theoretical result in Appendix A.3, θ should be optimized before learning φ .
To train the score network (cid:15) θ , we refer to the settings in (Ho et al., 2020; Chen et al., 2020; Song et al., 2021a) to deﬁne β as a linear noise schedule: β t = β start + tT
( β end − β start ) , for 1 ≤ t ≤ T, where β start and β end are two hyperparameter that speciﬁes the start value and the end value.
This results in Algorithm 1, which resembles the training algorithm in (Ho et al., 2020).
Next, based on the converged score network θ ∗ , we train the schedule network φ .
Similar to the training of the score network, we draw an n ∼ U{ 2 , . . . , N } at each training step, and then draw a t ∼ U{ ( n − 1) τ, ..., nτ } .
These together can be simpliﬁed into directly drawing t ∼ U{ 2 , ..., T − τ } .
Then, we sequentially compute the variables needed for calculating L ( n ) step ( φ ; θ ∗ ) , as presented in
[Table]
[Table] [Table] + τ α 2 t (cid:27) σ φ ( x t ) 9: C = 4 − 1 log( δ 2 n / ˆ β n ) + 2 − 1 D (cid:16) ˆ β n /δ 2 n − 1 (cid:17) 10: L ( n ) step = δ 2 n 2( δ 2 n − ˆ β n ) (cid:13)(cid:13)(cid:13) (cid:15) n − ˆ β n δ 2 n (cid:15) θ ∗ ( x t , α t ) (cid:13)(cid:13)(cid:13) 2 2 + C 11: Take a gradient descent step on ∇ φ L ( n ) step 12: until converged
Algorithm
1: Given θ ∗ , { ˆ β n } N s n =1 , ˆ x N s ∼ N ( 0 , I ) 2: { ˆ α n } N s n =1 = (cid:26)(cid:81) ni =1 (cid:113) 1 − ˆ β n (cid:27) N s n =1 3: for n = N s to 1 do 4: ˆ x n − 1 ∼ p θ ∗ (ˆ x n − 1 | ˆ x n ; ˆ α n , ˆ β n ) 5: end for 6: return ˆ x Algorithm 2.
We observed that, although a linear schedule is used to deﬁne β , the noise schedule for sampling ˆ β predicted by f φ is not limited into a linear one.
5. N OISE SCHEDULING FOR FAST AND HIGH - QUALITY SAMPLING
After the score network and the schedule network are trained, the inference procedure can divide into two phases: (1) the noise scheduling phase and (2) the sampling phase.
First, we run the noise scheduling process similarly to a sampling process with N iterations maximum.
Different from training, where α t is forward-computed, ˆ α n is instead a backward-computed variable (from N to 1 ) that may deviates from the forward one because { ˆ β i } n − 1 i are unknown in the noise scheduling phase during inference.
To start noise scheduling, we ﬁrst set two hyperparameters: ˆ α N and ˆ β N .
We use β 1 , the smallest noise scale seen in training, as a threshold to early stop the noise scheduling process, so that we can ignore small noise scales ( < β 1 ) that were never seen by the score network. Overall, the noise scheduling process is presented in Algorithm 3.
In practice, we apply a grid search algorithm of M bins to Algorithm 3, which takes O ( M 2 ) time, to ﬁnd proper values for (ˆ α N , ˆ β N ) .
We used M = 9 as in (Chen et al., 2020).
The grid search for our noise scheduling algorithm can be evaluated on a small subset of the training samples.
Empirically, even as few as 1 sample for evaluation works well in our algorithm.
Finally, given the predicted noise schedule ˆ β ∈ R N s , we generate samples with N s sampling steps, as shown in Algorithm 4.
E XPERIMENTS
We conducted a series of experiments on neural vocoding tasks to evaluate the proposed BDDMs.
First, we compared BDDMs against several strongest models that have been published: the mixture of logistics (MoL) WaveNet
(Oord et al., 2018) implemented in (Yamamoto, 2020), the WaveGlow (Prenger et al., 2019) implemented in (Valle, 2020), the MelGAN (Kumar et al., 2019a) implemented in (Kumar, 2019), the HiFi-GAN (Kong et al., 2020b) implemented in (Kong et al., 2020a) and the two most recently proposed diffusion-based vocoders, i.e., WaveGrad (Chen et al., 2020) and
DiffWave (Kong et al., 2021), both re-implemented in our code.
The hyperparameter settings of
BDDMs and all these models are detailed in Appendix B.
In addition, we also compared BDDMs to a variety of scheduling and acceleration techniques applicable to DDPMs, including the grid search (GS) approach in WaveGrad, the fast sampling (FS) approach based on a user-deﬁned 6-step schedule in DiffWave, the DDIMs (Song et al., 2021a) and [Table]
[Table]
For fair and reproducible comparison with other models and approaches, we used the LJSpeech dataset (Ito & Johnson, 2017), which consists of 13,100 22kHz audio clips of a female speaker.
All diffusion models were trained on the same training split as in (Chen et al., 2020).
We also replicated the comparative experiment of neural vocoding using a multi-speaker VCTK dataset (Yamagishi et al., 2019) as presented in Appendix C and obtained a result consistent with that obtained from the LJSpeech dataset.
6. S AMPLING QUALITY IN OBJECTIVE AND SUBJECTIVE METRICS
To assess the quality of each generated audio sample, we used both objective and subjective measures for comparing different neural vocoders given the same ground-truth spectrogram s as the condition, i.e., (cid:15) θ
( x , s , α t ) .
Speciﬁcally, we used two scale-invariant metrics: the perceptual evaluation of speech quality (PESQ) (Rix et al., 2001) and the short-time objective intelligibility (STOI) (Taal et al., 2010) to measure the noisiness and the distortion of the generated speech relative to the reference speech.
Mean opinion score (MOS) was also used as a subjective metric for evaluating the naturalness of the generated speech.
The assessment scheme of MOS is included in Appendix B.
In Table 1, we compared BDDMs against the state-of-the-art (SOTA) vocoders.
To predict noise schedules with different sampling steps (3, 7, and 12), we set three pairs of { ˆ α N , ˆ β N } for BDDMs by running on Algorithm 3 a quick hyperparameter grid search, which is detailed in Appendix B.
Among the 9 evaluated vocoders, only our proposed BDDMs with 7 and 12 steps and DiffWave with 200 steps showed no statistic-signiﬁcant difference from the ground-truth in terms of MOS.
Moreover, BDDMs signiﬁcantly outspeeded DiffWave in terms of RTFs.
Notably, previous diffusionbased vocoders achieved high MOS scores at the cost of an unacceptable RTF for industrial deploy- ment.
In contrast, BDDMs managed to achieve a high standard of generation quality with onlysampling steps (143x faster than WaveGrad and 28.6x faster than DiffWave).
In Table 2, we evaluated BDDMs and alternative accelerated sampling methods, which used the same score network for a pair-to-pair comparison.
The GS method performed stably when the step number was small (i.e., N ≤ 6 ) but not scalable to more step numbers, which were therefore bypassed in the comparisons of 7 and 12 steps.
The FS method by Song et al.
(2021a) was linearly interpolated to 3 and 7 steps for a fair comparison.
Comparing its 7-step and 3-step results, we observed that the FS performance degraded drastically.
Both the DDIM and the NE methods were stable across all the steps but were not performing competitively enough.
In comparison, BDDMs consistently attained the leading scores across all the steps.
This evaluation conﬁrmed that BDDM was superior to other acceleration methods for DPMs in terms of both stability and quality.
6. A BLATION S TUDY AND A NALYSIS
We attribute the primary advantage of BDDMs to the newly derived objective L ( n ) step for learning φ .
To better reason about this, we performed an ablation study, where we substituted the proposed loss with the standard negative ELBO for learning φ as mentioned by Sohl-Dickstein et al. (2015).
We plotted the network outputs with different training losses in Fig.
It turned out that, when using
L ( n ) elbo to learn φ , the network output rapidly collapsed to zero within several training steps; whereas, the network trained with L ( n ) step produced ﬂuctuating outputs.
The ﬂuctuation is a desirable property showing the network properly predicts t -dependent noise scales, as t is a random time step drawn from a uniform distribution in training.
By considering N = T , we also empirically validated that F ( t ) bddm ≥ F ( t ) elbo with their respective values at t ∈ [20 , 180] using the same optimized theta θ ∗ , and each value with 95% conﬁdence intervals, as illustrated in Fig.
In this experiment, we used the LJ speech dataset and set T = 200 and τ = 20 .
Notably, we dropped their common entropy term R
θ (ˆ x 0 , x t ) < 0 to mainly compare their KL divergences.
This explains those positive lower bound values in the plot.
The graph shows that our proposed bound F ( t ) bddm is always a tighter lower bound than the standard one across all examined t .
Moreover, we found that F ( t ) bddm attained low values with a relatively much lower variance around t ∈ [20 , 50] , whereas F ( t ) elbo became highly volatile.
This implies that F ( t ) bddm better tackles the difﬁcult training part, when t is approaching zero such that the score becomes more challenging to estimate.
C ONCLUSIONS
We introduced the bilateral denoising diffusion models (BDDMs), which parameterizes the forward and reverse processes with a schedule network and a score network.
From a bilateral modeling perspective, we derived a new lower bound that leads to the same training loss as in DDPMs (Ho et al., 2020), which thus enables inheriting any pre-trained score networks in DDPMs.
We also showed that training the schedule network after a well-optimized score network can be viewed as tightening the lower bound.
Followed from the theoretical results, an efﬁcient training algorithm and a noise scheduling algorithm were respectively designed for BDDMs.
Finally, in our experiments,
BDDMs showed a clear edge over the previous diffusion-based vocoders.
A T HEORETICAL DERIVATIONS FOR BDDM S
In this section, we provide the theoretical supports for the following:
• The derivation procedures for Remark 1 (see Appendix A.1).
• The score network θ trained with L ( t ) ddpm ( θ ) for the reverse process p θ ( x t − 1 | x t ) can be re-used for the reverse process p θ (ˆ x n − 1 | ˆ x n ) (see Appendix A.2).
• The scheduling network φ can be trained with L ( n ) step ( φ ; θ ∗ ) after the score network θ is optimized.
(see Appendix A.3).
A. D ERIVING AN UPPER BOUND FOR NOISE SCALE
Remark 1.
Suppose the noise schedule for sampling is monotonic, i.e., 0 < ˆ β 1 < . . .
<
ˆ β N < 1 , then, for 1 ≤ n < N , ˆ β n satisﬁes the following inequality:
[Equation]
Proof.
By the general deﬁnition of noise schedule, we know that 0 < ˆ β 1 , . . .
, ˆ β N < 1 (Note: no inequality sign in between).
Given that ˆ α n = (cid:81) ni =1 (cid:113) 1 − ˆ β
i , we also have 0 <
ˆ α 1 , . .
, ˆ α t < 1 .
First, we show that ˆ β n < 1 − ˆ α 2 n +1 1 − ˆ β n +1 : [Equation]
[Equation]
Next, we show that ˆ β n < 1 − ˆ α n +1 :
[Equation]
[Equation] Now, we have ˆ β n < min (cid:110) 1 − ˆ α 2 n +1 1 − ˆ β n +1 , 1 − ˆ α n +1 (cid:111) .
When 1 − ˆ α n +1 < 1 − ˆ α 2 n +1 1 − ˆ β n +1 , we can show that ˆ β n +
<
1 − ˆ α n +1 :
[Equation]
By the assumption of monotonic sequence, we also have ˆ β n < ˆ β n +1 .
Knowing that ˆ β n +1 < 1 − ˆ α n +1 is always true, we obtain a tighter bound for ˆ β n : 0 < ˆ β n < min (cid:110)
− ˆ α 2 n +1 1 − ˆ β n +1 , ˆ β n +1 (cid:111) .
A. D ERIVING THE TRAINING OBJECTIVE FOR SCORE NETWORK
First, followed from the data distribution modeling of BDDMs as proposed in Eq.
[Equation]
we can derive a new lower bound to the log marginal likelihood as follows:
Proposition 1.
Given an x t ∼ q β ( x t | x 0 ) , the following lower bound holds for n ∈ { 2 , . . . , N } :
[Equation]
where
[Equation]
Proof.
[Equation]
Inequality
[Equation]
Next, we show that the score network θ trained with L ( t ) ddpm ( θ ) can be re-used in BDDMs. We ﬁrst provide the derivation for Eq. We have q ˆ β (ˆ x n − 1 | x t ; (cid:15) n ) :=
[Equation]
Proposition 2. Suppose x t ∼ q β ( x t | x 0 )
[Equation]
then any solution satisfying θ ∗ =
argmin θ L ( t ) ddpm ( θ ) , ∀ t ∈ { 1 , ..., T } , also satisﬁes θ ∗ =
argmin θ L ( n ) score ( θ ) , ∀ n ∈ { 2 , ..., N } .
Proof.
By the deﬁnition in Eq.
(4), we have
[Equation]
Here, from the training objective in Eq.
(5), since x t = α t x 0 + (cid:112) 1 − α 2 t (cid:15) n , the noise scale argument for the score network is known to be α t .
Therefore, we can use (cid:15) θ ( x t , α t ) instead of (cid:15) θ ( x t , ˆ α n ) for expanding L ( n ) score ( θ ) . Since p θ ( ˆ x n − 1 | ˆ x n = x t ) and q ˆ β ( ˆ x n − 1 | x t ; (cid:15) n ) are two isotropic Gaussians [Equation] which is proportional to L
[Equation]
[Equation]
[Equation]
2 as deﬁned in Eq.
Thus
[Equation]
argmin
[Equation]
Next
[Equation] we can simplify R θ (ˆ x [Equation] x t ) to a reconstruction loss for ˆ x 0 :
[Equation]
[Equation]
[Equation]

 , (46) where p θ (ˆ x 1 | ˆ x n = x t ) can be efﬁciently sampled using the reverse process in (Song et al., 2021a). Yet, in practice, similar to the training in (Song et al., 2021a; Chen et al., 2020; Kong et al., 2021), we dropped R θ (ˆ x 0 , x t ) when training θ . In theory, we know that R θ (ˆ x 0 , x t ) achieves its optimal value at θ ∗ = argmin θ (cid:107) (cid:15) θ (ˆ x 1 , ˆ α 1 ) − (cid:15) 1 (cid:107) 22 , which shares a similar objective as L ( t ) ddpm .
By minimizing
L ( t ) ddpm , we train a score network θ ∗ that best minimizes ∆ (cid:15) t := (cid:107) (cid:15) t − (cid:15) θ ∗ ( α t x 0 + (cid:112) 1 − α 2 t (cid:15) t , α t ) (cid:107)for all 1 ≤ t ≤ T .
{+Since the ﬁrst diffusion step has the smallest effect on corrupting ˆ x 0 (i.e., β 1 ≈ 0 ), it sufﬁces to consider a ˆ α 1 = √ 1 − β 
+}
= α 1 , in which case we can jointly minimize
R θ (ˆ x 0 , x t ) by minimizing L (1) ddpm .
In this sense, during training, given x t ∼ q β ( x t | x 0 ) , we can train the score network with the same training objective as in DDPMs and DDIMs. Practically, it is beneﬁcial for BDDMs as we can re-use the score network θ of any well-trained DDPM or DDIM. A. D ERIVING THE TRAINING OBJECTIVE FOR SCHEDULING NETWORK Given that θ can be trained to maximize the log evidence with the pre-speciﬁed noise schedule β for training, the consequent question of interest in BDDMs is how to ﬁnd a fast and good enough noise schedule ˆ β ∈ R N for sampling given an optimized θ ∗ .
In BDDMs, this problem is reduced to how to effectively learn the network parameters φ . Proposition 3. Suppose θ has been optimized and hypothetically converged to the optimal θ ∗ {+, where by optimal it means that with θ ∗ we have p θ ∗ (ˆ x n − 
 |+}
ˆ x n = x t ) = q ˆ β (ˆ x n − 1 | x t ; (cid:15) n ) given a x t ∼ q β ( x t | x 0 ) . When ˆ β is unknown but we have x 0 =
ˆ x 0 and ˆ α n = α t , we can minimize the gap between the optimal lower bound F ( n ) score ( θ ∗ ) and log p θ ∗ (ˆ x 0 ) , i.e, log p θ ∗ (ˆ x 0 ) − F ( n ) score ( θ ∗ ) , by minimizing the following objective with respect to ˆ β n :
[Equation]
[Equation]
where
[Equation]
Proof.
Knowing that ˆ x 0 = x 0 , ˆ α n = α t , x t = α t x 0 + (cid:112) 1 − α 2 t (cid:15) n and p θ ∗ (ˆ x n − 1 | ˆ x n = x t ) = q ˆ β (ˆ x n − 1 | x t ; (cid:15) n ) , we apply a re-parameterization technique as follows:
[Equation]
[when x 0 is known, x t can use any (cid:15) n ∼ N [Equation] [Equation] I ) ]
[Equation]
where the (cid:15) n is sampled only once for x t , thus we move the coefﬁcient of (cid:15) n in Eq.
(34) to the variance to derive the form of q ˆ β (ˆ x n − 1 | x 0 ; α t ) :
q ˆ
[Equation]
Therefore
[Equation]
we can express the gap between log p θ ∗ (ˆ x 0 ) and F ( n ) score ( θ ∗ ) in the following form:
[Equation]
Next
[Equation]
we evaluate the above KL divergence term. By deﬁnition
[Equation]
we have
[Equation]
n = [Equation] [Equation] 64) therefore
[Equation]
As we use a scheduling network φ to estimate ˆ β n from (ˆ α n +1 , ˆ β n +1 ) as deﬁned in Eq. (13), we obtain the ﬁnal step loss for learning φ :
[Equation]
This proposed objective for training the scheduling network can be interpreted as to better model the data distribution (i.e., maximizing log p
θ (ˆ x 0 ) )
by correcting the gradient scale ˆ β n for the next reverse step (from ˆ x n to ˆ x n − 1 ) given the gradient vector (cid:15) θ ∗ estimated by the score network θ ∗ .
B E XPERIMENTAL DETAILS B. C ONVENTIONAL GRID SEARCH ALGORITHM FOR DDPM S
We reproduced the grid search algorithm in (Chen et al., 2020), in which a 6-step noise schedule was searched.
In our paper, we generalized the grid search algorithm by similarly sweeping the N -step noise schedule over the following possibilities with a bin width M = 9 :
[Equation]
[Equation]
where ⊗ denotes the cartesian product applied on two sets.
LS-MSE was used as a metric to select the solution during the search.
When N = 6 , we resemble the GS algorithm in (Chen et al., 2020).
Note that above searching method normally does not scale up to N > 6 steps for its exponential computational cost O (9 N ) .
B. H YPERPARAMETER SETTING IN BDDM S
Algorithm 2 took a skip factor τ to control the stride for training the scheduling network.
The value of τ would affect the coverage of step sizes when training the scheduling network, hence affecting the predicted number of steps N for inference – the higher τ is, the shorter the predicted inference schedule tends to be.
We set τ = 66 for training the BDDM vocoders in this paper.
For initializing Algorithm 3 for noise scheduling, we could take as few as 1 training sample for validation, perform a grid search on the hyperparameters { (ˆ α N = 0 .
1 α T i, ˆ β N = 0 .
1 j ) } for i, j = 1 , ..., 9 , i.e., 81 possibilities in total, and use the PESQ measure as the selection metric.
Then, the predicted noise schedule corresponding to the maximum PESQ was stored and applied to the online inference afterward, as shown in Algorithm 4.
Note that this searching has a complexity of only
O ( M 2 ) (e.g., M = 9 in this case), which is much more efﬁcient than O ( M N ) in the conventional grid search algorithm in (Chen et al., 2020), as discussed in Section B.1.
B. I MPLEMENTATION DETAILS
Our proposed BDDMs and the baseline methods were all implemented with the Pytorch library.
The score networks for the LJ and VCTK speech datasets were trained from scratch on a single NVIDIA
Tesla P40 GPU with batch size 32 for about 1M steps, which took about 3 days.
For the model architecture, we used the same architecture as in DiffWave (Kong et al., 2021) for the score network with 128 residual channels; we adopted a lightweight GALR network (Lam et al., 2021) for the scheduling network.
GALR was originally proposed for speech enhancement, so we considered it well suited for predicting the noise scales.
For the conﬁguration of the GALR network, we used a window length of 8 samples for encoding, a segment size of 64 for segmentation and only two GALR blocks of 128 hidden dimensions, and other settings were inherited from (Lam et al., 2021).
To make the scheduling network output with a proper range and dimension, we applied a sigmoid function to the last block’s output of the GALR network.
Then the result was averaged over the segments and the feature dimensions to obtain the predicted ratio: σ φ ( x ) =
AvgPool2D ( σ ( GALR ( x ))) , where GALR ( · ) denotes the GALR network, AvgPool2D ( · ) denotes the average pooling operation applied to the segments and the feature dimensions, and σ ( x ) :
= 1 / (1 + e − x ) .
The same network architecture was used for the NE approach for estimating [Table] α 2 t and was shown better than the ConvTASNet used in the original paper (San-Roman et al., 2021).
It is also notable that the computational cost of a scheduling network is indeed fractional compared to the cost of a score network, as predicting a noise scalar variable is intrinsically a relatively much easier task.
Our GALR-based scheduling network, while being able to produce stable and reliable results, was about 3.6 times faster than the score network.
The training of scheduling networks for
BDDMs took only 10k steps to converge, which consumed no more than an hour on a single GPU.
Regarding the image generation task, to demonstrate the generalizability of our method, we directly adopted a score network pre-trained on the CIFAR-10 dataset implemented by a third-party opensource repository.
Regarding the scheduling network, to demonstrate that it does not have to use specialized architecture, we replaced GALR by the VGG11 (Simonyan & Zisserman, 2014), which was also used by as a noise estimator in (San-Roman et al., 2021).
The output dimension (number of classes) of VGG11 was set to 1 .
Similar to the setting for GALR in speech synthesis, we added a sigmoid activation to the last layer to ensure a [0 , 1] output. Similar to the training in speech domain, we trained the VGG11-based scheduling networks while freezing the score networks for 10k steps, which normally can be ﬁnished in about two hours. Our code for the speech vocoding and the image generation experiments will be uploaded to Github after the ﬁnal decision of ICLR is released. B. C ROWD - SOURCED SUBJECTIVE EVALUATION
All our Mean Opinion Score (MOS) tests were crowd-sourced.
We refer to the MOS scores in (Protasio Ribeiro et al., 2011), and the scoring criteria have been included in Table 3 for completeness.
The samples were presented and rated one at a time by the testers.
C A DDITIONAL EXPERIMENTS
A demonstration page at https://bilateral-denoising-diffusion-model.
github.io shows some samples generated by BDDMs trained on LJ speech and VCTK datasets.
C. M ULTI - SPEAKER SPEECH SYNTHESIS
In addition to the single-speaker speech synthesis, we evaluated BDDMs on the multi-speaker speech synthesis benchmark VCTK (Yamagishi et al., 2019).
VCTK consists of utterances sampled at 48 KHz by 108 native English speakers with various accents.
We split the VCTK dataset for training and testing: 100 speakers were used for training the multi-speaker model and 8 speakers for testing.
We trained on a 44257-utterance subset (40 hours) and evaluated on a held-out 100utterance subset.
For the score network, we used the Wavegrad architecture (Chen et al., 2020) so as to examine whether the superiority of BDDMs remains in a different dataset and with a different score network architecture.
Results are presented in Table 4.
For this multi-speaker VCTK dataset, we obtained consistent observations with that for the single-speaker LJ dataset presented in the main paper.
Again, the proposed BDDM with only 16 or 21 steps outperformed the DDPM with 1,000 steps.
To the best of our knowledge, ours was the ﬁrst work that reported this degree of superior.
When reducing to 8 steps, BDDM obtained performance on par with (except for a worse PESQ)
the costly gridsearched 8 steps (which were unscalable to more steps) in DDPM.
For NE, we could again observe a degradation from its 16 steps to 21 steps, indicating the instability of NE for the VCTK dataset [Table]
likewise. In contrast, BDDM gave continuously improved performance while increasing the step number.
C. C OMPARING DIFFERENT REVERSE PROCESSES FOR BDDM S
This section demonstrates that BDDMs do not restrict the sampling procedure to a specialized reverse process in Algorithm 4.
In particular, we evaluated different reverse processes, including that of DDPMs as shown in Eq. (4) and DDIMs (Song et al., 2021a), for BDDMs and compared the objective scores on the generated samples.
DDIMs (Song et al., 2021a) formulate a non-Markovian generative process that accelerates the inference while keeping the same training procedure as DDPMs.
The original generative process in Eq.
(4) in DDPMs is modiﬁed into
[Equation]
where γ is a sub-sequence of length N of [1 , ..., T ] with γ N = T , and ¯ γ := { 1 , ..., T } \ γ is deﬁned as its complement; Therefore, only part of the models are used in the sampling process.
To achieve the above, DDIMs deﬁned a prediction function f ( t ) θ ( x t ) that depends on (cid:15) θ to predict the observation x 0 given x t directly: [Equation]
[Equation]
By leveraging this prediction function, the conditionals in Eq.
(74) are formulated as
[Equation]
where the detailed derivation of σ t and ς can be referred to (Song et al., 2021a).
In the original
DDIMs, the accelerated reverse process produces samples over the subsequence of β indexed by γ : ˆ β = { β n | n ∈ γ } .
In BDDMs, to apply the DDIM reverse process, we use the ˆ β predicted by the scheduling network in place of a subsequence of the training schedule β . [Table] [Table]
Finally.
the objective scores are given in Table 5.
Note that the subjective evaluation (MOS) is omitted here since the other assessments above have shown that the MOS scores are highly correlated with the objective measures, including STOI and PESQ.
They indicate that applying BDDMs to either DDPM or DDIM reverse process leads to comparable and competitive results.
Meanwhile, the results show some subtle differences: BDDMs over a DDPM reverse process gave slightly better samples in terms of signal error and consistency metrics (i.e., LS-MSE and MCD), while BDDM over a DDIM reverse process tended to generate better samples in terms of intelligibility and perceptual metrics (i.e., STOI and PESQ). C. U NCONDITIONAL IMAGE GENERATION
For the unconditional image generation task, we evaluated the proposed BDDMs on the benchmark CIFAR-10 (32 × 32) dataset. The score functions, including those initially proposed in DDPMs (Ho et al., 2020) or DDIMs (Song et al., 2021a) and those pre-trained in the above third-party implementations, are all conditioned on a discrete step-index. We estimated the noise schedule ˆ β in continuous space using the VGG11 scheduling network and then mapped it to discrete time schedule using the approximation method in (Kong & Ping, 2021).
Table 6 shows the performances of different sampling methods for DDPMs in CIFAR-10. By setting the maximum number of sampling steps ( N ) for noise scheduling, we can fairly compare the improvements achieved by BDDMs against related methods in the literature in terms of FID. Remarkably, BDDMs with 100 sampling steps not only surpassed the 1000-step DDPM baseline, but also produced the SOTA FID performance amongst all generative models using less than or equal to 100 sampling steps. [Equation]