P RIOR G RAD : I MPROVING C ONDITIONAL D ENOISING D IFFUSION M ODELS WITH D ATA -D EPENDENT A DAPTIVE P RIOR
A BSTRACT
Denoising diffusion probabilistic models have been recently proposed to generate high-quality samples by estimating the gradient of the data density.
The framework deﬁnes the prior noise as a standard Gaussian distribution, whereas the corresponding data distribution may be more complicated than the standard Gaussian distribution, which potentially introduces inefﬁciency in denoising the prior noise into the data sample because of the discrepancy between the data and the prior.
In this paper, we propose PriorGrad to improve the efﬁciency of the conditional diffusion model for speech synthesis (for example, a vocoder using a mel-spectrogram as the condition) by applying an adaptive prior derived from the data statistics based on the conditional information.
We formulate the training and sampling procedures of PriorGrad and demonstrate the advantages of an adaptive prior through a theoretical analysis.
Focusing on the speech synthesis domain, we consider the recently proposed diffusion-based speech generative models based on both the spectral and time domains and show that PriorGrad achieves faster convergence and inference with superior performance, leading to an improved perceptual quality and robustness to a smaller network capacity, and thereby demonstrating the efﬁciency of a data-dependent adaptive prior.
I NTRODUCTION
Deep generative models have been achieving rapid progress, by which deep neural networks approximate the data distribution and synthesize realistic samples from the model.
There is a wide range of this type of approach, ranging from autoregressive models (Oord et al., 2016a;b), generative adversarial networks (Goodfellow et al., 2014; Brock et al., 2019), variational autoencoders (Kingma & Welling, 2013; Vahdat & Kautz, 2020), and normalizing ﬂows (Rezende & Mohamed, 2015; Kingma & Dhariwal, 2018).
Denoising diffusion probabilistic models (DDPMs) (Ho et al., 2020) and score matching (SM) (Song & Ermon, 2019) are recently proposed categories that can be used to synthesize high-ﬁdelity samples with competitive or sometimes better quality than previous state-of-the-art approaches.
Consequently, there have been a variety of applications based on DDPM or SM (Saharia et al., 2021; Kawar et al., 2021).
Speech synthesis is one of the most successful applications, where the diffusion model can synthesize spectral or time-domain audio conditioned on text or spectral information, respectively, achieving a competitive quality but faster sampling (Chen et al., 2021;
Kong et al., 2021; Jeong et al., 2021; Lee & Han, 2021) than autoregressive models (Oord et al., 2016b; Kalchbrenner et al., 2018).
However, although the diffusion-based speech synthesis models have achieved high-quality speech audio generation, they exhibit potential inefﬁciency, which may necessitate advanced strategies.
For example, the model suffers from a signiﬁcantly slow convergence during training, and a prohibitively large training computation time is required to learn the approximate reverse diffusion process.
We investigate the diffuion-based models and observe the discrepancy between the real data distribution and the choice of the prior.
Existing diffusion-based models deﬁne a standard Gaussian as the prior distribution and design a non-parametric diffusion process that procedurally destroys the signal into the prior noise.
The deep neural network is trained to approximate the reverse diffusion process by estimating the gradient of the data density.
Although applying the standard Gaussian as the prior is simple without any assumptions on the target data, it also introduces inefﬁciency.
For example, in time-domain waveform data, the signal has extremely high variability between different segments such as voiced and unvoiced parts.
Jointly modeling the voiced and unvoiced segments with the same standard Gaussian prior may be difﬁcult for the model to cover all modes of the data, leading to training inefﬁciencies and potentially spurious diffusion trajectories.
Given the previous reasoning, we assessed the following question: For a conditional diffusion-based model, can we formulate a more informative prior without incorporating additional computational or parameter complexity?
To investigate this, we propose a simple yet effective method, called
PriorGrad, that uses adaptive noise by directly computing the mean and variance for the forward diffusion process prior, based on the conditional information.
Speciﬁcally, using a conditional speech synthesis model, we propose structuring the prior distribution based on the conditional data, such as a mel-spectrogram for the vocoder (Chen et al., 2021; Kong et al., 2021) and a phoneme for the acoustic model (Jeong et al., 2021).
By computing the statistics from the conditional data at the frame level (vocoder) or phoneme-level (acoustic model) granularity and mapping them as the mean and variance of the Gaussian prior, we can structure the noise that is similar to the target data distribution at an instance level, easing the burden of learning the reverse diffusion process.
We implemented PriorGrad based on the recently proposed diffusion-based speech generative models (Kong et al., 2021; Chen et al., 2021; Jeong et al., 2021), and conducted experiments on the
LJSpeech (Ito & Johnson, 2017) dataset.
The experimental results demonstrate the beneﬁts of PriorGrad, such as a signiﬁcantly faster model convergence during training, improved perceptual quality, and an improved tolerance to a reduction in network capacity.
Our contributions are as follows:
• To the best of our knowledge, our study is one of the ﬁrst to systematically investigate the effect of using a non-standard Gaussian distribution as the forward diffusion process prior to the conditional generative model.
• Compared to previous non-parametric forward diffusion without any assumption, we show that the model performance is signiﬁcantly improved with faster convergence by leveraging the conditional information as the adaptive prior.
• We provide a comprehensive empirical study and analysis of the diffusion model behavior in speech generative models, in both the spectral and waveform domains, and demonstrate the effectiveness of the method, such as a signiﬁcantly accelerated inference and improved quality.
B ACKGROUND
In this section, we describe the basic formulation of the diffusion-based model and provide related studies, along with a description of our contribution with PriorGrad.
Basic formulation Denoising diffusion probabilistic models (DDPM)
(Ho et al., 2020) are recently proposed deep generative models deﬁned by two Markov chains: forward and reverse processes.
The forward process procedurally destroys the data x 0 into a standard Gaussian x T , as follows:
[Equation]
where q ( x t | x t − 1 ) represents the transition probability at the t -th step using a user-deﬁned noise schedule β t ∈ { β 1 , ..., β T } .
Thus, the noisy distribution of x t is the closed form of q ( x t | x 0 ) =
N ( x t ; √ ¯ α t x 0 , (1 − ¯ α t ) I ) , where α t := 1 − β t , ¯ α t := (cid:81) ts =1 α s .
q ( x T | x 0 ) converges in distribution to the standard Gaussian N ( x T ; 0 , I ) if ¯ α T is small enough based on a carefully designed noise schedule.
The reverse process that procedurally transforms the prior noise into data is deﬁned as follows:
[Equation]
where p ( x T ) =
N ( x T ; 0 , I ) and p θ ( x t − 1 | x t ) corresponds to the reverse of the forward transition probability, parameterized using a deep neural network.
We can deﬁne the evidence lower bound (ELBO) loss as the training objective of the reverse process:
[Equation]
As shown in Ho et al.
(2020), q ( x t − 1 | x t , x 0 ) can be represented by Bayes rule as follows:
[Equation]
By ﬁxing p ( x T ) as a standard Gaussian, KL( q ( x T | x 0 )
|| p ( x T )) becomes constant and is not parameterized.
The original framework in Ho et al.
(2020) ﬁxed Σ θ ( x t , t ) as a constant ˜ β t I and set the standard Gaussian noise (cid:15) as the optimization target instead of ˜ µ t by reparameterizing x 0 =
[Equation]
¯
[Equation]
t ( x t −
[Equation]
¯ [Equation] t (cid:15) ) from q ( x t | x 0 ) to minimize the second and third terms in equation 3.
Based on this setup, in Ho et al.
, the authors further demonstrated that we can drop the weighting factor of each term and use a simpliﬁed training objective that provides a higher sample quality:
[Equation]
Related work Since the introduction of the DDPM, there have been a variety of further studies (Nichol & Dhariwal, 2021; Song et al., 2020), applications (Chen et al., 2021; Jeong et al., 2021;
Kong et al., 2021; Saharia et al., 2021), and a symbiosis of diffusion (Ho et al., 2020; Sohl-Dickstein et al., 2015) and score-based models (Song & Ermon, 2019; 2020) as a uniﬁed view with stochastic differential equations (SDEs) (Song et al., 2021).
From an application perspective, several conditional generative models have been proposed.
Waveform synthesis models (Chen et al., 2021; Kong et al., 2021) are one of the major applications in which the diffusion model is trained to generate time-domain speech audio from the prior noise, conditioned on a mel-spectrogram.
A diffusionbased decoder has also been applied to text-to-spectrogram generation models (Jeong et al., 2021;
Popov et al., 2021).
PriorGrad focuses on improving the efﬁciency of training such methods from the perspective of a conditional generative model.
We investigate the potential inefﬁciency of the current methods which require unfeasibly large computing resources to train and generate high-quality samples.
Studies on formulating an informative prior distribution for deep generative model are not new, and there has been a variety of studies investigating a better prior, ranging from hand-crafted (Nalisnick & Smyth, 2017; Tomczak & Welling, 2018), autoregressive (Chen et al., 2017), vector quantization (Razavi et al., 2019), prior encoder (Rezende & Mohamed, 2015), and data-dependent approaches similar to ours (Li et al., 2019).
We tackle the problem of training inefﬁciency of diffusion-based models by crafting better priors in a data-dependent manner, where our method can provide a better trajectory and can reduce spurious modes, enabling more efﬁcient training.
Nachmani et al.
[Figure] used Gamma distribution as the diffusion prior.
Note that there has also been a concurrent study conducted on leveraging the prior distribution on the acoustic model, Grad-TTS (Popov et al., 2021), in which the effectiveness of using the mean-shifted Gaussian as a prior with the identity variance was investigated.
Unlike the method in Popov et al.
(2021), which enforces the encoder output to match the target mel-spectrogram by using an additional encoder loss, our approach augments the forward diffusion prior directly through data and the encoder has no restriction on latent feature representations.
In Popov et al.
(2021), the forward diffusion prior is jointly trained and may induce additional overhead on convergence as the prior changes throughout the training, whereas our method provides guaranteed convergence through the ﬁxed informative prior.
M ETHOD
We investigate the following intuitive argument: When we structure the informative prior noise closer to the data distribution, can we improve the efﬁciency of the diffusion model?
In this section, we present a general formulation of PriorGrad, describe training and sampling algorithms, and provide beneﬁts of PriorGrad through a theoretical analysis.
PriorGrad offers a generalized approach for the diffusion-based model with the non-standard Gaussian as the prior and can be applied to a variety of applications.
3. G ENERAL F ORMULATION
[Figure]
In this section, we provide a general formulation of the method regarding using the non-standard Gaussian N ( µ , Σ ) as the forward diffusion prior.
PriorGrad leverages the conditional data to directly compute instancelevel approximate priors in an adaptive manner, and provides the approximate prior as the forward diffusion target for both training and inference.
Figure 1 and 2 presents a visual high-level overview.
We take the same parameterization of µ θ and σ θ as in the original DDPM
(Ho et al., 2020) from Section 2, as follows:
[Equation]
Assuming that we have access to an optimal Gaussian N ( µ , Σ ) as the forward diffusion prior distribution, we have the following modiﬁed
ELBO:
Proposition 1 Let (cid:15) ∼ N ( 0 , Σ ) and x 0 ∼ q data .
Then, under the parameterization in Eq.
equation 8, the ELBO loss will be
[Equation]
for some constant C , where (cid:107) x
(cid:107) 2 Σ − 1 = x T Σ − 1 x , γ t =
[Equation]
for t >
[Equation]
and γ [Equation] = [Equation]
α [Equation] The Proposition 1 is an extension of the ELBO in Equation 6 with the non-standard Gaussian distribution.
Contrary to original DDPM, which used N ( 0 , I ) as the prior without any assumption on data, through Proposition 1, we train the model with N ( µ , Σ ) , whose mean and variance are extracted from the data, as the prior for the forward process.
See appendix A.1 for full derivation.
We also drop γ t as the simpliﬁed loss L = (cid:107) (cid:15) − (cid:15) θ ( x t , c, t ) (cid:107) 2 Σ − 1 for training, following the previous work.
Algorithms 1 and 2 describe the training and sampling procedures augmented by the datadependent prior ( µ , Σ ) .
Because computing the data-dependent prior is application-dependent, in
Section 4 and 5, we describe how to compute such prior based on the conditional data on the given task.
Algorithm repeat ( µ , Σ ) = data-dependent prior
Sample x 0 ∼ q data , (cid:15) ∼ N ( 0 , Σ )
Sample t ∼ U ( { 1 , · · · , T } ) x t = √ ¯ α t ( x 0 − µ ) + √ 1 − ¯ α t (cid:15)
L = (cid:107) (cid:15) − (cid:15) θ ( x t , c, t ) (cid:107) 2 Σ −
Update the model parameter θ with ∇ θ L until converged Algorithm
( µ , Σ )
= data-dependent prior
Sample x T ∼ N ( 0 , Σ ) for t =
T, T − 1 , · · · , 1 do
Sample z ∼ N (0 , Σ ) if t > 1 ; else z =x t − 1 = 1 √ α t ( x t − 1 − α t √ 1 − ¯
α t (cid:15) θ ( x t , c, t ))
+ σ t z end for return x 0 + µ
3. T HEORETICAL A NALYSIS
In this section, we describe the theoretical beneﬁts of PriorGrad.
First, we discuss about the simpliﬁed modeling with the following proposition:
Proposition 2 Let L ( µ , Σ , x 0 ; θ ) denote the − ELBO loss in Proposition 1.
Suppose that (cid:15) θ is a linear function.
Under the constraint that det( Σ ) = det( I ) , we have min θ L ( µ , Σ , x 0 ; θ ) ≤ min θ L ( 0 , I , x 0 ; θ ) .
The proposition shows that setting the prior whose covariance Σ aligns with the covariance of data x 0 leads to a smaller loss if we use a linear function approximation for (cid:15) θ .
This indicates that we can use a simple model to represent the mean of q ( x t − 1 | x t ) under the data-dependent prior, whereas we need to use a complex model to achieve the same precision under a prior with isotropic covariance.
The condition det( Σ ) = det( I ) means that the two Gaussian priors have equal entropy, which is a condition for a fair comparison.
Second, we discuss the convergence rate.
The convergence rate of optimization depends on the condition number of the Hessian matrix of the loss function (denoted as H ) (Nesterov, 2003), that is, λ max ( H ) λ min ( H ) , where λ max and λ min are the maximal and minimal eigenvalues of H , respectively.
A smaller condition number leads to a faster convergence rate.
For L ( µ , Σ , x 0 ; θ ) , the Hessian is calculated as
[Equation]
Again, if we assume (cid:15) θ is a linear function, we have H ∝ I for L ( µ , Σ , x 0 ; θ ) and H ∝ Σ
+ I for
L ( 0 , I , x 0 ; θ ) .
It is clear that if we set the prior to be N ( µ , Σ ) , the condition number of H equals 1 , achieving the smallest value of the condition number.
Therefore, it can accelerate the convergence.
Readers may refer to the appendix A.2 for more details.
A PPLICATION TO V OCODER
In this section, we apply PriorGrad to a vocoder model, as visually described in Figure 3.
4. P RIOR G RAD FOR V OCODER
Our formulation of PriorGrad applied to a vocoder is based on DiffWave (Kong et al., 2021), where the model synthesizes time-domain waveform conditioned on a mel-spectrogram that contains a compact frequency feature representation of the data.
Unlike the previous method that used (cid:15) ∼
N ( 0 , I ) , PriorGrad network is trained to estimate the noise (cid:15) ∼ N ( 0 , Σ ) given the destroyed signal √ ¯ α t x 0 + √ 1 − ¯ α t (cid:15) .
Same as Kong et al.
(2021), the network is also conditioned on the discretized [Figure] index of the noise level √ ¯ α t with the diffusion-step embedding layers, and the mel-spectrogram c with the conditional projection layers.
Based on the mel-spectrogram condition, we propose leveraging a normalized frame-level energy of the mel-spectrogram for acquiring data-dependent prior, exploiting the fact that the spectral energy contains an exact correlation to the waveform variance 1 .
First, we compute the frame-level energy by applying roots of sum of exponential to c , where c is the mel-spectrogram from the training dataset.
We then normalize the frame-level energy to a range of (0 , 1] to acquire the data-dependent diagonal variance Σ c .
In this way, we can use the frame-level energy as a proxy of the standard deviation for the waveform data we want to model.
This can be considered as a non-linear mapping between the mel-scale spectral energy and the standard deviation of the diagonal Gaussian.
We set N ( 0 , Σ ) as the forward diffusion prior for each training step by upsampling Σ c in the framelevel to Σ in the waveform-level using a hop length for the given vocoder.
We chose a zero-mean prior in this setup reﬂecting the fact that the waveform distribution is zero-mean.
In practice, we imposed the minimum standard deviation of the prior to 0 .
1 through clipping to ensure numerical stability during training 2 .
We have tried several alternative sources of conditional information to compute the prior, such as voiced/unvoiced labels and phoneme-level statistics, but they resulted in a worse performance.
We justify our choice of using frame-level energy in the appendix A.3.
4. E XPERIMENTAL S ETUP
We used LJSpeech (Ito & Johnson, 2017) dataset for all experiments, which is a commonly used open-source 24h speech dataset with 13,100 audio clips from a single female speaker.
We used an 80-band mel-spectrogram feature at the log scale from the 22,050Hz volume-normalized speech, with the 1024-point FFT, 80Hz and 7,600Hz low- and high-frequency cutoff, and a hop length of 256.
We used 13,000 clips as the training set, 5 clips as the validation set, and the remainingclips as the test set used for an objective and subjective audio quality evaluation.
We followed the publicly available implementation 3 , where it uses a 2.62M parameter model with an Adam optimizer (Kingma & Ba, 2014) and a learning rate of 2 × 10 − 4 for a total of 1M iterations.
Training for 1M iterations took approximately 7 days with a single NVIDIA A40 GPU.
We used the default diffusion steps with T = 50 and the linear beta schedule ranging from 1 × 10 − 4 to 5 × 10 − 2 for training and inference, which is the default setting of DiffWave BASE model used in Kong et al. We also used the fast T infer = 6 inference noise schedule from DiffWave BASE without modiﬁcation.
4. E XPERIMENTAL R ESULTS
We conducted experiments to verify whether our method can learn the reverse diffusion process faster, by comparing to the baseline DiffWave model with (cid:15) ∼ N ( 0 , I ) .
First, we present the model convergence result by using a spectral domain loss on the test set to show the fast training of PriorGrad.
Second, we provide both objective and subjective audio quality results, where PriorGrad offered the improved quality.
Finally, we measure a tolerance to the reduction of the network capacity, where PriorGrad exhibits an improved parameter efﬁciency.
We leave the description of the objective metrics we collected to the appendix A.4.
[Figure]
[Table] [Table]
[Table]
Model convergence We used a widely adopted spectral metric with log-mel spectrogram mean absolute error (LS-MAE) as the proxy of the convergence for the waveform synthesis model.
We can see from Figure 4 that PriorGrad exhibited a signiﬁcantly faster spectral convergence compared to the baseline.
In the auditory test, we observed that PriorGrad readily removed the background white noise early in training, whereas the baseline needed to learn the entire reverse diffusion process starting from (cid:15) ∼ N ( 0 , I ) which contains little information.
Table 1 shows the 5-scale subjective mean opinion score (MOS) test of PriorGrad from Amazon Mechanical Turk.
We observed that PriorGrad outperformed the baseline DiffWave model with 2 times fewer training iterations.
PriorGrad also outperformed the baseline on objective speech and distance metrics as shown in Table 3, such as Multi-resolution STFT error (MR-STFT)
(Yamamoto et al., 2020), Mel cepstral distortion (MCD) (Kubichek, 1993) and F 0 root mean square error (F 0 RMSE) and a debiased Sinkhorn divergence (Feydy et al., 2019) between the prior and the ground-truth S ( x T , x 0 ) , or the generated samples and the ground-truth S (˜ x 0 , x 0 ) , consistent with the subjective quality results.
Refer to A.7 for comparison to other state-of-the-art approaches with varying number of inference steps.
Parameter efﬁciency We measured a tolerance to a reduced diffusion network capacity by setting the width of the dilated convolutional layers by half, leading to a smaller model with 1.23M parameters.
We trained the small DiffWave and PriorGrad with the same training conﬁgurations for 1M steps.
With T infer = 50 , Table 2 conﬁrmed that the performance degradation of PriorGrad is reduced compared to the baseline DiffWave model.
The small PriorGrad was able to maintain the quality close to the larger baseline model, suggesting that having access to the informative prior distribution can improve parameter efﬁciency of the diffusion-based generative model.
A PPLICATION TO A COUSTIC M ODEL
In this section, we present the application of PriorGrad to the acoustic models, as visually described in Figure 5.
forward process reverse process ( , ) data-dependent prior condition
In being comparatively modern “ ”
5. P RIOR G RAD FOR A COUSTIC M ODEL
The acoustic models generate a mel-spectrogram given a sequence of phonemes with the encoderdecoder architecture.
In other words, the acoustic model is analogous to the text-conditional image generation task (Reed et al., 2016; Ramesh et al., 2021), where the target image is 2D melspectrogram.
We implement the PriorGrad acoustic model by using the appraoch in Ren et al.
(2020) as a feed-forward phoneme encoder, and using a diffusion-based decoder with dilated convolutional layers based on Kong et al.
Note that a similar decoder architecture was used in
Jeong et al.
To build the adaptive prior, we compute the phoneme-level statistics of the 80-band mel-spectrogram frames by aggregating the frames that correspond to the same phoneme from the training data, where phoneme-to-frame alignment is provided by the Montreal forced alignment (MFA) toolkit (McAuliffe et al., 2017).
Speciﬁcally, for each phoneme, we acquire the 80-dimensional diagonal mean and variance from the aggregation of all occurrences in the training dataset.
Then, we construct the dictionary of N ( µ , Σ ) per phoneme.
To use these statistics for the forward diffusion prior, we need to upsample the phoneme-level prior sequence to the frame-level with matching durations.
This can be done by jointly upsampling this phoneme-level prior as same as the phoneme encoder output based on the duration predictor (Ren et al., 2019) module.
Following Algorithm 1, we train the diffusion decoder network with the mean-shifted noisy melspectrogram x t =
√ ¯ α t ( x 0 − µ ) + √ 1 − ¯ α t (cid:15) as the input.
The network estimates the injected noise (cid:15) ∼ N ( 0 , Σ ) as the target.
The network is additionally conditioned on the aligned phoneme encoder output.
The encoder output is added as a bias term of each dilated convolution layers of the gated residual block with the layer-wise 1 × 1 convolution.
5. E XPERIMENTAL S ETUP
For the acoustic model experiments, we implemented the Transformer phoneme encoder architecture identical to FastSpeech 2 (Ren et al., 2020) and the convolutional diffusion decoder architecture based on Jeong et al.
We adopted the open-source implementation of the DiffWave architecture (Kong et al., 2021) with 12 convolutional layers, same as Jeong et al.
We used the beta schedule with T = 400 for training and used the fast reverse sampling schedule with T infer =with a grid search method (Chen et al., 2021).
We followed the same training and inference protocols of Ren et al.
and used a pre-trained Parallel WaveGAN (PWG) (Yamamoto et al., 2020) vocoder for our controlled experiments.
For state-of-the-art comparison, we used HiFi-GAN (Kong et al., 2020) vocoder to match the previous work.
We conducted a comparative study of PriorGrad acoustic model with a different diffusion decoder network capacity, i.e., a small model with 3.5M parameters (128 residual channels), and a large model with 10M parameters (256 residual channels).
Training for 300K iterations took approximately 2 days on a single NVIDIA P100 GPU.
We leave the additional details in the appendix A.5.
5. E XPERIMENTAL R ESULTS
Model convergence We can see from Table 4 that applying N ( µ , Σ ) with PriorGrad signiﬁcantly accelerated the training convergence and exhibited higher-quality speech for different decoder capacities and training iterations.
A high-capacity (10M) PriorGrad was able to generate high-quality mel-spectrogram with as few as 60K training iteration, whereas the baseline model was not able to match the performance even after 300K iterations.
This is because the training iterations were insufﬁcient for the baseline to learn the entire diffusion trajectory with the standard Gaussian as the prior, leading to lower performance.
The small (3.5M) baseline model exhibited an improved
[Table]
quality compared to the large baseline for the same training iterations, which further suggests that the convergence of the large baseline model is slow.
By contrast, by easing the burden of learning the diffusion process by having access to the phoneme-level informative forward prior, PriorGrad achieved high-quality samples signiﬁcantly earlier in the training.
Refer to Appendix A.6 for additional experimental results including an alternative model with the jointly trainable diffusion prior estimation, and Appendix A.7 for an expanded comparison to previous work, where PriorGrad achieves a new state-of-the-art acoustic model.
Parameter efﬁciency Similar to the vocoder experiment, we assessed whether we can retain the high performance with a reduced diffusion decoder capacity from 10M to 3.5M parameters.
The small PriorGrad also consistently outperformed the baseline model and achieved an almost identical perceptual quality to the high-capacity PriorGrad.
The small PriorGrad outperformed both the small and large baseline models, which conﬁrms that PriorGrad offers an improved parameter efﬁciency and tolerance to the reduced network capacity.
This suggests that PriorGrad provides a way to build practical and efﬁcient diffusion-based generative models.
This holds a special importance for speech synthesis models, where the efﬁciency plays a key role in model deployment in realistic environments.
D ISCUSSION AND C ONCLUSION
In this study, we investigated the potential inefﬁciency of recently proposed diffusion-based model as a conditional generative model within the speech synthesis domain.
Our method, PriorGrad, directly leverages the rich conditional information and provides an instance-level non-standard adaptive Gaussian as the prior of the forward diffusion process.
Through extensive experiments with the recently proposed diffusion-based speech generative models, we showed that PriorGrad achieves a faster model convergence, better denoising of the white background noise, an improved perceptual quality, and parameter efﬁciency.
This enables the diffusion-based generative models to be signiﬁcantly more practical, and real-world applications favor lightweight and efﬁcient models for deployments.
Whereas our focus is on improving the diffusion-based speech generative models, PriorGrad has a potential to expand the application beyond the speech synthesis domain.
In the image domain, for example, PriorGrad can potentially be applied to image super-resolution tasks (Saharia et al., 2021) by using the patch-wise mean and variance of the low-resolution image to dynamically control the forward/reverse process.
The method could potentially be used for depth map conditional image synthesis (Esser et al., 2020), where the depth map can be mapped as the prior variance using the fact that the monocular depth corresponds to the camera focus and visual ﬁdelity between the foreground and background.
We leave these potential avenue of PriorGrad for future work.
PriorGrad also has limitations that require further investigation.
Although PriorGrad offers a variety of practical beneﬁts as presented, it may also require a well-thought out task-speciﬁc design to compute the data-dependent statistics (or its proxy), which may be unsuitable depending on the granularity of the conditional information.
Building an advanced approach to enable a more generalized realization of PriorGrad will be an interesting area of future research. A CKNOWLEDGEMENT
This work was supported by the BK21 FOUR program of the Education and Research Program for Future ICT Pioneers, Seoul National University in 2021, Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
[NO.2021-0-01343, Artiﬁcial Intelligence Graduate School Program (Seoul National University)], and the MSIT (Ministry of Science, ICT), Korea, under the High-Potential Indivisuals Global Training Program (2020-0-01649) supervised by the IITP (Institude for Information & Communications
Technology Planning & Evaluation) and Microsoft Research Asia.
A A PPENDIX A. D ERIVATION OF THE ELBO L OSS
Proposition 1 Let (cid:15) ∼ N ( 0 , Σ ) and x 0 ∼ q data .
Then, under the parameterization in Eq.
equation 8, the ELBO loss will be
[Equation]
for some constant C , where (cid:107) x
(cid:107) 2 Σ − 1 = x T Σ − 1 x , γ t =
[Equation]
2 t 2 σ 2 t α t (1 − ¯ α t ) for t > 1 , and γ {+1 = 
+}
α 1 .
Proof
According to Algorithm 1, the input x 0 is normalized by subtracting its mean µ .
In the following, we denote ˜ x 0 = x 0 − µ and x t = √ ¯ α t ˜ x 0 + √ 1 − ¯ α t (cid:15) .
According to Equation (3), the ELBO loss is
[Equation]
(cid:17) . Let (cid:15) i ’s i.i.d.
∼ N ( 0 , Σ ) .
Similarly to the Equation (1), we have q ( x t | ˜ x 0 )
=
N ( x t ; √ ¯ α t ˜ x 0 , (1 − ¯ α t ) Σ ) .
By Bayes rule and Markov chain property,
[Equation]
where (cid:107) x (cid:107) 2 Σ − 1 = x T Σ − 1 x .
Therefore,
[Equation]
Then
[Equation]
we calculate each term of the ELBO expansion.
The ﬁrst term is
[Equation]
The second term is
[Equation]
Finally [Equation] we have
[Equation]
Combining Equation (10), (11) and (12) together, we can get the result in the Proposition.
A. T HEORETICAL B ENEFITS OF P RIOR G RAD
Proposition 2 Let L ( µ , Σ , x 0 ; θ ) denote the − ELBO loss in Proposition 1.
Suppose that (cid:15) θ is a linear function.
Under the constraint that det( Σ ) = det( I ) , we have min θ L ( µ , Σ , x 0 ; θ ) ≤ min θ L ( 0 , I , x 0 ; θ ) .
Proof : We use L ( µ , Σ , x 0 ; θ ) to denote -ELBO.
According to Equation (10), (11) and (12), we have
[Equation]
where ˜ x 0 = x 0 − µ .
We assume that (cid:15) θ ( · ) is a scalar function with parameter freedom d , i.e., (cid:15) θ ( √ ¯ α t ˜ x 0 + √ 1 − ¯ α t (cid:15) , t )
= θ ( √ ¯ α t ˜ x 0 + √ 1 − ¯ α t (cid:15) ) , where θ ∈ R d × d with constraint ˜ θ =
Q θ = diag (˜ θ 1 , · · · , ˜ θ d ) and Q is the eigenmatrix for Σ − 1 , i.e., Σ − 1 = Q T ˜ Σ − 1 Q with ˜ Σ − 1 = diag
( 1 σ 1 , · · · , 1 σ d ) .
For the term (cid:80) Tt =
[Equation]
E x 0 , [Equation] (cid:107) [Equation]
−
[Equation]
√ [Equation] − ¯ [Equation]
) (cid:107) 2 Σ [Equation] 1 , we have
T
[Equation]
=
[Equation]
where the second equation is established by taking expectation of x 0 and (cid:15) [Equation]
The above equation will achieve its minimum at ˜ θ j
[Equation]
, ∀ j [Equation] and the minimum is
[Equation]
For L ( 0 , I , x 0 ; θ )
[Equation]
we have
[Equation]
Similarly, we can get the minimum of (cid:80)
Tt =1 γ t E x 0 , (cid:15) (cid:107) (cid:15) − θ ( √ ¯ α t x 0 + √ 1 − ¯ α t (cid:15) , t ) (cid:107) 2 with a diagonal θ is (cid:80) d
j =1 (cid:16)(cid:80) T t =1 γ t − ( (cid:80) t =1 γ t √ 1 − ¯ α t ) 2 (cid:80)
Tt =1 γ t (1 − ¯ α t +¯ α t σ j ) (cid:17) .
Under the condition that det( Σ ) = det( I ) , we have the minimum of (cid:80) dj
=
(cid:16)(cid:80)
Tt =
[Equation]
γ [Equation] − ( [Equation]
=
[Equation] √ [Equation] ¯ α t ) 2 (cid:80) Tt =1 γ t (1 − ¯ α t +¯ α t σ j ) (cid:17) (over all possibilities of ( σ 1 , · · · , σ d ) ) will be (cid:80)
[Equation] = [Equation] =
[Equation]
γ t (cid:17) (by solving the constrained minimization problem).
Thus, we get the result in the Proposition.
Remark:
From Equation (12), we have the second order derivative of L ( µ , Σ , x 0 ; θ ) with respect to θ j equals (cid:80)
Tt =1 γ t for all j = 1 , · · · , d .
Thus the condition number of the Hessian matrix equals 1 .
Similarly, the second order derivative of L ( 0 , I , x 0 ; θ ) with respect to θ j equals (cid:80) Tt =1 γ t (1 − ¯ α t + ¯ α t σ j ) .
Thus the Hessian matrix equals c 1 I + c 2 Σ with c 1 = (cid:80) Tt =1 γ t (1 − ¯ α t ) and c 2 = (cid:80) Tt =1 γ t ¯ α t , whose condition number is no less than 1 .
A. E XPLORATION ON THE C ONDITIONAL I NFORMATION S OURCE OF P RIOR G RAD V OCODER
In this section, we provide further discussion regarding the source of the conditional information for constructing PriorGrad vocoder and empirical justiﬁcation of selecting the frame-level spectral [Figure] energy as the prior.
Considering the waveform synthesis model as a vocoder, the main application of such model is a spectrogram inversion module of the text-to-speech pipeline.
That is, the vocoder model is usually combined with the acoustic model as a front-end, where the acoustic model generates the mel-spectrogram based on text input.
FastSpeech 2 (Ren et al., 2020) demonstrated a signiﬁcant improvement in the quality of the acoustic model leveraged by carefully designed additional speech-related features for ﬁne-grained supervision, such as voiced/unvoiced (V/UV) label obtained from F0 contours, or phoneme-to-frame alignment labels.
We explored V/UV or phoneme labels as alternative sources of information for constructing the prior.
Figure 6 shows scatter plots of the waveform audio clip from the test set corresponding to the standard deviation labels assigned by Σ c acquired from the training set.
Because the V/UV label is binary, we can only assign two different prior variances to the waveform distribution, which is an overly coarse assumption of the data distribution.
Phoneme-level prior can offer more ﬁne-grained approximate prior.
However, we found that the variance statistics acquired from the training set is misaligned with the unobserved waveform, where the actual variance can be signiﬁcantly different to the target prior, leading to an inconsistent result.
The frame-level spectral energy for PriorGrad exhibited the best alignment of the label and the unobserved waveform data and demonstrated consistent results in quality.
A. D ESCRIPTION OF OBJECTIVE METRICS FOR P RIOR G RAD VOCODER
In this section, we describe the objective metrics we collected to evaluate the performance of PriorGrad vocoder.
Log-mel spectrogram mean absolute error (LS-MAE)
This is a spectral regression error between the log-mel spectrogram computed between the synthesized waveform and the ground-truth.
We used the same STFT function for computing the mel-spectrogram as the conditional input of
DiffWave and PriorGrad.
Multi-resolution STFT error (MR-STFT)
This measures the spectral distance across multiple resolutions of STFT.
MR-STFT is widely adopted as a training objective of recent neural vocoders because using multiple resolution windows can capture the time-frequency distributions of the realistic speech signal.
We used the resolution hyperparameter proposed in Parallel WaveGAN (Yamamoto et al., 2020), which is implemented in an open-source library 4 .
Mel-cepstral distortion (MCD) MCD is a widely adopted speech metric (Kubichek, 1993) which measures the distance between the mel cepstra.
We used an open-source implementation with default parameters 5 .
F 0 root mean square error (F 0 RMSE)
This measures the accuracy (measured by RMSE) of the fundamental frequency (F 0 ) which is an approximate frequency of the (quasi-)periodic structure of 4 https: com
https://github
[Table]
voiced speech signals.
We used the Saw-tooth Waveform Inspired Pitch Estimation (SWIPE) (Camacho & Harris, 2008) with a hop size of 128 to obtain F 0 using an open-source implementation 6 .
Debiased Sinkhorn divergence This is a positive deﬁnite approximation of optimal transport ( Wasserstein ) distance between two data distributions (Feydy et al., 2019).
This metric can be used as a mathematically grounded assessment of the informative prior, which is evidenced by a recent study that the diffusion process coincides with the optimal transport map (Khrulkov & Oseledets, 2022).
We used the open-source library 7 with default parameters.
Using DiffWave and PriorGrad with T infer = 6 , We calculated the test set average of the Sinkhorn divergence using Monte Carlo estimate with 100 samples per test data point.
We denote S ( x T , x 0 ) as the distance between the prior and the real data, and S (˜ x 0 , x 0 ) as the distance between the generated samples and the real data.
A. A DDITIONAL D ETAILS OF P RIOR G RAD A COUSTIC M ODEL
In this section, we describe additional details of the experimental design of the PriorGrad acoustic model.
The feed-forward Transformer-based phoneme encoder has 11.5M parameters trained with the Adam optimizer with the learning rate schedule identically described in Ren et al.
The diffusion decoder is simultaneously trained with the same training conﬁgurations.
For a fair comparative study between different diffusion models, we searched for the best performing baseline model with N ( 0 , I ) ﬁrst, then applied PriorGrad to this baseline.
We applied T =for training the diffusion decoder with a linearly spaced beta schedule ranging from 1 × 10 − 4 to 5 × 10 − 2 .
We found that 1 × 10 − 4 to 2 × 10 − 2 used for waveform domain in Kong et al.
(2021) performed poorly for the baseline, where the model failed to capture the high-frequency details of the mel-spectrogram.
This indicates that the optimal training noise schedule can be different, depending on the data domain (Kong & Ping, 2021).
On the contrary, PriorGrad’s performance was similar under different choices of training noise schedules, suggesting that PriorGrad also features robustness regarding designing the noise schedules for model training.
We found that the fast reverse noise schedule with T infer = 6 described in Kong et al.
(2021) performed poorly for the baseline model with N ( 0 , I ) .
Thus, we applied grid search over the
T infer = 6 reverse schedule for each model, which is a similar approach to that in Chen et al.
This enabled a fair comparative study of each model conﬁguration when using the fast noise schedule for sampling.
We applied a ﬁne-grained grid search over every possible combination of the monotonically increasing betas based on the L1 loss between the model prediction and the target from the validation set.
We applied the following range for the grid search: { 1 , 2 , ..., 8 , 9 } × { 10 − 1 , 10 − 1 } for T infer = 2 , { 1 , 2 , ..., 8 , 9 } × { 10 − 4 , 10 − 3 , 10 − 2 , 10 − 2 , 10 − 1 , 10 − 1 } for T infer = 6 , and { [Table]
We found that setting relatively high values of beta at the last steps was important for the quality of the baseline model, whereas PriorGrad was signiﬁcantly more robust to the choice of the noise schedule.
Table 5 shows the optimal beta schedules for each model conﬁguration from the grid search method. [Table]
A. C OMPARISON TO TRAINABLE DIFFUSION PRIOR
In this section, we present additional results regarding the challenges of formulating a jointly trainable estimation of the diffusion prior.
Based on the improvements achieved by PriorGrad, it is natural to suppose that the jointly trainable diffusion prior distribution with additional parameterization might perform similar or better than the data-dependent prior.
In this section, we explored a feasibility of the diffusion prior with trainable parameters.
For acoustic model, we explored two setups: 1. replacing the ﬁxed data-dependent prior with the estimated N ( µ θ , Σ θ ) from the projection layer of the phoneme encoder, 2.
parameterizing N ( µ θ , Σ θ ) with an independently deﬁned convolutional layers with the data-dependent ( µ , Σ )
we applied to PriorGrad as an input to acquire a reﬁned prior.
For the vocoder model, instead of the spectral energy as the proxy of the waveform variance, we explored an independently deﬁned diffusion prior estimator by using the smaller-scale convolutional network with the mel-spectrogram as the input.
We found that all approaches failed to jointly estimate or reﬁne the informative prior distribution.
For the acoustic model, the estimated mean was signiﬁcantly noisier than our data-dependent one, and the estimated variance is collapsed to a single point which is uninformative.
These models performed signiﬁcantly worse than the baseline method with the ﬁxed N ( 0 , I ) , as measured by Table 6.
We visualize the data-dependent prior compared to the estimated version in Figure 7.
For the vocoder model, the diffusion prior estimator resulted in a divergence of the estimated mean and collapse of the estimated variance to zero, leading to the training failure.
We do not draw the conclusive claim that the jointly trainable diffusion prior is not possible.
However, the results suggest that the ﬁxed data-dependent diffusion prior with PriorGrad is an effective and easy-to-use method to improve the efﬁciency of the diffusion-based generative model without introducing additional complexities to the network.
[Table] [Table]
A. C OMPARISON TO STATE - OF - THE - ART
In this section, we present additional results of PriorGrad by comparing to recent state-of-the-art speech synthesis models and show that PriorGrad is competitive with or sometimes outperforms the previous models.
We provide a detailed analysis with a varying number of inference denoising steps ( T infer ), along with their speed measured by a real-time factor (RTF) on the NVIDIA A40 GPU, and the model capacity measured by the number of parameters.
Vocoder comparison Table 7 provides an expanded MOS result of the fully converged PriorGrad vocoder with the 1M training steps.
For the fast inference noise schedule, we used T infer =deﬁned as [0.0001, 0.001, 0.01, 0.05, 0.2, 0.5] from DiffWave BASE without modiﬁcation, and used
T infer = 12 deﬁned as [0.0001, 0.0005, 0.0008, 0.001, 0.005, 0.008, 0.01, 0.05, 0.08, 0.1, 0.2, 0.5].
For the full T infer = 50 , PriorGrad signiﬁcantly outperformed the baseline DiffWave.
Furthermore, PriorGrad achieved an even larger performance gap with a signiﬁcantly reduced network capacity, compared to the ﬂow-based vocoders such as WaveGlow (Prenger et al., 2019) and WaveFlow (Ping et al., 2020).
This suggests that PriorGrad is a competitive likelihood-based neural vocoder, and enables a step closer to the current state-of-the-art GAN-based model (HiFi-GAN, Kong et al.
The speed is close to real-time (RTF being close to 1), but it is noticeably slower than other approaches.
For the fast inference with T infer = 6 denoising steps, PriorGrad was able to match the performance to the baseline DiffWave model with T infer = 50 while still outperforming the ﬂow-based vocoders with comparable RTF.
This further showcase the efﬁciency of PriorGrad, where having access to the informative non-standard Gaussian prior accelerates the inference of the diffusion-based model.
Acoustic model comparison We additionally trained our baseline diffusion-based acoustic model and the enhanced model with PriorGrad which are compatible with the pretrained HiFi-GAN (Kong et al., 2020) vocoder.
This enabled a fair assessment of PriorGrad compared to recent state-of-theart acoustic models.
We trained our acoustic model for the full convergence with 1M training steps.
We found that the small PriorGrad with 3.5M decoder parameters performed almost identical to the high-capacity (10M) model.
Therefore, we chose the small decoder model as our ﬁnal choice for comparison.
Table 8 provides an expanded MOS result of PriorGrad acoustic model with varying T infer and comparison to the recent representative acoustic model from different categories: Feed-forward
(FastSpeech 2, Ren et al.
(2020)), Flow-based (Glow-TTS, Kim et al.
and the concurrent diffusion-based model (Grad-TTS, Popov et al. (2021)).
The results show that PriorGrad acoustic model is competitive to the state-of-the-art models.
PriorGrad with T infer = 12 provided an identical quality to the state-of-the-art Grad-TTS with T infer = 10 .
This is achieved with an approximately 1.75 times faster RTF and sets PriorGrad a new state-of-the-art acoustic model.
Armed with the informative non-standard Gaussian prior, PriorGrad is robust to the extremely small number of inference steps with T infer = 2 .
The fast PriorGrad matches the performance with the state-of-the-art ﬂow-based acoustic model, Glow-TTS, with fewer parameters.
The RTF becomes close to the feed-forward FastSpeech 2 as well as scoring higher MOS.
By contrast, the baseline acoustic model with N ( 0 , I ) suffered a signiﬁcant degradation for smaller T infer .
This result further highlights the efﬁciency of PriorGrad, where it is robust to the reduced number of denoising steps and enables a signiﬁcantly accelerated inference.
A. MOS EVALUATION DETAILS
We conducted a standard and widely adopted protocol for the MOS evaluation using Amazon Mechanical Turk.
We constructed 20 randomly selected clips from the test set, and applied a total ofevaluations for each model with the 5-scale scoring on audio naturalness.
To encompass the diverse set of listeners, a single listener can only evaluate the three randomly exposed samples.
Therefore, there are 150 unique listeners for each model.
For reliability, only skilled listeners with the evaluation approval rate higher than 98% and the number of previously approved evaluations higher than 100 are qualiﬁed to conduct the test.
The listeners were rewarded $0.2 for each evaluation.
Approximately $130 were spent as compensation for each set of the MOS test.
A. A UDIO SAMPLES DEMO PAGE
We provide a demo page of the audio samples used in this study: https://speechresearch. github.io/priorgrad/