I NFORMATION -T HEORETIC D IFFUSION
Anonymous authors
Paper under double-blind review
A BSTRACT
Denoising diffusion models have spurred signiﬁcant gains in density modeling and image generation, precipitating an industrial revolution in text-guided AI art generation.
We introduce a new mathematical foundation for diffusion models inspired by classic results in information theory that connect Information with
Minimum Mean Square Error regression, the so-called I-MMSE relations.
We generalize the I-MMSE relations to exactly relate the data distribution to an optimal denoising regression problem, leading to an elegant reﬁnement of existing diffusion bounds.
This new insight leads to several improvements for probability distribution estimation, including theoretical justiﬁcation for diffusion model ensembling. Remarkably, our framework shows how continuous and discrete probabilities can be learned with the same regression objective, avoiding domain-speciﬁc generative models used in variational methods.
I NTRODUCTION
Denoising diffusion models (Sohl-Dickstein et al., 2015) incorporating recent improvements (Ho et al., 2020) now outperform GANs for image generation (Dhariwal & Nichol, 2021), and also lead to better density models than previously state-of-the-art autoregressive models (Kingma et al., 2021).
The quality and ﬂexibility of image results have led to major new industrial applications for automatically generating diverse and realistic images from open-ended text prompts (Ramesh et al., 2022; Saharia et al., 2022;
Rombach et al., 2022).
Mathematically, diffusion models can be understood in a variety of ways: as classic denoising autoencoders (Vincent, 2011) with multiple noise levels and a new architecture (Ho et al., 2020), as VAEs with a ﬁxed noising encoder (Kingma et al., 2021), as annealed score matching models (Song & Ermon, 2019), as a non-equilibrium process that tractably bridges between a target distribution and a Gaussian (Sohl-Dickstein et al., 2015), or as a stochastic differential equation that does the same (Song et al., 2020; Liu et al., 2022).
In this paper, we call attention to a connection between diffusion models and a classic result in information theory relating the mutual information to the Minimum Mean Square Error (MMSE) estimator for denoising a Gaussian noise channel (Guo et al., 2005).
Research on Information and MMSE (often referred to as I-MMSE relations) transformed information theory with new representations of standard measures leading to elegant proofs of fundamental results (Verdú & Guo, 2006).
This paper uses a generalization of the I-MMSE relation to discover an exact relation between data probability distribution and optimal denoising regression.
The information-theoretic formulation of diffusion simpliﬁes and improves existing results.
Our main contributions are as follows.
• A new, exact relation between probability density and the global optimum of a mean square error denoising objective of the form,  log p ( x ) = [Equation]
[Equation] terms, with useful variations in Eq.
(8), (9) and (11).
Because this expression is exact and not a bound, we can convert any statements about density functionals such as entropy into statements about the unconstrained optimum of regression problems easily solved with neural nets.
• The same optimization problem can also be exactly related to discrete probability mass, Eq.
In contrast, the variational diffusion bound requires specifying a separate discrete decoder term in the generative model and associated variational bound. Our uniﬁcation of discrete and continuous probabilities justiﬁes empirical work applying diffusion to categorical variables.
• In experiments, we show that our approach can take pre-trained discrete diffusion models and re-interpret them as continuous density models with competitive log-likelihoods.
Our approach allows us to ﬁne-tune and ensemble existing diffusion models to achieve better NLLs.
F UNDAMENTAL P OINTWISE D ENOISING R ELATION
Let p ( z  | x ) be a Gaussian noise channel with z  = p
x + ✏ and ✏ ⇠ N (0 , I ) , where  represents the Signal-to-Noise Ratio (SNR) and p ( x ) is the unknown data distribution.
This channel has exploding variance as the SNR increases but we will see that the variance of this channel can be normalized arbitrarily without affecting results.
Our convention matches the information theory literature and signiﬁcantly simpliﬁes proofs.
The seminal result of Guo et al.
(2005) connects mutual information with MMSE estimators,
[Equation]
Here, the MMSE refers to the Minimum Mean Square Error for recovering x in this noisy channel,
[Equation]
We refer to ˆ x as the denoising function.
The optimal denoising function ˆ x ⇤ corresponds to the conditional expectation, which can be seen using variational calculus or from the fact that the squared error is a Bregman divergence ( Banerjee et al.
Prop. 1),
[Equation]
The analytic solution is typically intractable because it requires sampling from the posterior distribution of the noise channel.
Recent work on variational diffusion models writes a lower bound on log likelihood explicitly in terms of MMSE (Kingma et al., 2021), suggesting a potential connection to Eq.
However, the precise nature of the connection is not clear because the variational diffusion bound is an inequality while Eq.
(2) is an equality, and the variational bound is formulated pointwise for log p ( x ) at a single x , while Eq.
(2) is an expectation.
We now introduce a point-wise generalization of Guo et al’s result that is the foundation for all other new results in this paper.
[Equation]
The marginal is p ( z  ) =
R p ( z  | x ) p ( x ) d x , and the pointwise MMSE is deﬁned as follows,
[Equation]
Pointwise MMSE is just the MMSE evaluated at a single point x , and E p ( x ) [mmse( x ,  )]
= mmse(  ) .
Taking the expectation with respect to x of both sides of Eq.
(4) recovers Guo et al’s famous result, Eq.
Our proof of Eq.
(4) uses special properties of the Gaussian noise channel and repeated application of integration by parts, with a detailed proof given in Appendix A.1.
This result can also be seen as a special case of Theorem 5 from Palomar & Verdú (2005) by replacing their general channel with our channel written in terms of  and then using the chain rule.
In the rest of this paper, we show how this more general version of the I-MMSE relation can be used to reformulate denoising diffusion models.
D IFFUSION AS T HERMODYNAMIC I NTEGRATION
We ﬁrst use Eq.
(4) to derive an expression for log-likelihood that resembles the variational bound.
However, using results from the information theory literature (Verdú & Guo, 2006), we ﬁnd that the expression can be signiﬁcantly simpliﬁed and certain terms that are typically estimated can be calculated analytically.
Our derivation is inspired by the recent development of thermodynamic variational inference (Masrani et al., 2019;
Brekelmans et al., 2020) which shows how to construct a path connecting a tractable distribution to some target, such that integrating over this path recovers the log likelihood for the target model.
The paths between distributions can be generalized in a number of ways (Masrani et al., 2021;
Chen et al., 2021).
Typically, though, these path integrals are difﬁcult to estimate because they require expensive sampling from complex intermediate distributions.
A distinctive property of diffusion models is that the Gaussian noise channel, which transforms the target distribution into a standard normal distribution, can be easily sampled at any intermediate noise level.
We use the fundamental theorem of calculus to evaluate a function at two points in terms of the integral of its derivative,
[Equation]
This approach is known as “thermodynamic integration” in the statistical physics literature (Ogata, 1989; Gelman & Meng, 1998), where evaluation of the endpoints corresponds to a difference in the free energy or log partition function, and the derivatives of these quantities may be more amenable to Monte Carlo simulation.
Consider applying the thermodynamic integration trick to the following function, f ( x ,  ) ⌘ D [Equation] [ p
[Equation] z  | x ) k p ( z  )] . As before, we have p ( x ) , the data distribution, and a Gaussian noise channel, z
= p  x + ✏ with different signal-to-noise ratios  ,
[Equation]
In the second line, we expanded the KL divergence and used Bayes rule.
Next, we re-arrange and use
Eq.
(4) to re-write the integrand
[Equation]
[Table]
[Equation]
Comparing to a particular variational bound for diffusion models in the continuous time limit (Eq.
15 from Kingma et al.
, we see this expression looks similar (see App.
A.7 for more details).
However, our derivation so far is exact and we haven’t introduced any variational approximations.
Prior and reconstruction loss terms are stochastically estimated in the variational formulation, but we show this is unwise and unnecessary.
Consider the limit where  1 ! 0 .
In that case, the prior loss will be zero.
The reconstruction term, for continuous densities, becomes inﬁnite in the limit of large  0 .
For this reason, recent diffusion models only consider reconstruction for discrete random variables, P ( x | z  0 ) .
Estimation of conﬂicting divergent terms and the need for separate prior and reconstruction objectives can be avoided, as we now show.
Simple and exact probability density via MMSE
We consider applying thermodynamic integration to a slightly different function, and expand the range of integration to  2 [0 , 1 ) .
Consider sending samples from either the data distribution p ( x ) or a standard Gaussian p G ( x )
=
N ( x ; 0 , I ) through our Gaussian noise channel.
We denote the MMSE for the channel with Gaussian input as mmse G (  ) , and write its marginal output distribution as p G ( z  ) =
R p ( z  | x ) p G ( x ) d x .
Finally, we deﬁne the function f
( x ,  ) as
[Equation]
In the limit of zero SNR, we get lim  !
f ( x ,  ) = 0 .
In the high SNR limit we use the following result proved in App.
A.2,
[Equation]
Combining this with Eq.
(4), we can write the log likelihood exactly in terms of the log likelihood of a Gaussian and a one dimensional integral.
[Equation]
This expresses density in terms of a Gaussian density and a correction that measures how much better we can denoise the target distribution than we could using the optimal decoder for Gaussian source data.
The density can be further simpliﬁed by writing out the Gaussian expressions explicitly and simplifying with an identity given in App.
A.3.
[Equation]
This expression shows that density can be written solely in terms of the global optimum of a particular regression problem, the denoising MSE.
This is convenient because neural networks excel at unconstrained optimization of MSE loss functions.
If we take the expectation of  log p ( x ) using Eq.
(9), we recover a relatively recently discovered representation of the differential entropy, h ( p ) , in terms of MMSE (Verdú & Guo, 2006).
This highlights an advantage of our approach, as all density functionals can be rewritten in terms of the solution of an unconstrained regression problem,
[Equation]
Since the ﬁrst integral in Eq.
(9)-(10) does not depend on x , it is tempting to absorb it into a constant.
However, the ﬁrst integral diverges, and the second integral typically divergences as well – only the difference converges.
This observation will help improve density estimation, by noticing that only the gap between MMSEs is important and that the gap becomes small at high and low values of SNR.
[Equation]
[Figure]
[Equation]
[Figure]
We show example MMSE curves for denoising with Gaussian input versus non-Gaussian inputs in Fig.
The goal is to integrate the gap between these two curves, one analytic (for Gaussians) and one given by the data.
Note that the gap could in principle be positive or negative.
However, we know that Gaussians have the maximum possible MMSE, compared to any distribution with the same variance or covariance, when used as an input to a Gaussian noise channel (Shamai & Verdu, 1992).
Therefore, if we always compare to Gaussians with the same variance or covariance as the data, we can guarantee that the gap is positive.
In practice, if the gap ever becomes negative, which we show can occur for models appearing in the literature in Sec. 5, it means that the discovered estimator is sub-optimal and we can fall back to the optimal Gaussian estimator to improve results in this region.
We also see that for large and small SNR values, the gap becomes small.
At low SNR, the noisy data is approximately Gaussian.
At high SNR, the noise goes to zero, so a linear (Gaussian) denoiser is sufﬁcient in either case.
Recognizing that the important signals come from intermediate SNR values can help save computation over approaches that indiscriminately optimize over the whole curve, as we discuss in Sec. 4.
Standardizing data to have unit variance to ensure that the MMSE gap is always positive may be inconvenient, so we derive a variation of Eq.
(9) that takes the scale into account.
Instead of using p G ( x ) =
N ( x ; 0 , I ) , we can take the base measure to be a Gaussian, p G ( x ) =
N ( x ; µ , ⌃ ) , with mean and covariance that match the data.
Letting  1 , . .
,  d be the eigenvalues of the covariance matrix ⌃ , we derive a compact expression in App.
A.4,
[Equation]
Gaussian entropy
[Equation]
| {z }
Deviation from Gaussianity .
This expression tells us precisely which part of the reconstruction mean square error curve is actually important, namely the part that deviates most from Gaussianity.
Note that if the eigenvalues of ⌃ are not feasible to estimate, we can use a diagonal covariance matrix for the base Gaussian and still preserve the desired property that the gap between the data MMSE and Gaussian MMSE is non-negative (Shamai & Verdu, 1992).
Finally, we make several remarks to relate our exact expression for the likelihood in Eq.
(9) to existing work in the diffusion literature.
• Since the MSE minimization is performed at each  , reweighting terms with different  (Kingma et al., 2021) should not have an effect as long as we achieve the global minima at each  .
Several heuristics have been suggested for improving the “noise schedule”, which corresponds to points for evaluating the MMSE integral in our formulation (Ho et al., 2020; Kingma et al., 2021; Nichol & Dhariwal, 2021).
Our formulation highlights the importance of sampling from SNRs where there is a large gap, and we suggest a simple and effective strategy for this in Sec. 4.
• The literature on diffusion models also suggests more complex denoising distributions that also model the covariance (Nichol & Dhariwal, 2021).
Modeling the covariance of the denoising model is unnecessary in our approach, as our exact expressions never require it.
• Compare Eq.
(9) to the exact, continuous density expression in Eq.
39 of Song et al.
(2020), which combines neural ODE ﬂows and diffusion models.
The form of that expression is:
[Equation]
The ﬁrst step to using this expression is to solve a differential equation for the full trajectory, x ( t ) , that depends on many evaluations of a learned denoising diffusion (or score) model.
Then, estimating the integral is highly non-trivial as g is complex and its divergence is expensive to compute, necessitating some stochastic approximations.
See Sec.
6 for additional discussion.
Discrete probability estimator
Modeling probability distributions over continuous versus discrete random variables typically requires rather different approaches.
An unusual feature of I-MMSE relations is that they naturally handle both types of random variables.
In this subsection, consider that x 2 X ⇢ Z d , with a domain that is discrete but numeric (non-numeric discrete variables can be handled via an appropriate mapping function (Guo et al., 2005)).
We will use capital P ( x ) to denote probability (rather than probability density ) over this discrete domain.
Note that the Gaussian noise channel, z  = p
x + ✏ , is still continuous with an associated probability density, p ( z  | x ) .
Re-doing the analysis in Eq.
(6) leads to the same expression, but replacing p ( x ) with P ( x ) .
The ﬁrst two terms in Eq.
(6) go to zero,
[Equation]
This leads to the following form for the discrete probability.
[Equation]
At large SNR, z  is very concentrated around a discrete point speciﬁed by x .
In that case, we should be able to recover the true value from the slightly perturbed value with high probability.
Sec. 4 derives tail bounds used for approximating this integral numerically.
Taking the expectation of this general result recovers an equation for the Shannon entropy from Section VI of Guo et al. [Equation]
[Equation]
Conveniently, our discrete and continuous probability representations are nearly identical – they rely on the same integral and optimization and differ only by constant terms.
We discuss comparisons to the variational bounds in App. A.7 and numerical implementation details in Sec. 4.
I MPLEMENTATION
While we start with exact expressions for density and discrete probability in Eqs.
(9) and (12)
[Equation]
in practice there will be two sources of error when implementing our approach numerically.
First, we must parametrize our estimator as a neural network that may not achieve the global minimum mean square error and, second, we are forced to rely on numerical integration.
MMSE Upper Bounds
While the likelihood bounds in Eqs.
(6), (8), and (9) are exact, evaluating each mmse( x ,  ) term requires access to the optimal denoising function or conditional expectation
ˆ x ⇤ ( z  ,  ) =
E p ( x | z  ) [ x ] .
Using a suboptimal denoising function ˆ x ( z  ,  ) instead of ˆ x ⇤ ( z  ,  ) to approximate the MMSE, we obtain an upper bound whose gap can be characterized as
E
[Equation]
In App. A.
[Equation]
we derive this upper bound and its gap using results of Banerjee et al.
for general
Bregman divergences.
This translates to an upper bound on the Negative Log Likelihood (NLL),
[Equation]
Restricting Range of Integration
As noted in Fig.
1, only the difference mmse G (  )  mmse(  )  0 contributes to our expected NLL bound.
The non-negativity of this difference is guaranteed by the results of Shamai & Verdu (1992), but may not hold in practice due to our suboptimal estimators of the MMSE, E p ( x , z  )
[ k x  ˆ x ( z  ,  ) k 22 ]  mmse(  ) .
In regions    0 or    1 where the difference in mean square error terms appears to be negative, we can simply deﬁne ˆ x ( z  ,  ) ⌘ ˆ x ⇤ G
( z  ,  ) , where ˆ x ⇤ G ( z  ,  ) is the optimal, linear decoder for a
Gaussian with the same covariance as the data (App. A.5).
For this choice of decoder, the integrand becomes zero so that we may drop the tails of the integral outside of the appropriate range  2 [  0 ,  1 ] ,
E p ( x ) [  log p ( x )]  1 / 2 log det(2 ⇡ e ⌃ )  1 /
Z d  d
X
[Equation]
=
[Equation]
[Equation]
[Equation]
Parametrization
We parametrize ˆ x ( z  ,  ) ⌘
( z   ˆ ✏ ( z  ,  )) /
p  , 8  2 [  0 ,  1 ] .
With this deﬁnition, we can re-arrange to see that the error for reconstructing the noise, ✏ , is related to the error for reconstructing the original image, ˆ ✏ ( z  ,  )  ✏ = p  ( x  ˆ x ( z  ,  )) .
Our neural network then implements ˆ ✏ , a function that predicts the noise.
The inputs to this network are ( z  ,  ) .
Note that we can equivalently parametrize our network to use the inputs ( z  / p 1 +  ,  ) .
This choice is preferable so that the variance of the noisy image representation is bounded, while the SNR is unchanged.
Numerically, it is better to work with log SNR values, so we change variables, ↵ = log  , which leads to the following form
[Equation]
Here, f ⌃ ( ↵ ) ⌘
P di =1  ( ↵ + log  i ) , using the traditional sigmoid function,  ( t ) =
1 / (1 + e  t ) .
Numerical Integration
The last technical issue to solve is how to evaluate the integral in Eq.
(14) numerically.
We use importance sampling to write this expression as an expectation over some distribution, q ( ↵ ) , for which we can get unbiased estimates via Monte Carlo sampling.
This leads to the ﬁnal speciﬁcation of our loss function E p ( x ) [  log p ( x )]  L , where
[Equation]
All that remains is to choose q ( ↵ ) .
We use our analysis of the Gaussian case to motivate this choice.
Note that f ⌃ ( ↵ ) is a mixture of CDFs for logistic distributions with unit scale and different means.
Mixtures of logistics with different means are well approximated by a logistic distribution with a larger scale (Crooks, 2009).
So we take q ( ↵ ) to be a truncated logistic distribution with mean µ = Mean i
(  log  i ) and scale s
= p 1 + 3 / ⇡ 2 Var i (log  i ) , based on moment matching to the mixture of logistics implied by f ⌃ ( ↵ ) .
Samples can be generated from a uniform distribution using the quantile function, ↵ = µ + s log t/ (1  t ) , t ⇠ U [ t 0 , t 1 ] .
The quantile range is set so that ↵ 2 [ µ  4 s, µ + 4 s ] .
The objective is estimated with Monte Carlo sampling, providing a stochastic, unbiased estimate of our upper bound in Eq.
Comparing between continuous and discrete probability estimators
Recent diffusion models are trained assuming discrete data.
We can measure how well they model continuous data by viewing continuous density estimation as the limiting density of discrete points (Jaynes, 2003).
Treating p ( x ) as a uniform density in some d -dimensional box or bin of volume  d around each discrete point leads to the relation, E [  log p ( x )]
=
E [  log P ( x 2 bin ) / (  ) d ]
= E [  log P ( x 2 bin )  d log  ] .
In the other direction, when we use a continuous density estimator to model a discrete density, we use uniform dequantization (Ho et al., 2019).
Direct discrete probability estimate
Finally, we derive a practical upper bound on negative log likelihood for discrete probabilities.
[Equation]
In App.
B.1, we analytically derive upper bounds for the left and right tail of the integral, expressed using c (  0 ,  1 ) .
We also get an upper bound from using a denoiser that is not necessarily globally optimal.
For training, we use the Monte Carlo estimator of the integral so that they objective is exactly equivalent to the continuous density estimator. At inference time, we would like a low variance numerical estimate of the integral.
Importance sampling gives low variance estimates if the importance sampling distribution well matches the integrand. While our proposed importance sampler matches the continuous estimator well, it does not match the discrete integrand, leading to high variance (App. B.5). Therefore, for test time inference of discrete log likelihood, we use the trapezoid rule for integration (Lartillot & Philippe, 2006; Friel et al., 2014;
Hug et al., 2016). Monotonicity of mmse( x ,  ) could alternately suggest using a left Riemann sum approximation (Kingma et al. Fig. 2, Masrani et al.
E XPERIMENTS
Our results establish an exact connection between the data likelihood and the solution to a regression problem, the Gaussian denoising MMSE.
If we integrate the MSE curve of a particular denoiser and get a worse bound, it simply means that the denoiser does not achieve the MMSE.
Interestingly, variational diffusion models optimize an objective that can be made quite similar with appropriate choices for variational distributions (App. A.7).
Therefore, we can take previously trained diffusion models and evaluate alternate likelihood bounds using the methods described above.
We can attempt to improve these bounds by directly optimizing the denoising MSE.
Finally, a straightforward implication of our results is that we would like the minimum MSE denoiser at each SNR value; therefore, if we have different denoisers with lower error at different SNR values, we can combine them to get a density estimator which outperforms any individual one.
Ensembling diffusion models that “specialize” in different SNR ranges can likely be used to construct new SOTA density models.
For the following experiments we use the CIFAR-10 dataset, scaled to have pixel values in [  1 , 1] , and consider a pre-trained DDPM model (Ho et al., 2020) and an Improved DDPM we refer to as
IDDPM (Nichol & Dhariwal, 2021).
We use 4000 diffusion steps for calculating variational bounds, and for our IT based methods we used 100 steps since further increases did not change the bound.
More details on models and training can be found in App.
B.3.
Continuous Density Estimation with Diffusion
We ﬁrst explore the results of continuous probability density estimation on test data using variational bounds versus our approach.
Recent diffusion based models treat the data as discrete, bounding log P ( x ) , however, we could also use diffusion models to give continuous density estimates by interpreting the last denoising step as providing a Gaussian distribution over x .
For comparison, we can also take the variational bound for discrete data and create a density estimate by making density uniform within each bin.
The results are shown in Fig.
For our estimator, the shaded integral between the Gaussian MMSE and [Table] the denoising MSE is used, and this provides noticeably improved density estimates.
Note that for our bound calculated using the IDDPM, we throw away the part of their network that estimates the covariance and still achieve improved results, validating our assertion that variational modeling of covariance is unnecessary.
Continuous density estimators applied to intrinsically discrete pixel data, like CIFAR-10, can in principle lead to very low NLLs if the model learns to put large mass on discrete points.
In our case, we are comparing the same model so the comparison would still be meaningful.
Furthermore, we can see that the diffusion models are not putting probability mass on discrete points by comparing the
MSE curves to denoisers where discretization is purposely introduced in Fig.
Discrete Probability
NLL bounds for diffusion models in recent work are for discrete probability estimates.
We compare to both our estimate based on an exact relation between MMSE and discrete likelihood (Eq.
(13), (16)), and based on our continuous NLL estimator treated as a discrete estimator with uniform dequantization.
Because existing diffusion architectures do not naturally concentrate predictions on grid points at high SNR, we have to explicitly add rounding via a “soft discretization” function (see App. B.4), which leads to much lower error at high SNR, as seen in Fig. Further improvements from ensembling the best models at different SNRs are described below.
Fine-tuning
Fig. 2 and 3 show that we can improve log likelihoods by ﬁne-tuning existing models using our regression objective derived in Eq. 15, rather than the variational bound.
In particular, we see that the improvements for continuous density estimation come from reducing error at high SNR levels.
The inability of existing architectures to exploit discreteness in the data is the reason ﬁne-tuning improves the continuous but not the discrete estimator in Table 2.
Better results for discrete estimators could be obtained by introducing a trainable soft discretization nonlinearity into the architecture before tuning.
See App.
B.3 for additional experimental details.
Ensembling
Finally, we propose to ensemble different denoising diffusion models by choosing the denoiser with the lowest MSE at each SNR level in Eq.
Since our estimator depends only on estimating the smallest MSE at each  , our likelihood estimates will beneﬁt from combining models whose relative performance differs across SNR regions.
In Fig.
2 and 3, we report improved results from ensembling DDPM
(Ho et al., 2020), IDDPM (Nichol & Dhariwal, 2021), ﬁne-tuned versions, and rounded versions (in the discrete case).
For discrete estimators, rounding is useful at high SNR but counterproductive at low SNR.
The ensemble MSE (shaded region of Fig. 3) gives a better NLL bound than any individual estimator.
R ELATED W ORK
The developments in diffusion models that have enabled new industrial applications (Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) build on a rich intellectual history of ideas spanning many ﬁelds including score matching(Hyvärinen & Dayan, 2005), denoising autoencoders (Vincent, 2011), nonequilibrium thermodynamics (Sohl-Dickstein et al., 2015), neural ODEs (Chen et al., 2018), and stochastic differential equations (Song et al., 2020; Huang et al., 2021).
The observation that
[Table]
part of the variational diffusion loss could be written as an integral of MMSEs was made by Kingma et al.
To the best of our knowledge, the connection between I-MMSE relations in information theory and diffusion models has so far gone unrecognized.
Song et al.
(2020) introduced a different way to use diffusion to model probability densities via differential equations.
This approach requires solving a differential equation of a dynamic trajectory for each sample.
Our estimate can be computed in a single step by evaluating the score model at each SNR in parallel, but the neural probability ﬂow ODE will require thousands of sequential score function evaluations to solve. Concurrent work uses stochastic differential equations along with
Girsanov’s theorem, a stochastic version of the change of variables formula, to show that optimizing a regression objective is sufﬁcient to exactly match a stochastic process to some target density (Liu et al., 2022; Ye et al., 2022).
The results are used to improved sampling, but they use standard variational bounds for density estimation.
Our work, in contrast, exactly relates regression to density estimation but does not address sampling.
The form of their results for discrete distributions suggests a deeper connection between their work and this one.
Our work focused on density modeling, so approaches that forgo density modeling to improve image generation, for instance by doing diffusion in a latent space (Rombach et al., 2022), were not considered.
Bansal et al.
(2022) observe that denoising diffusion models using many types of noise can lead to good image sampling, however, our results highlight the special connection between
Gaussian denoising and density modeling.
Recent work has studied applicability of diffusion to discrete and categorical variables Austin et al. ; Gu et al. ; Chen et al.
(2022) has studied diffusion model’s applicability to categorical variables.
Our exact relation between denoising and discrete probability provides theoretical justiﬁcation for this promising line of research.
Finally, this paper gives a theoretical justiﬁcation for ensembling diffusion models, to achieve the best MSE at different SNR levels.
We see from concurrent work that this strategy is already being successfully employed Balaji et al.
C ONCLUSION
Variational methods are powerful, but require many choices in designing variational approximations.
More complex choices for the variational distributions could lead to tighter bounds, and a fair amount of work has explored this possibility.
Interestingly, though, the most successful reﬁnements of diffusion models have led to objectives that are more similar to the denoising MSE, and the results in this paper ﬁnally make it clear that regression is all that is needed for exact probability estimation, for both continuous and discrete variables.
We generalized the classic I-MMSE relation from information theory to introduce this simple, exact relationship between the data probability distribution and optimal denoisers,  log p [Equation] = [Equation]
R [Equation] [Equation]
constant terms.
This result pares away the unnecessary ingredients in the variational approach, and the simple and evocative form of the result can inspire many improvements and generalizations to be explored in future work.