Paper: DiffFlow: A Unified SDE for Score-Based Diffusion Models and Generative Adversarial Networks (ICLR-2024)

Reviewer comment (Theory):
It seems the instantiation to GANs is given via the coarse approximation which is not as direct and also is based on the density ratio. Some more popular GAN methods do not admit a density ratio perspective such as IPM-based GANs such as Wasserstein GAN, etc.

Author response:
Yes, you are correct. As pointed out by reviewer Pk86,  the unifying framework only encompasses derivations of the vanilla GAN and non-saturating GANs from Goodfellow et al. (2014), and not all GANs (like Wasserstein GANs, for example). We explicitly state this limitations starting from abstract.

Reviewer comment (Experiments):
There are no experimental studies on more novel / unique choices of weighting parameters. Due to the space requirements however, this is understandable.

Author response:
In the revised version, we added some preliminary experiments on the implementation of GANs in the DiffFlow framework. We also have some preliminary implementation on DiffFlow with score-matching by separate networks. We would be grateful if you could further point out additional experiments to strengthen the theory of current paper.

Reviewer comment (Experiments):
Is there a way to consider more general GAN frameworks beyond the vanilla GAN? such as Wasserstein GANs or non-saturating GANs.

Author response:
A1:  Yes. We have considered both the vanilla GANs and non-saturating GANs in Appendix B. We also discuss how non-saturating GANs avoid gradient vanishing from the perspective of particle gradient optimization.

Reviewer comment (Novelty):
There are other methods that use discriminators with diffusion models as a refining process [1]. Can you comment on how such a method relates to the unification provided in this paper?

Author response:
A2: [1] has fundamental difference from our current framework: given a \textbf{fixed} pretrained score network $\nabla \log p_{model}(x_t, t)$ that may deviates from real data distribution, the authors train an additional time-dependent discriminator $d(x_t, t)$ that discriminates between the real data and generated data at time step $t$ and correct the score network by $$\nabla \log p_{new}(x_t, t) = \nabla \log p_{model}(x_t, t) + \nabla \log \frac{d(x_t, t)}{1-d(x_t, t)}~.$$

In our framework, the score network and the discriminator network should be trained \textbf{jointly}. Furthermore, the noise in the discriminator should align with the diffusion process. Contrastly in [1], there are no noise added on the target distribution $q(x)$ during the training of discriminator. We have added this discussion in Appendix C in the revised submission.

Reviewer comment (Theory):
Is there any comment you can provide about how different weighting parameters will affect convergence?

Author response:
A3: We have discussed this point at the end of Appendix H. In particular, we discuss how regularization and noise weighting on the smoothed target distribution could affect the convergence rate. The general principle is to ensure the isoperimetric property of the target along the optimization path.

Reviewer comment (Reproducibility):
Can you provide more concrete guidance for practitioners willing to use this method and how the weighting parameters inform various choices in practice for different domains?

Author response:
A4: One message conveyed by unifying SDE of GANs and diffusion models is to provide a theoretic grounded trade-off between sampling quality and sampling speed, as demonstrated empirically in previous work of TDPM (see table 1,2, and 3 in [2]). 

In terms of the choice of weighting parameters, more empirical and theoretic investigations are needed. A preliminary theoretic analysis shows (discussed in appendix H) that the explicit weighted noise smoothing on the target distribution would benefit for the training and sampling process.

Reviewer comment (Presentation):
Although I understand that the current organization of materials is largely due to the page limitation, it is possible to make the presentation more complete by emphasizing important contributions. In my perspective, this work can benefit significantly from the following strategies: (1) rearrange the material in Section 3 heavily, e.g., move the majority of 3.2 into the appendix, and dedicate more space for the Diffusion-GAN algorithm; (2) It might be better to just state the informal versions and offer intuitive explanations of major theorems in the main text without getting into details about technical lemmas and assumptions, and defer the formal statements and conditions to the appendix.

Author response:
Thanks for your detailed comments on the presentation of the paper. We have re-arranged Section $3$, and in particular Section $3.2$ according to your advise. Furthermore, we added several experiments about DiffFlow-GAN to justify its feasibility in appendix L.

Reviewer comment (Experiments):
Lack of experiments: it is generally not fair to ask for experiments on a theory-oriented paper. However, a major selling point of such formulation is that it leads to a natural design of the diffusion-GAN method. Particularly, given the extensive discussion of the advantage of DiffFlow over vanilla GAN in appendix C, it is important to provide empirical verification on the performance of DiffFlow-GAN.

Author response:
Thanks for your detailed comments on the presentation of the paper. We have re-arranged Section $3$, and in particular Section $3.2$ according to your advise. Furthermore, we added several experiments about DiffFlow-GAN to justify its feasibility in appendix L.

Reviewer comment (Evaluation):
As it is currently written, the paper suffers from strong overclaiming of its results. Clarifying the claims and adjusting them so that they are properly supported by theoretical or empirical evidence is a necessity.

Author response:
This work claimed three contributions: i) a unified SDE for GANs and SDMs;  ii) a flexible trade-off between high sample quality and fast sampling speed; iii) exact likelihood inference. While the claims ii) and iii) seem vague in the original version, we argue that these two claims can be supported based on extensive related works:

Claim ii)  a flexible trade-off between high sample quality and fast sampling speed. This idea was initially introduced by TDPM [1] and our work is partially motivated from it. [1] combines the generation process of GANs and diffusion models that enables flexible trade-off between sample quality (FID) and sampling speed (NFE). The authors did extensive experiments on various datasets to validate this idea. While the idea of [1] is promising, the main focus is on empirical validations of compatibility between GANs and diffusion models that enables such a trade-off. Our work give a strict math formulation that justifies this idea theoretically. The unified framework incorporates the TDPM as special case (See R6 for more details), and does have the ability to achieve a flexible trade-off between high sample quality and fast sampling speed.

Claim iii) exact likelihood inference. This part is moved to appendix I.  It has been well investigated in the literature of diffusion models to derive the maximal likelihood training objective for diffusion SDE/ODE that enables likelihood estimation (for example see [2, 3, 4]). The core idea is to derive a trainable continuous-time ELBO for diffusion SDE. For the case of GANs, if we add small gaussian noise in the particle dynamics, a continuous-time ELBO can be derived that enables likelihood estimation. We add more discussion about this point in the appendix I of revised submission.

Since claim iii) is not the main argument in the paper and the techniques adopted are mature in the literature, we only claim (i) and (ii) in the revised version:


\emph{Can we obtain a unified theoretic framework for GANs and SDMs that enables a flexible trade-off between high sample quality and fast sampling speed?}



We hope this adjustment of claims could eliminate the confusion raised by reviewers and readers.

Reviewer comment (Experiments):
The stated research question claims developing a unified theoretical framework for GANs and SDMs that allows for a flexible trade-off between high sample quality and fast sampling speed, while enabling exact likelihood inference. While the unifying framework does exist, there is no element supporting the rest of the claim in the paper. There are new introduced algorithms, but their properties are not assessed with any experiment. Moreover, there is no concrete discussion as to why the framework enables exact likelihood inference in its general case (particularly for GANs).

Author response:
This work claimed three contributions: i) a unified SDE for GANs and SDMs;  ii) a flexible trade-off between high sample quality and fast sampling speed; iii) exact likelihood inference. While the claims ii) and iii) seem vague in the original version, we argue that these two claims can be supported based on extensive related works:

Claim ii)  a flexible trade-off between high sample quality and fast sampling speed. This idea was initially introduced by TDPM [1] and our work is partially motivated from it. [1] combines the generation process of GANs and diffusion models that enables flexible trade-off between sample quality (FID) and sampling speed (NFE). The authors did extensive experiments on various datasets to validate this idea. While the idea of [1] is promising, the main focus is on empirical validations of compatibility between GANs and diffusion models that enables such a trade-off. Our work give a strict math formulation that justifies this idea theoretically. The unified framework incorporates the TDPM as special case (See R6 for more details), and does have the ability to achieve a flexible trade-off between high sample quality and fast sampling speed.

Claim iii) exact likelihood inference. This part is moved to appendix I.  It has been well investigated in the literature of diffusion models to derive the maximal likelihood training objective for diffusion SDE/ODE that enables likelihood estimation (for example see [2, 3, 4]). The core idea is to derive a trainable continuous-time ELBO for diffusion SDE. For the case of GANs, if we add small gaussian noise in the particle dynamics, a continuous-time ELBO can be derived that enables likelihood estimation. We add more discussion about this point in the appendix I of revised submission.

Since claim iii) is not the main argument in the paper and the techniques adopted are mature in the literature, we only claim (i) and (ii) in the revised version:


\emph{Can we obtain a unified theoretic framework for GANs and SDMs that enables a flexible trade-off between high sample quality and fast sampling speed?}



We hope this adjustment of claims could eliminate the confusion raised by reviewers and readers.

Reviewer comment (Presentation):
The unifying framework only encompasses derivations of the vanilla GAN model from Goodfellow et al. (2014), and not all GANs (such as Wasserstein GANs). This limitation should be explicitly stated, starting from the abstract.

Author response:
Thanks for your useful advice. We stress this limitation in the abstract and conclusion with red font.

Reviewer comment (Theory):
An important weakness is the link of DiffFlow with GANs in Section 3.3. While central in the paper's claims and contributions, the link with GANs is loose and presented in an unclear fashion.

Author response:
This is a misconception. It is well established in the existing literature [7, 8] that the vanilla GAN or non-saturating GAN dynamics (eqn 27 and 29) can be *exactly* recovered by *one-step distillation* of the discriminator-guided particle dynamics (i.e., equation 12 or 13). The equivalence between the vanilla GAN and discriminator-guided dynamics GANs can be obtained by *rescaling the gradient* of the least square distillation objective in algorithm $1$. In the work of MonoFlow, the authors also discussed the problem of gradient vanishing in vanilla GANs arises from too small rescaled vector fields of discriminator-guided particle dynamics and it fixes the gradient vanishing problem by simply adding a constant in the generator output combined with a monotonic function (section 5.2 of [7]).  

In the revision, we add additional discussions about this point in the main paper and the end of appendix C. Due to space limitations and since the connection of DiffODE and vanilla GANs are mature in the literature, we will not move too much content in the main paper.

Reviewer comment (Presentation):
The development of Section 3.3 is incomplete for fully understanding the presented contribution, as it lacks details on how to handle the generator. The development in Appendix Section B should be moved into the main paper for completeness.

Author response:
This is a misconception. It is well established in the existing literature [7, 8] that the vanilla GAN or non-saturating GAN dynamics (eqn 27 and 29) can be *exactly* recovered by *one-step distillation* of the discriminator-guided particle dynamics (i.e., equation 12 or 13). The equivalence between the vanilla GAN and discriminator-guided dynamics GANs can be obtained by *rescaling the gradient* of the least square distillation objective in algorithm $1$. In the work of MonoFlow, the authors also discussed the problem of gradient vanishing in vanilla GANs arises from too small rescaled vector fields of discriminator-guided particle dynamics and it fixes the gradient vanishing problem by simply adding a constant in the generator output combined with a monotonic function (section 5.2 of [7]).  

In the revision, we add additional discussions about this point in the main paper and the end of appendix C. Due to space limitations and since the connection of DiffODE and vanilla GANs are mature in the literature, we will not move too much content in the main paper.

Reviewer comment (Theory):
Even taking into account Appendix Section B, the link with GANs remains loose. The paper shows that the gradient of the log ratio corresponds to an optimal vanilla GAN discriminator, but how this can be articulated with GAN generators is left unfinished in the appendix. For example, Equations 27 and 28 deal only with generator parameter evolution over training time, which does not correspond to Equations 12 or 13 in the main paper, hence not to DiffFlow.

Author response:
As we discussed before, a forward pass through a generator can be trained by both vanilla GANs or *one-step distillation* of the discriminator-guided particle dynamics with rescaled gradients.  These two methods of training the generator are equivalent up to numerical precision. 

We implement a simple code in Jupyter notebook to show the equivalence between two methods and further confirm the equivalence between DiffFlow-GANs and vanilla GANs.

Reviewer comment (Theory):
Appendix Section K illustrates the issue, where TPDM (Zheng et al., 2022) is framed using the DiffFlow equation. The generating equation of DiffFlow is split into a discriminator-guided dynamics part (assimilated to the first step of TPDM) and a score-guided dynamics part (second step of TPDM). However, in TPDM, the first step is not discriminator-guided dynamics but a forward pass through a generator, which is not encompassed in the proposed framework.

Author response:
As we discussed before, a forward pass through a generator can be trained by both vanilla GANs or *one-step distillation* of the discriminator-guided particle dynamics with rescaled gradients.  These two methods of training the generator are equivalent up to numerical precision. 

We implement a simple code in Jupyter notebook to show the equivalence between two methods and further confirm the equivalence between DiffFlow-GANs and vanilla GANs.

Reviewer comment (Novelty):
By Proposition 2, the convergence results (minimization of the KL and Theorem 1) seem to be direct consequences of the studied dynamics being equivalent to the well-known Langevin dynamics; cf. Jordan et al. (1998) and extensive literature on the topic.

Author response:
This is a misunderstanding. While the minimization of KL (decreasing lemma) can be easily obtained by the well-established theorems from Wasserstein gradient flows, the convergence is hard to obtain for a general target $q(x)$. In the standard literature of sampling with Langevin dynamics, it is assumed that $q(x)$ satisfies some property of isoperimetry inequality, such as poincare or logarithmic sobolev inequality. Then, the convergence rate can be explicitly obtained based on the estimation of constants of isoperimetry  inequalities (see a seminar work of [5]). 

While in Theorem 1, we did not assume any isoperimetry conditions on the target $q(x)$. We instead show that by explicitly smoothing the the functional objective,  we can obtain an asymptotic convergence of the dynamics of the diffusion GAN.

From a technical perspective, the asymptotic convergence is obtained by constructing a Gaussian-Poincare inequality based on the estimation of the smoothed log density ratio (see Lemma $1$ for details). To the best of our knowledge, this series of techniques to obtain Theorem 1 (eqn37--eqn52 in the original paper) is nontrivial and there is no similar technique adopted in the previous literature. 

In appendix H, we discuss in Theorem 2 on how to use standard techniques in the well-established Langevin dynamics literature to obtain a linear convergence analysis of diffusion GAN dynamics under stronger assumptions of the target distribution $q(x)$. We defer Theorem 2 to appendix since it can be easily obtained based on the fact that the studied dynamics is equivalent to the well known Langevin dynamics.