Paper: Denoising Diffusion Bridge Models (ICLR-2024)

Reviewer comment (Novelty):
The authors' work severely lacks a detailed related works section. Many related works and contributions have been briefly commented in passing. Without proper acknowledgement of prior work, the authors' contributions appear inflated.

Author response:
We thank the review for pointing out our paper’s lack of discussion of related works. We greatly appreciate the thoughtful suggestions. Due to space limitation of the main text, we could not properly discuss the large body of related works in detail. According to suggestions from you and reviewer H8Nr, we have included a revised version of related works in the new supplementary files and we intend to replace the corresponding section in main text with the revised version. Please be sure to take a look.

Reviewer comment (Novelty):
Lacking reference [2], this appears to be the same idea but for discrete interpretation of diffusion models / bridge, derived through a variational approach similar to DDPM rather than time reversal - but which are well known to be equivalent. [2] does not condition on the end marginal in the network, but does in the h-transform.

Author response:
- [2] proposes to connect two image domains via discrete Brownian bridges, and proposes to reverse the bridge for generation. This is a discrete-time special case of the VE bridge we consider, and our method also shows success on VP bridges that work well in continuous time. In addition, our sampler takes much fewer steps for best generation results.

Reviewer comment (Novelty):
[5] is highly related in that the high level approach is the same: train a network between two points by reversing a diffusion bridge. The differences lie in [5] focusing on general / nonlinear forward process for sampling bridges rather than generative modelling. [5] requires simulating the forward diffusion as nonlinear forward bridges cannot be sampled in closed form. [5] is for a fixed pair rather than amortized across pairs. Equation 6 within Theorem 1 of this work appears to coincide with the equation written below Equation (4) in Section 2.1 of [5].

Author response:
> Is Theorem 1 different from Section 2.1 in (Heng et al. 2021)?

Section 2.1 in Heng et al. 2021 presents time reversal for a single diffusion bridge (pinned on both ends). Our Theorem 1 generalizes it to tackle $x_0$ drawn from a distribution. In addition, we present the probability-flow ODE version (which is crucial for our ODE sampler) that produces the same marginal distribution as the SDE.

Reviewer comment (Novelty):
The authors claim "A related work (Somnath et al., 2023) similarly establishes a diffusion bridge that translates between two distributions and has seen success in protein differentiable domain, but the training objective requires sampling an entire SDE trajectory for computation". As far as the reviewer is aware this is not true and it is unclear how the authors came to this conclusion.

Author response:
> Misinterpreting (Somnath et al.,2023) 

We appreciate the reviewer pointing this out. This mistake has been corrected.

Reviewer comment (Theory):
It is not clear why [3] is being cited for the Schrodinger bridge interpretation of diffusion models. The approach of [3] is not a Schrodinger bridge between marginals but two diffusion models back to back with a Gaussian between; there are significantly more relevant works. This is potentially misleading to the reader.

Author response:
> Should cite other more related works than [3].

Thank you for pointing this out. We will more prominently discuss other more related works as suggested.

Reviewer comment (Theory):
The statement "A recent work (Liu et al., 2023) considers a special case of SB" is inaccurate. I2I SB [9] does not actually result in a Schrodinger Bridge, neither does the same method detailed in Aligned SB [1]. They both perform bridge matching with respect to Brownian bridge reference diffusion but for a data driven coupling. Given the data driven coupling does not correspond to the coupling from Brownian motion, the bridge matching procedure returns a Markovian projection and hence breaks the coupling. The reviewer references discussion in [7] that the Schrödinger Bridge is the unique path measure which satisfies the initial and terminal conditions, is Markov, and is in the reciprocal class of Q. One needs to iterate on the coupling and drift in order to obtain a SB. This is a tangential remark as one does not need an optimal diffusion in order to perform conditional generation if it can be supervised with given paired samples from a coupling, but the reviewer feels errors in this area should be addressed.

Author response:
> I$^2$SB is not SB?

Precisely speaking, according to Liu et al. (2023), the class of SB they consider is a “tractable SB with the Dirac delta boundary” (Corollary 3.2), which assumes special structure on data distribution for the algorithm to be tractably handled. We do agree that Aligned SB do not result in SB and is doing bridge matching (as the first iteration specified in the algorithm in Shi et al. (2023)).

Reviewer comment (Theory):
The connection of diffusion bridges to flow matching has been established and detailed in [7], detailing the limit as the noise coefficient goes to 0. The original flow matching paper [8] is even derived using results from [4] on bridge matching, so this is widely known. Given the score function in this work is conditioned on x_T, the learnt backward diffusion is non-Markovian, hence retains the initial marginal coupling and would not recover rectified flow / Schrödinger bridge by iterative Markovian fitting as in [7] or in rectified flow. Therefore, the reviewer argues this work does not subsume flow matching / drift matching.

Author response:
> Does DDBM subsume Flow-Matching/Rectified Flow?

The additional input of $x_T$ is a helpful design choice that guides generation towards the data distribution. This is mostly a practical decision. However, note that our score-matching objective is sound without injecting $x_T$ as additional condition, in which case the reverse-time SDE is Markovian. Since the target score as described exactly reduces to that of Flow-Matching (the OT formulation) and Rectified Flow, our framework reduces these special cases. We have noted this point in our additional write-up.

Reviewer comment (Experiments):
The authors only validate the work with image-to-image translation. In this domain, ControlNet is the dominant solution. I value that the 2 models might not be competing solutions. Can the authors clarify why they didn't compare against ControlNet in the image-to-image task?

Author response:
You are correct. Our framework is orthogonal to ControlNet, which is an architectural design at its core. Our framework can also use ControlNet architecture and start from pretrained diffusion weights for the image-to-image tasks. However, due to limited resources and time, we did not proceed to larger scale experiments using ControlNet. Nevertheless, we do plan to scale up our framework using more popular architectures in the future.

Reviewer comment (Novelty):
With generalized flow matching models we can in theory move between any distributions. What is the advantage of DDBMs compared to flow matching? I found some reasoning in the paper but didn't fully understand, therefore I appreciate a brief explanation from the authors.

Author response:
In theory they both aim to learn a model to move between distributions. However, we designed our framework with the practical motivation about how can can maximally transfer empirical success of diffusion models to the bridge framework, which has seen limited empirical success in these high-dimensional image translation tasks. The framework is developed to allow reuse/generalization of many prior successful implementation choices such as the preconditioning in EDM and higher-order sampler, which Flow-Matching lack. With these choices, our framework can achieve better performance in quality and speed.

Reviewer comment (Experiments):
Why didn't the authors compare against ControlNet?

Author response:
You are correct. Our framework is orthogonal to ControlNet, which is an architectural design at its core. Our framework can also use ControlNet architecture and start from pretrained diffusion weights for the image-to-image tasks. However, due to limited resources and time, we did not proceed to larger scale experiments using ControlNet. Nevertheless, we do plan to scale up our framework using more popular architectures in the future.

Reviewer comment (Novelty):
What is the advantage of DDBMs when compared to FM given that both can in theory model flow between arbitrary distributions?

Author response:
In theory they both aim to learn a model to move between distributions. However, we designed our framework with the practical motivation about how can can maximally transfer empirical success of diffusion models to the bridge framework, which has seen limited empirical success in these high-dimensional image translation tasks. The framework is developed to allow reuse/generalization of many prior successful implementation choices such as the preconditioning in EDM and higher-order sampler, which Flow-Matching lack. With these choices, our framework can achieve better performance in quality and speed.

Reviewer comment (Novelty):
[4] shows how to do this for both an ODE and an SDE (with the score), also in finite time. It also shows how to avoid the added complexity of doob, and how SBDM is a subcase.

Author response:
- [4] is a general theory that directly constructs a bridge using an interpolation map and avoids the use of Doob’s h-function. Our method is based on Doob’s h-function from which a direct interpolation map can be constructed for marginal sampling. Furthermore, the training objective is quite different as ours is based on denoising score-matching without any regularization, while theirs is not. Our construction also shows stronger tie with diffusion models, which allow extending existing designs for the new bridge construction. These include reusing noise schedules, speicalized higher-order sampler, and network preconditioning. It’s not clear if these choices can be directly applied to [4].

Reviewer comment (Novelty):
[7] shows how to use either an ODE or SDE to connect densities under varying coupling.

Author response:
- [7] constructs a simulation-free training plan using brownian bridge as the interpolation map, and aims to match against both flow vector field and bridge score. Our method does not only consider brownian bridge as the bridge construction, and shows success for other types of bridge in practice. We only consider a denoising score matching loss without additional flow-matching loss as proposed. A set of practical design choices are also proposed to further improve quality and speed of our bridge model.

Reviewer comment (Novelty):
The related work section and the introduction should be thoroughly reworked to not overlook these clear contributions. The paper currently suggests there is a clear need to develop tools for this task, as if many works have not addressed it, which could be seen as overselling a concept by overlooking other work.

Author response:
> To the reviewer, it is really poor form to just stuff citations in the supplementary material. The purpose of a submission is to present work in the context of the literature to add to our shared knowledge. While I appreciate you taking the time to assemble a thorough related work, it is not a bad thing for your work to be similar to others, indeed it can be a good thing! It means multiple perspectives arising to similar conclusions, which is what we hope to do (I think!). As such, as a reviewer, I don't aim to suggest you need to differentiate your work as better or above other perspectives. Just contextualize it :)

Unfortunately, due to ICLR rules, we are only allowed to submit a 9 page paper (increasing this would otherwise disqualify our paper). Furthermore, we are also running out of time to address all reviewer comments due to the shortened discussion period (where we can update the main pdf) and conflicting deadlines, meaning that it would be extremely untenable for us to cut the main manuscript to fit in our enhanced discussion. We included the discussion in the appendix to give us the freedom to write down all of our thoughts while also avoiding damaging the flow of the main paper. We apologize if this came off as “stuffing” citations in the supplementary material; our goal was simply to find space to address all reviewer concerns about prior work before fully integrating the enhanced discussion.

Generally, we agree that our work fits into this (extremely) broad field of diffusion models with bridge additions. In particular, we agree that most of our developed theory can be (relatively easily) rederived from existing frameworks, as most papers in this field build off of the same few constructions. In fact, this is actually similar to how Schrodinger Bridge and Entropic Optimal Transport are fundamentally the same thing despite their different formulations. However, the difference with our paper (that we must stress) is the ability to connect with the existing diffusion model literature, which is necessary for achieving good empirical results. As such, it’s worth emphasizing why the existing literature has largely been unable to scale to these types of problems that we consider in our paper.

Reviewer comment (Experiments):
The statement that ODE-based methods [1,2,3] "tend to severely underperform when compared to diffusion models" is unsupported by evidence. Those papers aim to present the contrary, e.g. [3]. If the authors wish to make this statement, they should demonstrate it experimentally.

Author response:
> Statement regarding ODE-based methods’ performance. 

For ODE-based methods like the papers [1, 2, 3] mentioned above, we apologize for the misleading statement (which unfairly discredits these papers as being fundamentally unable to compete with diffusion). 

However, we wanted to emphasize that, although these methods have seen some improvements over the last year, they have yet to see the empirical success that diffusion models are accustomed to. In particular, the only case where ODE methods have consistently outperformed SDE methods (that we are aware of) is in the CIFAR10 dataset, but even then most of the improvements have stemmed from core work in the diffusion model/probability flow ODE space (e.g. EDM). We believe that this gap is fundamentally caused by a difference in research volume: diffusion models have seen many more small improvements (e.g. sampling, network parameterization, noise schedule)

As such, we have updated this statement to be “ODE methods have not achieved the same empirical success as diffusion models”.

Reviewer comment (Theory):
There may be a misinterpretation of what optimal transport is and its role in generative models constructed from continuous-time transport plans. The authors refer to "VE (OT)" bridges without providing a clear definition of OT.

Author response:
> Misinterpreting OT? 

We want to stress that our framework is not solving OT, and the VE bridge resembles OT plan in that they share a similar coupling in the form of $(1-t)x_0+t x_1$ in the noiseless limit (with factor changed as $t^2$ instead of $t$ for our case). We have deleted the “OT” in brackets after “VE” in the corresponding section to avoid the potentially misleading meaning. The OT-Flow-Matching we use in our text is only short for the Flow-Matching case in [3] where the author designs their conditional probability path by OT plan. We agree that this is a slight misnomer, and our updates intend to fix any possible misinterpretation issues.

Reviewer comment (Theory):
The second way to learn an optimal transport between distributions is for independent coupling $p(x_0)p(x_1)$, to learn a process $x_t = f_{\theta}(t,x_0,x_1)$ that gives a time-dependent velocity/score and density $v_t, p_t(x)$ that induces a minimum action (or least cost) in terms of Benamou–Brenier transport cost.

Author response:
> Different coupling between x_1 and x_0.

Note that for image translation tasks we assumed distribution coupling of the form $p(x_0, x_1)=p(x_0|x_1)p(x_1)$ (e.g. colored image samples depends on edge map) while for unconditional generation tasks we consider $p(x_0, x_1)=p(x_0)p(x_1)$. In both cases, we construct VP or VE bridges (coupling derived from underlying VP and VE processes), and we find both cases work well in practice for translation and unconditional generation.

Reviewer comment (Theory):
The introduction of the weighting factor in equation (13) is not shown in the text to preserve an exact/unbiased transport. The authors should justify this or state that it introduces a bias, even if a beneficial one.

Author response:
> Bias introduced in the weighting factor

Indeed, the weighting factor introduced causes the process to no longer follow the same marginal distribution as the bridge process if $w\neq 1$. We empirically find that this factor is useful because it allows us to interpolate between probability-flow ODE of unconditional diffusion model and that of a bridge model (by setting $w=0.5$ vs $w=1$), which is beneficial for unconditional generation with the ODE sampler generalized from EDM.

Reviewer comment (Experiments):
It would be better if the author could compare the DDBMs with some more advanced image-to-image translation algorithm such as controlnet.

Author response:
We want to note that our framework is orthogonal to ControlNet as ControlNet is primarily an architectural design. Our framework is theoretically different from (and based on) diffusion models, and we can also use ControlNet with pretrained diffusion weights as our starting point for training our bridge framework. One of the motivations for our work is to develop a bridge-based framework such that it is easy to transfer many successful design choices over to improve the empirical results of bridge-based frameworks, which have been met with limited empirical success. We have experimented with those from EDM, but ControlNet is certainly another choice one can adopt. However, due to limited resources and time we did not proceed to larger-scale experiments with ControlNet.

Reviewer comment (Novelty):
Is there any advantage of DDBMs over some previous image-to-image translation methods such as controlnet?

Author response:
We want to note that our framework is orthogonal to ControlNet as ControlNet is primarily an architectural design. Our framework is theoretically different from (and based on) diffusion models, and we can also use ControlNet with pretrained diffusion weights as our starting point for training our bridge framework. One of the motivations for our work is to develop a bridge-based framework such that it is easy to transfer many successful design choices over to improve the empirical results of bridge-based frameworks, which have been met with limited empirical success. We have experimented with those from EDM, but ControlNet is certainly another choice one can adopt. However, due to limited resources and time we did not proceed to larger-scale experiments with ControlNet.

Reviewer comment (Experiments):
It would be better to show the variation ability of DDBMs in terms of the generation results. For example, when translating edges into an image, there exists various solutions. Are DDBMs able to generate these various results?

Author response:
Yes indeed. This is one of the motivations for introducing stochasticity to the bridge sampler, as the conditional generation is multimodal. We indeed observe some variations in image translation when we set different seeds, and this variation comes from the introduced stochasticity in our hybrid sampler. We show some variations in the additional pdf page in the supplementary.