Paper: Deterministic Diffusion for Sequential Tasks (ICLR-2024)

Reviewer comment (Novelty):
The method of selecting the source distribution seems too trivial. When using perturbed history, the agent might get stuck in the previous information, which may harm the performance. The idea of using DDIM inversion is not novel as well.

Author response:
Concern 1.2: The agent getting stuck in the previous information is possible with warm starting where previous information is perturbed only at inference time (see eg. this paper). We, on the other hand, **train** the model to use previous data points to reach the next data point. Thus, the agent will not be stuck, as the model is trained for this scenario. We validate this empirically, showing in Fig. 4 that our algorithm has significantly lower FVD (which is better) than DDIM. In addition, we extended the Fig. 4 experiments in the supplemental rebuttal PDF (Section B.1), adding $DDIM_n$ initialization on the PHYRE dataset. We show that this initialization also provides superior performance than the other baselines (while still being inferior to the history-based initialization).
DDIM Inversion: With DDIM-based initialization, we use the DDIM result as the initial point for the deblending process which then attempts to reach the original data point (without reversing the order of DDIM sampling).

Reviewer comment (Experiments):
This paper does not compare to many works in speeding up diffusion models. The only compared baselines are standard diffusion models without speeding up tricks. It's better to compare to IADB or other works.

Author response:
Baselines: This is a valuable insight, which is expressed by the other reviewers as well. In the experiments described in the general comment with results in the supplementary rebuttal PDF (Sections A and C.2) we address this concern by comparing our approach to Rectified Flow, DPM-Solver++ and Consistency Models.

Reviewer comment (Novelty):
This paper is extensively based on Heitz et al. (2023), which is not a well-known paper yet. The proposed initialization scheme should be tested in other well-known diffusion schemes and that paper should be presented in a separate section.

Author response:
We address each point separately:

1. The target distribution is the distribution of the data.  

2. Heitz et al. (2023) is mentioned extensively in the Introduction and the Background (section 2.2 is devoted to it).  Building on and using results from published work is a standard practice in scientific writing. 


    Our paper is self contained up to the proof of Thm 4.1. Since the proof of Thm 4.1 is a slight variation of a proof in Heitz et al. 
   (2023) [this is the sentence "The only difference from Heitz et al. (2023) is the conditioning on C," that the reviewer mentions], we 
   felt that a proof sketch is sufficient. However, we can easily add a full proof to the appendix.


   This field moves fast. The other reviewers asked to **experimentally compare** with a paper from ICML 2023 (Consistency 
   Models). The Heitz et al. (2023) work is on arXiv since 2022, along with other similar work on deterministic diffusion (see 
   comment to reviewer 1n8o). Well versed readers should easily tackle the material in our paper.

3. Our contribution focuses on selecting the source distribution. For this investigation we devoted a whole section (Section 4) to illustrative experiments with graphics (and even videos in the supplementary material!). This was recognized, e.g., by reviewer HtfP. Our experimental results are state-of-the-art in diffusion policies.

Reviewer comment (Experiments):
The efficacy of deterministic deblending module is unclear. One can stack two separate DDIM modules by replacing the IADB with another DDIM. Probably such a cascade (yet incremental) trick could also yield good performance and model efficiency.

Author response:
Stacking DDIMs: assuming the suggestion is to run DDIM and use the result for initialization of another DDIM, we believe that this is not possible as DDIM expects a Gaussian source distribution and the output of the first DDIM would not be Gaussian. This is one of the reasons we use deterministic diffusion, which is a central component of our work.  
An alternative is to simply run DDIM for more steps - we compare with this baseline in Figure 8.  
If we misunderstood what the reviewer meant by stacked DDIMs, we kindly ask for clarification.

Reviewer comment (Experiments):
On BAIR and PHYRE, compare DDIM & DDPM & ASPeed that deblend previous history or Gaussian noise. It would be clearer about the choice of deblending history (see Weakness 3).

Author response:
BAIR and PHYRE: We thank the reviewer for this suggestion. We added the experiments on PHYRE in the supplemental rebuttal PDF (Section B.1). As mentioned in the general comment above, $DDIM_n$ Initialization in the video prediction domain is inferior to history-based initialization, and the initial approximation accuracy is important as there is a significant difference in the results of initializing with $DDIM_2$ or $DDIM_3$. We will add experiments on BAIR after the rebuttal as they take too long given our available compute resources.

Reviewer comment (Experiments):
On Robomimic and Push-T, compare to i) Consistency Model, and ii) two-stage DDIM, which resembles ASPeed but replaces the second-stage deterministic deblending module with another DDIM (see Weakness 1 and 2). If stacked DDIM works, probably this paper should be positioned as cascade diffusion models instead.

Author response:
Stacking DDIMs: assuming the suggestion is to run DDIM and use the result for initialization of another DDIM, we believe that this is not possible as DDIM expects a Gaussian source distribution and the output of the first DDIM would not be Gaussian. This is one of the reasons we use deterministic diffusion, which is a central component of our work.  
An alternative is to simply run DDIM for more steps - we compare with this baseline in Figure 8.  
If we misunderstood what the reviewer meant by stacked DDIMs, we kindly ask for clarification.

Robomimic and Push-T: We have added DPM-Solver++ and Consistency Models as baselines in the supplemental rebuttal PDF (Section C.2), which both perform worse compared to ASPeeD. We will update the relevant plots in the final version of the paper.

Reviewer comment (Experiments):
The results are not particularly surprising, particularly when the relatively weak baselines used for comparison are considered. As mentioned in 5.1, the DDIM sampling approach starts from Gaussian noise, while the proposed approach bootstraps from the perturbed history. The core idea to bootstrap the diffusion process from a good initial guess has been applied to accelerate diffusion processes previously (e.g., Lyu et al.), which may be a better baseline to consider.

Author response:
Baselines: Thank you for this suggestion. In the supplementary rebuttal PDF (Sections A and C.2) we address this concern by comparing our algorithm to Rectified Flow, DPM-Solver++ and Consistency Models, showing that our approach outperforms the latter two on robotic control tasks. As mentioned in the related work section, Lyu et al. [1] (as well as others) only focus on image generation, and have not attempted to extend their work to video prediction or robotic control. In particular, the method proposed by Lyu et al. requires pre-training a different (non-diffusion) generative model, which is avoided in our work by using the insight that deterministic diffusion does not require a Gaussian prior. To our understanding, the new baselines that we added are stronger than Lyu et al.’s ES-DDPM, but we can add an ES-DDPM baseline if the reviewer finds it pertinent. 

We emphasize that our approach can start with a distribution that is *not Gaussian*, which we exploit in the Diffusion Policy experiments. This is made possible by using deterministic diffusion. To our knowledge, this method of using deterministic diffusion to bootstrap from non-Gaussian source distributions is novel.

Reviewer comment (Experiments):
The comparisons in Fig 8 seem unfair, as the proposed approach takes 10 steps, but this requires an additional 3 steps for the blending step (please correct me if I am wrong). This seems to indicate that 10 DDPM steps is doing comparatively well on this task.

Author response:
Fig. 8: To clarify, ASPeeD takes a TOTAL of 10 steps. In the Push-T experiment, 2 DDIM steps and 8 deblending steps are taken, while in the Tool-Hang experiment 3 DDIM steps and 7 deblending steps are taken; in both cases the total is 10 which is a fair comparison with 10 DDPM or 10 DDIM steps. If this is unclear in the text, we can update the caption to better convey this point.

Reviewer comment (Experiments):
Fig 8 — Why is 100 DDPM worse than 10 DDPM in push-T/tool hang? How well does 30 DDIM/DDPM (I think this is equivalent to 10 Aspeed steps?) do in comparison to Aspeed?

Author response:
Regarding 10 DDPM outperforming 100 DDPM: Inference with 10 steps of DDPM was not attempted by the authors of Diffusion Policy; we conducted this additional experiment for a full fair comparison. We found these results slightly surprising as well. Therefore we’ve re-verified the results during the rebuttal period, and found that 100 DDPM outperforms 10 DDPM on Tool Hang, and performs similarly to 10 DDPM on Push-T.

Reviewer comment (Experiments):
Fig 10 — These ablations don't seem comprehensive enough to answer questions or support the claims. How dependent are these choices on the nature of the task?

Author response:
Ablations: We added ablation studies for the Push-T task in the supplementary rebuttal PDF (Section C.1), in addition to the ablation study on the Tool-Hang task reported in the paper. The DDIM ablation on Push-T shows that the reward is not very sensitive to the ratio of DDIM-deblending steps, unlike the MSE. The perturbation ablation on the Push-T verifies the observation we made based on the Tool-Hang ablations – that there is no benefit in adding gaussian perturbation to the DDIM approximation. 
Additionally, we’ve added ablations on the PHYRE dataset with $DDIM_n$ initialization in the supplementary rebuttal PDF (Section B.1). $DDIM_n$ Initialization in the video prediction domain is inferior to history-based initialization, and the initial approximation accuracy is important as there is a significant difference in the results of initializing with $DDIM_2$ or $DDIM_3$.

Reviewer comment (Novelty):
The iterative α-(de)blending method utilized in this research appears to bear a close resemblance to the rectified flow, as detailed in this [paper](https://arxiv.org/pdf/2209.03003.pdf). An important opportunity seems to have been missed by the authors in not acknowledging this significant work. A thorough review of related literature is crucial in any research endeavor to avoid overlooking key contributions in the field.

Author response:
We thank the reviewer for their suggestions and valuable insights.

Related Work:
Rectified flow is indeed a beautiful paper that we were not aware of. Interestingly, it appears that an essentially very similar idea was investigated independently by several authors (Iterative α-(de)blending [1], Rectified Flow [2], Cold Diffusion [3] and Direct Iteration [4]), with insufficient cross-citations among them, unfortunately. Thank you for pointing out this work and others.  We duly extended our related work section; please refer to the rebuttal PDF (Section D).


We emphasize, however, that our work is complementary to the line of work above - while all said papers investigate how to deterministically map different distributions, NONE OF THEM explore **the choice of the source distribution**, which is the main focus of our research. Our method builds on top of these papers while focusing on sequential settings. We demonstrate the fact that our method is independent of the blending algorithm by running our instructive example experiments (Section 4.2 of the paper) using Rectified Flow. See Section A of the supplementary rebuttal PDF for details.

Reviewer comment (Novelty):
The concept of selecting an appropriate source or prior distribution is indeed valuable. However, it's worth noting that a similar discussion has already been conducted in [PriorGrad](https://openreview.net/pdf?id=_BNiN4IjC5). Furthermore, other methods from VAE research such as [SVG-LP](https://arxiv.org/pdf/1802.07687.pdf) provide more straightforward approaches to learning a source distribution. It is essential to acknowledge these existing methods and differentiate the new work from them.

Author response:
We thank the reviewer for their suggestions and valuable insights.

We emphasize, however, that our work is complementary to the line of work above - while all said papers investigate how to deterministically map different distributions, NONE OF THEM explore **the choice of the source distribution**, which is the main focus of our research. Our method builds on top of these papers while focusing on sequential settings. We demonstrate the fact that our method is independent of the blending algorithm by running our instructive example experiments (Section 4.2 of the paper) using Rectified Flow. See Section A of the supplementary rebuttal PDF for details.

PriorGrad discusses the importance of the source distribution of the diffusion process in the audio prediction domain. In this work, the authors select the mean and variance of the Gaussian source distribution based on the data. Though they present a novel source distribution, it is still Gaussian and the algorithm is based on the standard diffusion process. In contrast, our approach is based on deterministic denoising models and can start the deblending process from non-Gaussian distributions, eg. $DDIM_n$ in the robotic control tasks, which achieves state-of-the-art results on this domain. Additionally, we focused on different domains - video prediction and robot control - for which it is not clear how to apply the PriorGrad source distribution.   
SVG-LP explored prior selection for video prediction based on VAE - it is not clear to us how this could be applied to our diffusion-based approach. 
Despite their dissimilarities from our approach, we’ve added both of these papers to the extended related work section; please refer to the supplementary rebuttal PDF (Section D).

Reviewer comment (Experiments):
In terms of video prediction, the study could benefit from the inclusion of at least one higher-resolution dataset. A resolution of 128x128 would be sufficient to provide a more challenging and realistic evaluation of the model's performance. This would enable a more robust assessment of the model's capabilities in real-world applications.

Author response:
We thank the reviewer for their suggestions and valuable insights.

128X128 Resolution video prediction results: We will assess how to extend our evaluation to these datasets - it is not trivial in terms of our available compute resources. Note that our results on the Diffusion Policy domains [5] are state-of-the-art, and evaluated on the most challenging tasks in the original paper.