Paper: Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation (ICLR-2024)

Reviewer comment (Theory):
The approach lacks a theoretical foundation, so it is not very intuitive to express why it can work.

Author response:
- Our method mainly builds upon the observed "diffusion phenomenon" of the pre-trained model as demonstrated in Fig. 2. This phenomenon is inspired by the nature that the diffusion model follows a stochastic Markovian chain between the data distribution and Gaussian distribution, characterized by a gradual entropy increase through Gaussian perturbations. In the reverse diffusion process, models focus on different levels of information at different phases, essentially embodying an extremely-noisy-to-semantic-formation-to-refine process.
    - The two main proposed manners for utilizing the above properties are "One-Way" control and the "Cyclic" construction.  By constantly replacing in the Semantic Formation Phase, We exert influence from the given visual condition to the rest of the generated image while preserving the content of the visual condition, thus achieving a one-way conditioning. However,  the pretrained model can easily fail to generate a pleasant result with a roughly spliced starting point. So we design a cyclic "disturb and reconstruct" process to recursively exploit the former generation progress under the visual condition, thus achieving better generation results. 
    - Hope this additional explanation can help solve your concerns.

Reviewer comment (Experiments):
The results of comparison methods are a bit too bad. More information about the setting should be given.

Author response:
- We follow the corresponding official codes to conduct all the comparison methods. For ControlNet, we directly use the official webui of Stable Diffusion. We have included codes of other methods in the updated supplementary materials. 
    - The unsatisfying performance of compared customization methods (DreamBooth, TI) on semantically complex conditions like human face has also been reported in prior works [a, b, c]. One major drawback of these methods is that they are not training-free and thus tend to easily overfit the limited finetune data and incorrectly entangle object identity and spatial information. Instead, our work is learning-free, leveraging the property of diffusion denoising itself for the customized generation.
    - The unsatisfying performance of SD-inpainting and ControlNet comes from the compulsively preserved visual condition, which may easily generate results with significant fragmentation (as shown in Fig. 1 of the main paper). Providing a small area as visual condition is a big challenge for these two methods. Meanwhile, these methods lack flexibility of fitting in multiple conditions, thus cannot be used for attribute editing or style transfer applications where the users use text prompts to slightly revise the visual conditions.

[a] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411, 2023.

[b] Yuheng Li, Haotian Liu, Yangming Wen, and Yong Jae Lee. Generate anything anywhere in any scene. arXiv preprint arXiv:2306.17154, 2023.

[c] Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning. arXiv preprint arXiv:2211.11337, 2022.

Reviewer comment (Reproducibility):
For the consideration of reproducibility, the code of the proposed method is suggested to be provided.

Author response:
The anonymous code has already been included in the Supplementary Meterial of the initial submission. We will make the code publicly released upon acceptance.

Reviewer comment (Experiments):
Are the comparison methods learning-free? If so, it would be beneficial to provide additional details about the experimental settings. Furthermore, for the comparison experiments, it might be advisable to incorporate some learning-free methods, such as LIVR [1], to ensure a comprehensive evaluation.

Author response:
- Among the compared four methods, only SD inpainting is learning-free. The other three (DreamBooth, TI, ControlNet) have to be finetuned before generation. For SD inpainting and our model, we input the given visual condition and text prompt at the same time to the model for direct inference. For DreamBooth and TI, we first use the given visual condition to update the model parameters or prompt tokens according to their default training settings, and then generate the target image using the text prompt and finetuned model. ControlNet first trains an extra module on edge images, and then generate the target image using the edge image of the given visual conditons.
    - ILVR [a] is essentially a visual conditioned method by utilizing classifier guidance to generate images closer to reference images on face model. However, this work only takes visual conditions as input and generates a random new face that is similar to the given face. Thus this work could not be mitigated to the text-visual customization task (TV2I), where we intend to generate a new image that not only contains the visual condition but also follows the text prompt guidance. The core of this task is to fit in both two visual/textual conditions to generate a harmonious result. As far as we know, we are the first and only learn-free TV2I work so far. We have added these discussions in the related work section of the revised main paper.

[a] Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. arXiv preprint arXiv:2108.02938, 2021.

Reviewer comment (Theory):
Sections 3.2 and 3.3 lack a theoretical foundation or pseudo-code to facilitate clearer understanding and reading.

Author response:
Thank you for your suggestion. We added a pseudo-code in Sec. A.5 of the revised paper to help clearer understanding.

Reviewer comment (Writing):
It is not clear in Section 3.3, paragraph 4. What is the difference between two steps/two ends and t-th diffusion step, and how to use them?

Author response:
- Two steps $x_{t_{1}}$, $x_{t_{2}}$ in paragraph 5 are identical to $x_{t-k}$, $x_{t}$ in paragraph 4. They refer to the starting point (high noise) and ending point (low noise) of our cycle process.  In experiments, we use the starting point t2 of 700-th, and the ending point t1 of 500-th. Since there is a large redundant in the cyclical process, we can apply the skip-step strategy of DDIM to speed up the "reconstruct" process by 100x, i.e., we only use three steps (700-th, 600-th, 500-th) for the "reconstruct" process.
    - After the reconstruction part, we further disturb the generation process by changing the noise level back to 700-th (can be viewed as returning back to the starting point of 700-th). A new cycle begins.
    - We have updated the corresponding description in the manuscript to make it clear.

Reviewer comment (Presentation):
The evaluation in Table 2 and in Section 4.2 is not matched.

Author response:
Thanks for pointing out the typo. The 'conditional consistency' in Table 2 and  'conditional correspondence' in Section 4.2 are the same thing.  We have corrected this typo and updated it in the new version.

Reviewer comment (Experiments):
Most of these examples are based on putting an object in bounding box to a large image by adding backgrounds. This setting has applications, however, whether this approach can be applied to whole image generation/editing instead of pasting object on a larger image.

Author response:
**A1:** Thanks for the valuable suggestion! We conduct experiments on the whole image generation/editing, achieving superior results. We have added those samples in the new appendix (Fig. 11 of the appendix).

Reviewer comment (Evaluation):
In the main body of this paper, the authors should present failure cases if it has, and analyze the reasons.

Author response:
**A2:** 
    - Thanks for the valuable suggestion. We have added the failure cases and the analysis in 13 of the revised paper.
    - We found the proposed method usually failed in two cases. First, there is a strong orientation mismatch between the visual condition face and the text prompt (e.g., back view text prompt but with front view face visual condition). Second, the model may fail when there is a clear text instruction of face coverings but the visual condition is a unobstructed face. In this case, the model may generate the covering at other random positions in the image.

Reviewer comment (Writing):
There are typos in the paper, e.g., t around eq.(1). Please check the whole paper.

Author response:
**A4:** Thanks for pointing it out! We have checked the whole paper and polished the writing again. The updated version has been uploaded in OpenReview.

Reviewer comment (Theory):
On page 4, the cycles are divided into three phases. Is there strict division boundary between these phases, and why these discussions are introduced to the main body of this paper, and how to support these conclusions on the existing of three phases.

Author response:
**A5:** 
    - There are no strict boundaries, but these phases can be properly traced to a certain range. The absence of a strict boundary is due to the nature of the diffusion model, which inherently simulates a gradual, evolving process (Diffusion model iteratively/gradually transforms data to approach the true underlying data distribution by introducing and removing noise at each step.)
    - The transition between distributions is achieved through Gaussian perturbations, reflecting a continuous process, from noisy chaos, to semantic formation, to refinement, which can be approximated by measuring the distance between distributions (more theoretical details can be found in [a] ).
    - Introducing guidance during the first phase (Chaos Phase) hinders its reflection in the final image due to the uncontrollable interference caused by excessive noise while introducing it too late during the third phase (Quality Boost Phase) does not allow for the desired high-level semantic modifications. These properties were well demonstrated in Sec. A.3 of Appendix and we can also see clear differences among the three phases in Fig. 15 and Fig. 17. 
    - We emphasize this part in the main paper, not only because it establishes the foundation of our methodological design, but also conveys a valuable message to the community: the learning-free controlling of generative output can be much more efficient than the current learning-based and step-wise paradigms. Specifically, current methods seek to integrate conditional guidance in a step-wise manner by adding conditional information at every diffusion step throughout the entire denoising process. However, we demonstrate that the above step-wise conditioning is not necessary to achieve fine-grained control of the generative output. Instead, operating on a critical range is sufficient (20% of the entire steps in this case).


[a] Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, and Yan Yan. Boundary guided mixing trajectory for semantic control with diffusion models. NeurIPS, 2023.

Reviewer comment (Experiments):
There are no quantitative analyses of the number of cycles, or positions of the start and end points. These experiments are important for us to understand the effectiveness of the proposed method.

Author response:
- We further conduct an ablation study on different hyper-parameters of COW. The table below provides quantitative analysis on the ablation of different settings of cycle numbers, the position of the starting and the ending points. We also included the results in Table 4 of the revised Appendix.
    - Here we use three quantitative metrics to analyze different models. CLIP-T is the CLIP-text cosine-similarity, which is the similarity of generated image and text condition calculated by the prestrained CLIP (ViT-B/32) model. To better evaluate the visual quality of the generated samples, we also perform a human evaluation on those ablation models with 10 volunteers. Each volunteer carefully went through 216 images generated by the six model variants, and rated the best model variant as 5 and the worst as 0. Here we report the averaged ratings for each model. Since different starting and ending points would involve different inversion steps, the time cost is slightly difference according to the diffusion inversion cost. 
    - The results show the model performance is sensitive to the starting and ending points, which confirms our motivation that we should repeatedly go through the Semantic Formation phase to have a controllable and directional generation. 

    | Cycle | Position &nbsp;&nbsp;&nbsp;           \|  | Clip-T$\uparrow$ | Human Rating$\uparrow$ | Time Cost$\downarrow$ |
    |:-----:|---------------------:|:----------------:|:----------------------:|:---------------------:|
    | 10    | 700$\Rightarrow$500 &nbsp;\| | 0.31             | 4.6                    | 5.21                  |
    | 10    | 900$\Rightarrow$700 &nbsp;\|   | 0.30             | 1.5                    | 5.52                  |
    | 10    | 800$\Rightarrow$400 &nbsp;\|   | 0.30             | 3.2                    | 7.96                  |
    | 10    | 500$\Rightarrow$300 &nbsp;\|   | 0.27             | 0.8                    | 4.85                  |
    | 5      | 700$\Rightarrow$500 &nbsp;\|   | 0.29             | 2.5                    | 3.32                  |
    | 20    | 700$\Rightarrow$500 &nbsp;\|   | 0.30             | 3.6                    | 8.83                  |

Reviewer comment (Evaluation):
According to Table 1, the proposed method is inferior to SD inpainting on both performance and efficiency. The only superiority of the proposed method is training free. However, since it needs cyclical diffusion & denoising, its inference cost is higher than SD inpainting. The superiority may be weakened.

Author response:
- Thanks for the comments. 
    - First, SD inpainting could not handle the cases where the textual condition conflict with the visual condition, both theoretically and empirically. Differently, as provided in the response to Q1 above, our model has the flexibility of fitting in different levels of changes applied to the visual condition, significantly broadening its range of applications in real-world scenarios.
    - Second, our quality is far better than SD inpainting, as evidenced by the human study evaluation (Table 2 of the original manuscript). Our model won 69.7% of "best condition consistency'' votes and 51.9% of "best general fidelity'' votes among five models, while SD inpainting only has 6.3% and 2.1% votes, respectively. This clearly shows our model performance is better than SD inpainting. More qualitative results can be seen in Fig. 12 of the appendix. 
    - Lastly, the cyclical diffusion and denoising are only conducted in part of the diffusion denoising process (20% of the whole process). In addition, since there is large redudant in the cyclical process, we can apply skip steps strategy of DDIM to speed up the cyclic "disturb'' and "construct'' process by 100x.  Thus the overall cost of our model is only slightly higher than the original SD inpainting (6s versus 5s).