Paper: Image Translation as Diffusion Visual Programmers (ICLR-2024)

Reviewer comment (Experiments):
Dependency on Modules: DVP's systematic design implies that if one module falters, the entire system could be compromised. The robustness of such an interconnected system becomes a pertinent question. If one cog in the machine is faulty, can the system still deliver optimal results, or will it be critically hampered?

Author response:
**A1:** Thank you for the great insight. The apprehension regarding the dependency of our proposed DVP on its constituent modules and the potential impact on the system's robustness is a critical aspect of our design considerations. In DVP, we integrate GPT ($i.e.,$ as a planner) and various built-in modules, where the interconnected nature is indeed a defining characteristic. DVP is designed with a modular architecture, where each component, while important, does not singularly dictate the system's overall functionality. The modular design allows for a quick debugging during the process. This is critical in maintaining the system's operational integrity even when individual components face challenges. We have discussed the failure and success cases in Fig. 7. However, if a module is **fully compromised**, the system's performance will be affected.

Reviewer comment (Novelty):
Comparison with Other Techniques: Some results showcased by DVP can be achieved using established perception techniques like Segment Anything (SAM) or conditional control generation methods like ControlNet's point-to-point mode. A comparative discussion on the advantages of DVP over these techniques would provide clarity on DVP's unique value proposition.

Author response:
**A2:** Thank you for your insightful comments. We agree that established perception techniques like SAM generate promising results in various applications. These methods might not be directly comparable with DVP as they are not designed for image translation, and thus more suitable for replacing the current segmentation model ($i.e.,$ Mask2former [ref6]) for performance comparison. To understand how these models can further improve DVP, we conduct additional experiments by replacing our segmentation model with SAM. The preliminary results show that it indeed enhances the overall performance of DVP, indicating the flexibility of DVP for adopting established perception techniques. We’ve supplemented the additional experimental results in the Appendix Sec.H. Thank you for the great suggestion.

Reviewer comment (Experiments):
Textual Argument Handling: Even though DVP marries a condition-flexible diffusion model with the GPT architecture, its prowess in handling complex textual arguments requiring intricate reasoning remains questionable. The paper does not offer substantial experimental evidence to vouch for the enhanced capabilities of integrating GPT, leaving room for skepticism.

Author response:
**A3:** Thank you for the great suggestion. DVP integrates the power from GPT as a planner, which is renowned for its ability to process and analyze complex textual data, forming the backbone of our approach. We’ve provided more experiments in Appendix Sec.K, focusing on complex instructions involving spatial relations.

Reviewer comment (Evaluation):
Handling Complex Scenarios: DVP's recurrent challenge is its inability to adeptly manage intricate situations. As highlighted in the failure case study, the system falters in the face of occlusions, inadequate lighting conditions, and densely populated scenes. This limitation raises concerns about the system's applicability in real-world, diverse scenarios.

Author response:
**A4:** This is a great observation. Regarding the handling of complex scenarios, such as those involving occlusions, the modular architecture of DVP is designed to be both scalable and adaptable. The current limitations observed in the failure case study are indeed points of concern, but they also serve as catalysts for ongoing improvements. Due to the modular nature, each module within DVP can be continuously developed and updated. This advantage allows us to progressively enhance the performance of the whole system, even in challenging conditions. Also, we need to emphasize that poor translation results in complex scenarios is a common problem in the concurrent community. As stated in Appendix Sec.L and N, we highlight it as a future direction of our DVP such as reconstructing the occluded objects before editing from additional amodal segmentation models for a better translation result.

Reviewer comment (Experiments):
It seems the conditional-flexible diffusion model is the prompt-to-prompt method with proposed instance normalization. However, the authors only compare the results of using instance normalization and different guidance scales. It is also important to ablate the instance normalization in DVP to demonstrate instance normalization significantly outperforms a fixed scale when incorporating in-context visual programming.

Author response:
**A1:** Thank you for the great suggestion. We have conducted an additional ablation by comparing instance normalization in DVP with a fixed scale. Our results show that instance normalization significantly outperforms a fixed scale when incorporating in-context visual programming, indicating the effectiveness of instance normalization. We have also included the additional results and discussions in Appendix Sec.B Figure 11.

Reviewer comment (Reproducibility):
The proposed method is an aggregation of different models planned by an LLM. However, how the components (e.g., mask2former, Repaint, BLIP, position manipulator) are aggregated is not clearly clarified. The authors may explain various translation procedures by providing more step-wise illustrations like Fig. 7. Besides, the authors may provide the prompt they use for the GPT planner.

Author response:
**A2:** Thank you for the suggestion. We leverage the capabilities of GPT to create a sequence of operations. This planning process yields a sequence of distinct interpreters, such as SegmentInterpreter and EDITInterpreter. The method of translating is identical to that shown in Figure 7. For the prompts specified in the GPlan, we instruct GPT-4 to generate code that adheres to the same format employed for each individual pre-defined component. We’ve provided more clarification on how the components are aggregated in the revision.

Reviewer comment (Reproducibility):
Some important details are missing. For example, how the conv layer in instance normalization is parameterized? Is there any other difference between the conditional-flexible diffusion model and the prompt-to-prompt model, except for instance normalization? How to decide the order of operation sequences, automatically by GPT or manually by humans?

Author response:
**A3.1:** As outlined in our implementation, we employ the null-text optimization method [ref1] to tune the conv layer. This approach involves optimizing the convolutional layers and the unconditional embedding, initialized with a null-text embedding. This technique, as we have employed, ensures a balance between high-quality image reconstruction and the provision of intuitive editing capabilities. We’ve added more details in the revision.  

**A3.2:** Thanks for the insightful question. The process of obtaining these various sequences of operations is primarily facilitated by GPT-4, which acts as an automated planner. This automated generation often results in a diverse array of operation sequences, each potentially leading to distinct outcomes. Additionally, we can also force GPT-4 to follow a pre-defined order of operations by providing proper instruction.

In our current framework, we do not definitively conclude which plan is superior. Instead, GPT-4's role as an automated planner is to provide a range of reasonable and logically structured operation sequences. However, identifying the ‘’best’’ plan for the results remains an open question for our future exploration.

Reviewer comment (Reproducibility):
The authors use a conv layer in instance normalization (Eq. 5). But there is no evidence of how the conv layer is trained or parameterized. Can authors give more details?

Author response:
**A3.1:** As outlined in our implementation, we employ the null-text optimization method [ref1] to tune the conv layer. This approach involves optimizing the convolutional layers and the unconditional embedding, initialized with a null-text embedding. This technique, as we have employed, ensures a balance between high-quality image reconstruction and the provision of intuitive editing capabilities. We’ve added more details in the revision.

Reviewer comment (Reproducibility):
In Figure 7, we can see different plans derive different translated results. How do authors obtain different orders of operation sequences? How do authors decide which plan is the best for the final result?

Author response:
**A3.2:** Thanks for the insightful question. The process of obtaining these various sequences of operations is primarily facilitated by GPT-4, which acts as an automated planner. This automated generation often results in a diverse array of operation sequences, each potentially leading to distinct outcomes. Additionally, we can also force GPT-4 to follow a pre-defined order of operations by providing proper instruction.

In our current framework, we do not definitively conclude which plan is superior. Instead, GPT-4's role as an automated planner is to provide a range of reasonable and logically structured operation sequences. However, identifying the ‘’best’’ plan for the results remains an open question for our future exploration.

Reviewer comment (Evaluation):
In Fig. 8, are reconstructed results with automatic labels and human labels generated by null-text inversion? To my knowledge, null-text inversion produces quite similar reconstructed images to the original ones regardless of the input prompt. Can authors explain why using GPT-generated prompts derive better reconstructed results than human annotations?

Author response:
**A4:**  Sorry for the confusion. The reconstruction results shown in this figure use only 40% of the optimization steps. We aim to demonstrate the advantage of our method (fine-grained accurate annotation) over the baseline using a much shorter schedule. We acknowledge the necessity to revise the image description in our paper to clearly delineate this distinction (added in revision paper Sec. 4.3). Regarding the full null-text inversion process, it generally yields reconstructed images that closely match the originals, regardless of the input prompt, even when the prompts are entirely incorrect.

Reviewer comment (Experiments):
**Lack of Comparative Analysis**: The paper's strength in performance could be further substantiated by a more thorough comparative analysis. A comparison with alternative approaches like instructive tuning-based diffusion models, such as instruct2pix, would provide a clearer perspective on its relative merits.

Author response:
#### **Q2 Lack of Comparative Analysis:**

**A2:** Thank you for the great suggestion. We agree that additional comparison with instructive tuning-based diffusion models would provide a clearer understanding of the advantage of our model. We conduct a thorough comparative study with instruct2pix [ref3] as suggested. As shown in Fig. 16, our approach consistently yields high-quality performance due to its robust capacity for in-context reasoning. We have included these qualitative findings in Appendix Sec.I.

Reviewer comment (Experiments):
**Limited Experimentation on Instance Normalization**: The paper introduces instance normalization as a valuable element of the DVP framework, which can have broader applications. However, there is a need for more extensive experimentation to validate the advantages of instance normalization in the context of general diffusion-based image generation.

Author response:
#### **Q3 Instance normalization can have broader applications.**

**A3:** We appreciate the insightful suggestion. In our work, instance normalization is specifically coupled with the null-text optimization technique, which is tailored to suit the unique requirements of our approach and is not yet generalized for broader diffusion-based image generation applications. However, we do recognize that instance normalization harbors a considerable potential that remains largely untapped. Therefore, we conduct a preliminary study for the text-to-image task. Specifically, we initially generate an image following comprehensive instructions and then proceed with the fine-tuning process, eliminating the need for complete regeneration of the image from scratch. These preliminary results together with the discussions are supplemented in Appendix Sec.N and Fig. 20.

Reviewer comment (Experiments):
While the paper showcases DVP's impressive performance, there is a need for further exploration of its adaptability to a range of image translation tasks. These tasks might include object replacement, color/texture/material editing, and style transfer. Currently, the paper's experiments encompass a holistic approach without distinct analysis of how DVP caters to varying task requirements.

Author response:
#### **Q4 Adaptability to a range of image translation tasks:**

**A4:** We showcase several examples of object replacement in specific locations to highlight the robust in-context reasoning capability of the proposed DVP, addressing limitations observed in previous methods. It's important to note that DVP is a general framework for image translation, applicable to various task variants. To further demonstrate the adaptability of DVP, we provide additional results and discussions on style transfer (see Fig. 9 in the Appendix) and editing (refer to Figures 14, 16, 17, and 18 in the Appendix) as suggested.

Reviewer comment (Experiments):
The paper's reliance on a relatively small self-collected dataset comprising 100 image-text pairs may present limitations in terms of the breadth and diversity of data for evaluation.

Author response:
#### **Q5 The dataset is self-collected:**

**A5:** Due to the scarcity of publicly accessible datasets, we have introduced a new dataset featuring detailed textual descriptions. For quantitative evaluation, we adhere to established methods [ref1, ref4-5] and execute these on our newly developed dataset, detailed in Appendix Sec.E. To ensure a level playing field, we apply all methods uniformly across our dataset. Additionally, we plan to expand our data collection to encompass a broader diversity, and release our dataset.


[ref1] AnyDoor: Zero-shot Object-level Image Customization. ArXiv

[ref2] Inst-Inpaint: Instructing to Remove Objects with Diffusion Models. ArXiv

[ref3] InstructPix2Pix: Learning to Follow Image Editing Instructions. CVPR 2023

[ref4] Prompt-to-Prompt Image Editing with Cross Attention Control. ICLR 2023

[ref5] Text2LIVE: Text-Driven Layered Image and Video Editing. ECCV 2022

We appreciate your thoughtful comments. The above discussions are incorporated in our revised paper (in blue). We hope our response addresses your concerns. Please let us know if there are any additional questions, and we are happy to discuss further.

Reviewer comment (Experiments):
The proposed method is a systematic approach for image translation tasks incorporating different components. A potential drawback is its inference speed. It would be beneficial if the authors could compare inference speed with other image translation tasks.

Author response:
**A1:** Thank you for the great suggestion. The inference speed should be highlighted in the image translation task. We thus compare our proposed DVP with the baselines in the Table below.

| Methods  | Editing time |
| :-: | :-: |
| SDEdit|  ~ 10s |
| VQGAN-CLIP | ~ 1.5m |
| Text2live|  ~ 10m |
| VISPROG|  ~ 1.5m |
| **Ours**|  ~ 2m |

Related experiments and discussions are added in Appendix Sec.F.

Reviewer comment (Experiments):
The quantitative evaluation is only the proposed dataset, which contains fine-grained edit instructions. The effectiveness of DVP could be further proved by evaluating simple or even ambiguous instructions.

Author response:
**A3:** Thank you for the excellent suggestion. We conducted an additional ablation test on DVP under conditions of ambiguous instructions. Specifically, we limited the instructions for the GPT planner to a single word. The results are promising, showing that DVP is able to handle simple or ambiguous instructions. The details of these experiments are provided in the Appendix, Sec.J. We plan to conduct more experiments in this direction.

Reviewer comment (Reproducibility):
I have questions about the learning process of the 1×1 conv layer in equation (5). How is it exactly trained? And is it sensitive to the training sample size?

Author response:
**A4:** As stated in Sec. 2, our approach does not necessitate a full training phase. Instead, we include a pivotal optimization process ($i.e.,$ null-text optimization [ref1]) for each image during inference. More specifically, for each input image, our approach optimizes the 1x1 convolutional layer and the unconditional embedding concurrently, which initially starts from a null-text embedding. Therefore, it is not sensitive to the training sample size. Sorry for the confusion. We’ve supplemented the above discussion in Appendix Sec.A to make it more clear.

Reviewer comment (Experiments):
Will instance normalization also work in text-to-image tasks? It will be interesting to see if it could generate higher fidelity images with semantic meaning more aligned with the provided text prompts.

Author response:
**A5:** Thanks for the insightful question. The instance normalization is not directly applicable to text-to-image tasks, as it is integrated under null-text optimization [ref4], which requires pivotal optimization between the image and descriptive text before initiating the translation process. However, instance normalization can be effectively utilized in text-to-image tasks with additional fine-tuning of the produced images. We have elaborated on the prospective developments of instance normalization in Appendix Sec.N. We believe that instance normalization holds significant, yet largely unexplored potential, particularly in various applications.