Paper: Integrating View Conditions for Image Synthesis (ICLR-2024)

Reviewer comment (Novelty):
The scale of technical novelty is limited. The whole system looks like an ensemble of various existing approaches (e.g., GPT-4, Segment-Anything, Zero-123, and ControlNet), with any significant modification to them. While I do acknowledge the engineering effort that the authors have put into carefully selecting proper components and figuring out a reasonable way to compose them together, the novelty of the paper, from a technical perspective, is insufficient for ICLR.

Author response:
**Q**: Why integrate view conditions for image synthesis. What is technically unique and novel in it?

**A**: While drawing upon established methodologies, our unique innovation rests in the integration of view conditions for object synthesis. This novel approach presents a **distinct** and **pragmatic** solution to a prevalent challenge in image synthesis. The strength of our framework lies in its adaptability across diverse domains while requiring minimal training resources (e.g., 1 GPU for 6 hours), setting it apart from resource-intensive alternatives (e.g., a concurrent method that demands 8 GPUs for 144 hours). We underscore the novelty within the synthesis framework itself, rather than focusing solely on individual components.

Our system's innovation stems from the fusion of established components, specifically designed for viewpoint-guided image editing. The uniqueness lies in our methodological approach, a novel combination and application of these components, specifically tailored to tackle the complexities of viewpoint-based image synthesis. Similar to the best paper "Planning-oriented Autonomous Driving" showcased at CVPR 2023, our method extends and refines existing methodologies to effectively address the practical challenges in this domain.

Reviewer comment (Experiments):
The evaluation is incomplete. First, the results shown throughout the paper are all on indoor scenes. An experiment should be done to test the ability of the proposed method in handling outdoor images, which is not included in the current paper.

Author response:
**Q**: Incomplete evaluation.

**A**: We appreciate your feedback regarding the "Incomplete evaluation". In response, our revised version will feature additional experiments involving outdoor scenes, aiming to illustrate the versatility of our method. Besides, it's essential to note that our method primarily focuses on modifying specific objects under defined view conditions, irrespective of the background. Hence, the distinction between indoor and outdoor scenes may not significantly impact our core findings.

Furthermore, while we have outlined an evaluation of controllability in Sec. 4.3.3, we acknowledge the need for a more comprehensive assessment by human evaluation. To address this, we conduct a human evaluation where evaluators will judge the alignment of synthesized object poses with input conditions. The resulting scores from this evaluation will be incorporated into the revised manuscript to provide a more detailed and comprehensive evaluation of controllability.

| Methods         | Consistency ($\uparrow$) | Harmony ($\uparrow$) | Controllability ($\uparrow$) | Aesthetics ($\uparrow$) |
|-----------------|---------------------------|-----------------------|------------------------------|--------------------------|
| Paint-by-Example| 2.67                      | 2.61                  | 1.93                         | 4.92                     |
| Paint-by-Sketch | 2.79                      | 2.21                  | 1.87                         | 3.93                     |
| ViewControl (Ours) | **4.44**               | **4.54**              | **4.53**                     | **5.37**                 |

Reviewer comment (Evaluation):
The evaluation is incomplete. Second, only two out of the three dimensions including consistency, controllability and harmony are considered in the experiments, as shown in Table 2. The controllability of different methods should also be tested, e.g., through a human evaluation where subjects can be asked to tell how well the pose of the synthesized object aligns with that of the corresponding object in the source image.

Author response:
**Q**: Incomplete evaluation.

**A**: We appreciate your feedback regarding the "Incomplete evaluation". In response, our revised version will feature additional experiments involving outdoor scenes, aiming to illustrate the versatility of our method. Besides, it's essential to note that our method primarily focuses on modifying specific objects under defined view conditions, irrespective of the background. Hence, the distinction between indoor and outdoor scenes may not significantly impact our core findings.

Furthermore, while we have outlined an evaluation of controllability in Sec. 4.3.3, we acknowledge the need for a more comprehensive assessment by human evaluation. To address this, we conduct a human evaluation where evaluators will judge the alignment of synthesized object poses with input conditions. The resulting scores from this evaluation will be incorporated into the revised manuscript to provide a more detailed and comprehensive evaluation of controllability.

| Methods         | Consistency ($\uparrow$) | Harmony ($\uparrow$) | Controllability ($\uparrow$) | Aesthetics ($\uparrow$) |
|-----------------|---------------------------|-----------------------|------------------------------|--------------------------|
| Paint-by-Example| 2.67                      | 2.61                  | 1.93                         | 4.92                     |
| Paint-by-Sketch | 2.79                      | 2.21                  | 1.87                         | 3.93                     |
| ViewControl (Ours) | **4.44**               | **4.54**              | **4.53**                     | **5.37**                 |

Reviewer comment (Reproducibility):
In Fig.2, what is the personalized pre-trained diffusion model? What is the trainable “personalization” module and how is it trained?

Author response:
**Q**: Details about personalization module.

**A**: In Fig. 2, the personalized pre-trained diffusion model denotes a model fine-tuned to accommodate unique object characteristics, such as Dreambooth, LoRA, and their successive versions. The personalization module represents some trainable parameters, and it is optimized by images acquired throughout the pose synthesis process.

The specific method utilized within this personalization module is not explicitly detailed due to the rapidly development of this domain. The selection of method for personalization should be guided by practical applications, aiming to find an optimal balance between efficiency and performance. For example, if your want to preserve more details of the reference object, Dreambooth is a good choice. But if you need faster training speed, LoRA might be a better choice.

Reviewer comment (Experiments):
Can the system work well if there are more than one object to be replaced in the source image? For example, what about if the source image in Fig. 2 contains two laptops on the table?

Author response:
**Q**: Can the system work well if there are more than one object to be replaced in the source image?

**A**: Yes, this is equivalent to going through the process multiple times, processing only one object each time. And our system can sucessfully handle it, seen as **Appendix A.4 Additional Visualization Results** of the revised PDF. This will work well when the Segmentor module works well.

Reviewer comment (Experiments):
What dataset is used for the comparisons in Sec. 4.2?

Author response:
**Q**: What dataset is used for the comparisons in Sec. 4.2?

**A**: We used the dataset comprised of product images spanning various categories, gathered by our team from publicly available repositories, as described in Sec. 4.1 of our manuscript. It's important to note that we strictly maintain a clear distinction between the training set and the test set, ensuring that the data used in comparisons have not been part of the training process.

Reviewer comment (Novelty):
While the problem setting is fresh, the technique to address it lacks originality, relying heavily on previously established works without introducing significant innovative components or showing strong technical contributions.

Author response:
**Q**: Why integrate view conditions for image synthesis. What is technically unique and novel in it?

**A**: While drawing upon established methodologies, our unique innovation rests in the integration of view conditions for object synthesis. This novel approach presents a **distinct** and **pragmatic** solution to a prevalent challenge in image synthesis. The strength of our framework lies in its adaptability across diverse domains while requiring minimal training resources (e.g., 1 GPU for 6 hours), setting it apart from resource-intensive alternatives (e.g., a concurrent method that demands 8 GPUs for 144 hours). We underscore the novelty within the synthesis framework itself, rather than focusing solely on individual components.

Reviewer comment (Experiments):
The proposed pose estimation method, based on the relative camera rotation matrix and translation vector, is rudimentary. Its simplistic model might effect its generalization in a practical settings. For example in figure 1 and 3, all the reference photos are in roughly the same orientations (e.g. 30 degrees left forward facing) and the source objects are either in the same orientations with the reference or forward facing directions. The paper might not have sufficiently demonstrated the effectiveness of their method in diverse scenarios.

Author response:
**Q**: Similar orientations in figures 1 and 3.

**A**: The similar orientations in the reference images depicted in figures 1 and 3 can be attributed to their adherence to the default view predetermined within our pose estimation module. Acknowledging the necessity for a more varied representation, we include a broader array of examples showcasing with objects in diverse orientations, which is presented as Appendix 4 in the final section of the revised PDF now. Specifically, we estimate both the relative view information ($v1$) of the original object and the relative view information ($v2$) of the reference object. With the relative view information ($v3$) provided by user prompt, by computing $|v2-v1+v3|$, we derive the relative view information required to transform the reference object effectively into the view of the original object.

Regarding scale information, if the original objects and reference objects are of the same type, there are no issues. However, when dealing with different types of objects, our approach involves employing a series of traditional transformations to adjust the reference object to the mask area. Consequently, there might be some challenges in open domain data. We leave it for future work, by providing a solution where users can manually scale objects through the graphical user interface (GUI).

Reviewer comment (Experiments):
Considering the similar orientations in figures 1 and 3, how does the method perform with objects in diverse orientations? Would it be possible to provide more challenging examples or demonstrate the same object transformed under various orientations and scales?

Author response:
**Q**: Similar orientations in figures 1 and 3.

**A**: The similar orientations in the reference images depicted in figures 1 and 3 can be attributed to their adherence to the default view predetermined within our pose estimation module. Acknowledging the necessity for a more varied representation, we include a broader array of examples showcasing with objects in diverse orientations, which is presented as Appendix 4 in the final section of the revised PDF now. Specifically, we estimate both the relative view information ($v1$) of the original object and the relative view information ($v2$) of the reference object. With the relative view information ($v3$) provided by user prompt, by computing $|v2-v1+v3|$, we derive the relative view information required to transform the reference object effectively into the view of the original object.

Regarding scale information, if the original objects and reference objects are of the same type, there are no issues. However, when dealing with different types of objects, our approach involves employing a series of traditional transformations to adjust the reference object to the mask area. Consequently, there might be some challenges in open domain data. We leave it for future work, by providing a solution where users can manually scale objects through the graphical user interface (GUI).

Reviewer comment (Evaluation):
The performance of pose estimation shows a RMSE value of 9.7, even in the best scenario as shown in table 3, seems relatively high. Is there room for improvement?

Author response:
**Q**: Details about table 3.

**A**: We appreciate your feedback and have updated Table 3 as follows with more experiments with more suitable hyperparameters to improve accuracy. 

| Methods   | \#Params | GFLOPs | MAE ($\downarrow$) | RMSE ($\downarrow$) |
|-----------|----------|--------|---------------------|---------------------|
| ResNet-50 | 26.20 M  | 4.13 G | 4.31                | 7.45                |
| CLIP      | 87.88 M  | 4.37 G | 3.28                | 10.59               |
| ViT       | 86.34 M  | 16.86 G| 1.65                | 6.56                |
| DINO-v2   | 85.61 M  | 21.96 G| **0.80**            | **5.01**            |

Reviewer comment (Evaluation):
In section 'EFFECTS OF VIEW CONDITIONS', suggesting that predictions are marginally affected within a 20-degree range, is perplexing. This statement seems to undermine the paper's central argument, suggesting that pose estimation and view conditions may not be as critical or even necessary. Could the authors elaborate on this?

Author response:
**Q**: Details about section "Effects of View Conditions". 

**A**: We apologize for any confusion arising from our previous explanation regarding the "Effects of View Conditions". Allow us to clarify further. Our initial intent was to highlight the robustness of our predictions within a reasonable variance. Nevertheless, it is crucial to claim the importance of precise pose estimation and accurate view conditions in achieving realistic object synthesis.

The slight column indicates a level of robustness in our predictions because in these tests, they can combine the semantic information of the background and the pose information of the reference object to comprehensively determine the generated pose information. But in many cases, such as empty outdoor scenes, there is no clear background semantic information to guide the pose of generated objects. Besides, when comparing these slight variations to the perfect column, it becomes evident how crucial precise pose estimation and accurate view conditions are in achieving a more accurate outcome in object synthesis. Or in other words, robustness determines the lower bound of the generated effect, and effectiveness determines the upper bound of the generated effect.

Reviewer comment (Presentation):
I think the title of this paper does not really satisfy its content. "View-conditioned image synthesis" normally refers to controlling the camera view of the entire image, but the proposed method actually sounds more like editing local regions with view conditions. It is more like a system design and the current title does not fit it well.

Author response:
Thank you for your insightful comments. We will appropriately revise the title and add additional visualization results, which is presented as **Appendix A.4 Additional Visualization Results** of the revised PDF now.

**Q**: Why integrate view conditions for image synthesis?

**A**: The integration of view conditions in image synthesis addresses one limitation in current image editing methods, which often lack precise control over view-related information. It's essential to highlight that, similar to text-guided and image-guided image editing techniques, our approach specifically applies view conditions to the object of interest rather than the entire image. This utilization of view conditions significantly enhances controllability. The title, "View-conditioned Image Synthesis," aligns well with this approach, and we elaborate its significance in the Abstract.

Inspired by diverse text-guided and image-guided regional image editing methodologies, we formulated this title. Your constructive feedback is immensely appreciated, and **we intend to incorporate your suggestions**. Here are some refined title options:

"Integrating Object View Conditions for Image Synthesis"

"Integrating View Conditions for Regional Image Synthesis"

"Integrating View Conditions for Image Editing"

Reviewer comment (Reproducibility):
To put the new object in the original scenes with the same 6D pose, the authors proposed to use a pose estimation module. Does this pipeline require the new object to be presented in the canonical view? It seems that the pose estimator only produces relative camera parameters.

Author response:
**Q**: Provide more details about the pose estimation module.

**A**: In general, without the view estimation of reference object, our pipeline needs the presentation of the new object in a canonical view. However, we have devised strategies to achieve this without relying explicitly on a canonical view. For instance, we estimate both the relative view information ($v1$) of the original object and the relative view information ($v2$) of the reference object. With the relative view information (v3) provided by user prompt, by computing $|v2-v1+v3|$, we derive the relative view information required to transform the reference object effectively into the view of the original object.

Reviewer comment (Experiments):
To put the new object in the original scenes with the same 6D pose, the authors proposed to use a pose estimation module. Why does the generated image in the presented results always follow the 6D pose of input images? What if we choose a different pose? Will it fail the framework?

Author response:
The images generated in the presented results align with both the 6D pose of input images and accompanying text descriptions. While we have showcased only a selection of examples without text descriptions, our revised version will include more diverse instances, such as "Replace A with B, and turn left 90 degrees." Generally, when the new pose harmonizes with the background, the resulting effect tends to be favorable.

Reviewer comment (Experiments):
To put the new object in the original scenes with the same 6D pose, the authors proposed to use a pose estimation module. In most results, the newly added object and the existing object are from the same category. What about different categories?

Author response:
In scenarios where the newly added object and the existing object do not belong to the same category, it still works well. Nevertheless, in practical applications, replacing objects across different categories is rare. Our system is tailored to address realistic scenarios, ensuring, for example, that a shoe would not inadvertently substitute for a hat. Despite this, our revised version will include more scenarios where where the newly added object and the existing object do not belong to the same category. 

We highly value your feedback and aim to address these concerns comprehensively in the revised manuscript by incorporating additional examples and explanations. **We have included some visualization results in Appendix 4 of the revised PDF, and we encourage you to review these to address any remaining concerns you might have**.