Paper: 3D-GOI: 3D GAN Omni-Inversion for Multifaceted and Multi-object Editing (ICLR-2024)

Reviewer comment (Reproducibility):
The overall method still lies in the hybrid optimization method, which requires code tuning after the pre-trained encoder gives a coarse estimation.

Author response:
Due to the limitations of current segmentation models, achieving 100% accurate segmentation is challenging. Additionally, GIRAFFE exhibits issues with multiple attribute codes that are interdependent, relying solely on the encoder fails to obtain accurate codes. Our approach of involving more variety of codes lead to accumulated inaccurate reconstructions using only the encoder (as shown in **Appendix D.3 Figure 14**), necessitating additional optimization. Using only the optimizer significantly prolongs reconstruction time (approximately 10 minutes or more) and may fail to reconstruct images from multiple codes. Therefore, we currently consider this hybrid optimization structure as the optimal choice.

Reviewer comment (Evaluation):
Lacks qualitative comparison with existing encoder-based method such as E3DGE, only Tab. 1 shows the quantitative comparisions.

Author response:
Please find the comparison results in **Figure 14 in Appendix D.3.**

Reviewer comment (Experiments):
Also, any editing result based on manipulating the latent space, such as using InterfaceGAN?

Author response:
We want to thank the reviewer for bringing up this more granular editing. We will definitely consider it as an improvement strategy for enhancing the generation capabilities of GIRAFFE in the future.

Reviewer comment (Evaluation):
Since you allocate a single encoder to each code, what's the computational cost compared with existing encoder-based 3D GAN inversion framework such as E3DGE.

Author response:
Due to the inapplicability of existing methods to multi-object image editing, we conducted tests on the same task of synthesizing new viewpoints for faces. Compared to encoder-based methods, 3D-GOI with multiple encoders requires a total of 169M parameters and 165 GFLOPs, while E3DGE[1] has 90M parameters and 50 GFLOPs, and TriplaneNet[2] has 247M parameters and 112 GFLOPs. Inference times on a single A100 GPU are about 0.1 second for all three methods.  
|            | TriplaneNet (single encoder)| E3DGE(single encoder) | 3D-GOI(multi encoders) |
| ---------- | --------------------- | --------------------- | ---------------------- |
| parameters | 247M        | 90M                   | 169M                   |
| FLOPs      | 112G        | 50G                   | 165G   |

Reviewer comment (Theory):
I really wonder why having a single optimizer to optimize all the codes in Sec.3.4 will fail, since these codes should be independent to each other?

Author response:
As mentioned in **Appendix E**, we observed that in GIRAFFE, different object attribute codes are not entirely independent. For example, the shape code of an object can affect color control, and all attribute codes slightly impact the background as shown in **Figure 4**.   Failed cases in simultaneous optimization primarily manifest as poor background reconstruction and errors in object rotation, color, and shape features. To address this, we employ multiple optimizers to tackle these issues.

Reviewer comment (Evaluation):
In the comparison with encoder-based 3D GAN inversion methods, TriplaneNet performs slightly better to E3DGE over MSE and LPIPS, while slightly worse on ID loss, any intuition behind it?

Author response:
The MSE and LPIPS metrics assess pixel-level and perceptual losses for the entire image, including the background. On the other hand, the ID loss (utilizing pre-trained Arcface for human faces and ResNet-50 for cars) evaluates object attribute similarity. As shown in **Figure 14 in Appendix D.3**, E3DGE outperforms TriplaneNet in reversing object attributes, especially in multi-car scenarios, while sometimes exhibiting weaker background inversion.

Reviewer comment (Theory):
The Neural Inversion Encoder's inability to precisely predict codes independently underscores a fundamental limitation, necessitating the round-robin optimization approach and adding another layer of complexity.

Author response:
Thank you for this valuable feedback. Indeed, encoders that can capture precise codes  wouldn't require additional optimization steps. Unfortunately, all mainstream GAN inversion encoders fail to accurately predict all codes in settings with multiple objects and numerous codes, as shown in **Figure 14** . As we discussed in the limitations, we found that in GIRAFFE, the attribute codes of different objects are not entirely independent. For instance, the shape code of an object can influence its color, and all attribute codes slightly affect the background as shown in **Figure 4**. Therefore, as the number of objects (and hence codes) increases, their interrelationships become more complex. Meanwhile the accuracy of encoder predictions varies across different attributes; yet only with precise predictions for all attributes can we accurately reconstruct the image.  Therefore, using just the segmented images of objects seem to become a dead end for obtaining precise codes. As such, we propose to use the round-robin optimization algorithm to accurately determine the code values.

Reviewer comment (Evaluation):
The model's significant deviations in predicting background codes for multi-object scenes highlight challenges in accurately reconstructing detailed and complex backgrounds.

Author response:
In our tests on  unseen vehicle images, we found that 3D-GOI performs well in inversing and multifaceted editing vehicles but faces challenges in reconstructing the background, resulting in less ideal background reconstruction.  This is due to the difference between the background distributions of the dataset and the real-world. We plan to incorporate methods like optimizing the generator parameters on parallel, similar to PTI, in the future work.

Reviewer comment (Experiments):
With multiple dedicated encoders for fine-tuning specific attributes, there's a risk of the model fitting too closely to training data, resulting in poor generalization to unseen data. It would be valuable to see empirical evidence showcasing the model's performance on diverse, unseen data.

Author response:
In our tests on  unseen vehicle images, we found that 3D-GOI performs well in inversing and multifaceted editing vehicles but faces challenges in reconstructing the background, resulting in less ideal background reconstruction.  This is due to the difference between the background distributions of the dataset and the real-world. We plan to incorporate methods like optimizing the generator parameters on parallel, similar to PTI, in the future work.

Reviewer comment (Evaluation):
Differentiating between closely related attributes, such as scaling and translation, can be tricky. If the encoders aren't adequately designed to distinguish between such nuances, they might produce overlapping or redundant codes, hindering precise scene reconstruction and resulting in ambiguity when deciphering the attributes of a given scene. Visualizations of the latent space could shed light on potential overlaps and provide deeper insights into the latent representations.

Author response:
Thank you very much for your suggestion. Visualizing all the codes is challenging for us because codes for different attributes reside in separate latent spaces. Additionally, GIRAFFE couples them together using affine transformations and a NeRF neural network, making it much more complex than the typical GAN process of mapping from latent space to images. We will consider this as part of our future work.

Reviewer comment (Experiments):
The model's performance is highly contingent on accurate segmentation. If the segmentation phase is off even slightly, the downstream processes, including scene representation and attribute extraction, can be significantly affected. This sensitivity to initial processing stages may reduce resilience against imperfect inputs. It would be beneficial to see results or demonstrations of how the model performs with imperfect segmentation inputs, providing a more holistic view of its resilience.

Author response:
We tested an inaccurate segmentation model with an AP50 of 0.7 (where the current SOTA is 0.8 in InternImage-H) as shown in **Appendix D.6** and **Figure 19**. Despite the coarse codes being less accurate, our round robin optimization algorithm was still able to refine them effectively.

Reviewer comment (Novelty):
This paper still follows the framework of GIRAFFE by using its scene decomposition manner and training strategy, even though the authors claim that they have more object properties to encode.

Author response:
Our work fundamentally differs from GIRAFFE which generates images by initializing random codes as input, while our method takes multi-object scene images as input and map them back to the GIRAFFE code space. Therefore, although some parts of our method resemble GIRAFFE, there are fundamental mechanism differences.

Reviewer comment (Writing):
Sec 3.2, and Intro, there is no need to have so many texts to elaborate on why you use the segmentation method. It is pretty intuitive.

Author response:
Thank you for this suggestion. We will shorten the relevant text in the final version.

Reviewer comment (Reproducibility):
During training, you mentioned that you train an encoder for one code at a time, and keep the other codes at their true values (Sec 3.3). In Round-robin Optimization (Sec 3.4), you also mentioned that you prioritize optimizing the code with decreasing loss. If it does not decrease, then change to optimize the other codes. How do you combine the two training strategies?

Author response:
Our approach first trains the Neural Inversion Encoders (NIE) for each object. Afterwards, we input the coarse code generated by the frozen NIEs to the optimizer and use the round-robin optimization algorithm to refine them into accurate codes.

Reviewer comment (Presentation):
The paper should be written in a more concise way (see 3.). There are many important sections that should not be moved to the supplementary, e.g., related works, and qualitative comparisons with baselines.

Author response:
Thank you for this suggestion. We will streamline Section 3, including related work, more details of the encoder design, and the experimental part in the main text.

Reviewer comment (Evaluation):
Since the authors claim that they use an optimizer for each code, and the Round-robin Optimization method optimizes each code in turn, what is the time efficiency of this method compared with the baselines?

Author response:
Our method is very time-efficient compared to  methods that need optimization. For instance, to invert a single image, SPI[1] requires 550 seconds of optimization, PTI[2] requires 55 seconds and our 3D-GOI only needs 30s. This is primarily because, unlike other approaches, we only optimize the codes but not the generator parameters.

|                   | SPI  | PTI  | 3D-GOI |
| ----------------- | ------| ------ | -------- |
| optimization time | 550s | 55s  |   30s    |

Reference  
[1]Fei Yin, Yong Zhang, Xuan Wang, Tengfei Wang, Xiaoyu Li, Yuan Gong, Yanbo Fan, Xiaodong Cun, Ying Shan, Cengiz Oztireli, et al. 3d gan inversion with facial symmetry prior. arXiv preprint arXiv:2211.16927, 2022.

[2]Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based
editing of real images. ACM Transactions on Graphics (TOG), 42(1):1–13, 2022.