Paper: InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image (ICLR-2024)

Reviewer comment (Experiments):
The results shown in the paper lack race diversity. There are almost no Asian or black people. I'm worried whether the proposed method does not perform well on those cases.

Author response:
Thank you for raising this important point. We acknowledge the importance of diversity in the evaluation of image editing methods and strive to provide a well-rounded assessment. **Our models are indeed capable of handling different races, hairstyles, ages, and other attributes.**

As described in Appendix A.1.1 Experimental Settings, our model is trained on the FFHQ dataset, which includes a diverse set of faces, and evaluated on the CelebA-HQ dataset. This diverse training data ensures that our model can generalize to various races and attributes.

To address your concern, we have included editing results for both Asian and black people in Figures 14 and 15. These additional examples show the diversity of our method in handling different races.

Reviewer comment (Evaluation):
The identity may change after applying the proposed method. For example, in Fig. 1 first example, the eye shape changed after the beard was removed. In Fig. 3 middle example, the girl seems to look more Asian and the nose shape changed after editing. These are not analyzed in the limitation section.

Author response:
Thank you for your careful observation. Identity preservation and attribute disentanglement have posed challenges for face editing. 3D-aware editing becomes more difficult than 2D editing in this regard since only one input image is visible. 

The ID score for 2D editing(e4e, HFGI) is about 0.8, and for structured 3D-aware editing(PREIM3D, IDE-3D) is 0.5~0.6. We have added quantitative results of InstructPix2Pix and img2img on 30 attribute editings, highlighted in red, in Table 5. **Tables 2, 3, and 5 show that our ID score is above 0.6, higher than the previous work. Furthermore, the results of our user study (Table 6) support the notion that our method achieves superior identity preservation and attribute disentanglement.**

We also recognize that there is room for improvement in preserving finer details such as eye shape and eyelashes, and we acknowledge these as areas for future research in this domain.

Moreover, note that the ID score reported in our paper is multi-view identity consistency, which is calculated between the novel views after editing and the input view.
Like EG3D, the state-of-the-art 3D face generation method, we also uniformly rendered novel views from yaw angles between $[-30^{\\circ},30^{\\circ}]$ and pitch angles between $[-20^{\\circ},20^{\\circ}]$ for an input image.
EG3D reported a score of 0.77 for identity similarity in synthetic faces in their paper.
Although we are working with real-world faces, not synthetic ones, and with editing, we still achieve a score of 0.6 or above.
**To the best of our knowledge, our method is the current state-of-the-art in preserving identity in 3D-aware face editing.
Therefore, We highlight our ability to preserve identity after editing in different poses.**

Reviewer comment (Experiments):
The proposed method adopts two streams of inputs (real and edited images). However, the ablation study does not show the necessity of them. Will only one stream work?

Author response:
We appreciate your interest in the necessity of the two streams of inputs (triplet data mechanism) in our method.  
The triplet data mechanism plays a crucial role in achieving accurate and disentangled image editing. To provide a more thorough understanding of its importance, we conducted an ablation study and presented the results below.

Our comparison involves the img2img model, which can be considered as using a text-image pairing data mechanism, in contrast to our method which utilizes the triplet data mechanism. Img2img is trained on the instructions and edited images in our prepared dataset.Unlike InstructPix2NeRF, img2img has no paired images, but the rest of the network structure is same. 

**The results of the ablation study, summarized in Table R3.2, demonstrate that the triplet data mechanism significantly contributes to the quality of editing in terms of identity preservation (ID) and attribute dependency (AD) when attribute altering (AA) is close to equal.**

- **ID Score**: Our method consistently outperforms img2img in preserving identity across various attributes, as indicated by the higher ID scores.

- **AA Score**: Our method achieves an editing effect that is close to or superior to that of img2img.
  
- **AD Score**: The triplet data mechanism helps reduce attribute dependency, ensuring that changes to one attribute do not excessively affect others.

**These results highlight that the triplet data mechanism encourages the model to learn the correlations between changes in pairs of images and the corresponding instructions, leading to more precise and disentangled editing.** In conclusion, the triplet data mechanism is essential for achieving high-quality image editing results.

Table R3.2. The triplet data mechanism ablation. Identity consistency (ID)  measures if it is the same person. Attribute altering (AA) measures the change of the desired attribute. Attribute dependency (AD) measures the change in other attributes when modifying one attribute.
|Method|bang|eyeglasses|smile|multiple instructions|five o’clock shadow |bushy eyebrows|chubby|double chin|high cheekbones|pale skin|goatee|
|:--------|:-------:|:-------:|:------:|:------:|:-------:|:-------:|:------:|:------:|:------:|:------:|:------:|
|img2img-ID$\\uparrow$ |0.40|0.42|0.46|0.37|0.52|0.53|0.50|0.51|0.51|52|0.50|
|ours-ID$\\uparrow$    |**0.56**|**0.59**|**0.60**|**0.55**|**0.59**|**0.64**|**0.64**|**0.64**|**0.64**|**0.66**|**0.65**|
|img2img-AA$\\uparrow$ |0.99|3.33|1.47|1.39|0.94|**1.02**|**1.05**|0.96|0.96|1.03|1.12|
|ours-AA $\\uparrow$  |**1.05**|**3.37**|**1.50**|**1.50**|**0.95**|1.00|**1.05**|**1.00**|**1.07**|**1.11**|**1.21**|
|img2img-AD$\\downarrow$|0.61|0.79|0.76|0.88|0.77|0.62|0.65|0.76|0.62|0.53|0.57|
|ours-AD$\\downarrow$   |**0.53**|**0.64**|**0.61**|**0.69**|**0.74**|**0.51**|**0.54**|**0.63**|**0.54**|**0.47**|**0.50**|

Reviewer comment (Writing):
The methods section could be written better, with a clear exposition of losses during training and the forward pass during inference. To that end, Fig 2 should be expanded to include both training and inference settings.

Author response:
We apologize for any confusion regarding our methods section. We have taken your feedback into account and have made improvements to clarify the total training loss and the forward pass during inference.

In Section 4.3, we have included a clear exposition of the total loss used in our method, which consists of the diffusion loss $\mathcal{L}\_{diff}$ and the identity loss $\mathcal{L}\_{ID}$. The total loss is expressed as:
$$
\mathcal{L} = \mathcal{L}\_{diff} + \lambda\_{id} \mathcal{L}\_{ID}
$$
where $\lambda\_{id}$ is set to 0.1 in our experiments. This addition aims to provide a more comprehensive understanding of our training process.

**In Section 4.4, we illustrate the inference process in detail.** Specifically, we add a little bit of noise to the latent of the input image (usually 15 steps) to obtain $w\_{ot}$ and then use our model to perform conditional denoising. The model predicts three score estimates, the image-text conditional $\epsilon\_{\theta}(w\_{ot}, c\_{I}, c\_{T})$, the only-image conditional $\epsilon\_{\theta}(w\_{ot}, c\_{I}, \emptyset)$, and the unconditional $\epsilon\_{\theta}(w\_{ot}, \emptyset, \emptyset)$.
$c\_{T}=\emptyset$ indicates that the text takes an empty character. $c\_{I}=\emptyset$ means that the concatenation $w\_{o}$ takes zero and identity modulation takes zero. 
Image and text conditioning sampling can be performed as follows:
$$
  \tilde{\epsilon}\_{\theta}(w\_{ot}, c\_{I}, c\_{T})=\epsilon\_{\theta}(w\_{ot}, \emptyset, \emptyset) + s\_{I}(\epsilon\_{\theta}(w\_{ot}, c\_{I}, \emptyset) - \epsilon\_{\theta}(w\_{ot}, \emptyset, \emptyset)) + \\\\ s\_{T}(\epsilon\_{\theta}(w\_{ot}, c\_{I}, c\_{T}) - \epsilon\_{\theta}(w\_{ot}, c\_{I}, \emptyset))
$$
where $s\_{I}$ and $s\_{T}$ are the guidance scales for alignment with the input image and the text instruction, respectively.
  
To further enhance clarity, **we have also modified Figure 2 to include a switch, distinguishing between the training and inference processes.** This change helps illustrate the transition from training to inference and highlights the specific steps involved.

Reviewer comment (Evaluation):
While the identity consistency is better preserved than prior work, there still remains an identity drift during editing.

Author response:
Thank you for acknowledging our work. Identity preservation and attribute disentanglement have posed challenges for face editing. 3D-aware editing becomes more difficult than 2D editing in this regard since only one input image is visible. 

The ID score for 2D editing(e4e, HFGI) is about 0.8, and for structured 3D-aware editing(PREIM3D, IDE-3D) is 0.5~0.6. We have added quantitative results of InstructPix2Pix and img2img on 30 attribute editings, highlighted in red, in Table 5. **Tables 2, 3, and 5 show that our ID score is above 0.6, higher than the previous work. Furthermore, the results of our user study (Table 6) support the notion that our method achieves superior identity preservation and attribute disentanglement.**

We also recognize that there is room for improvement in preserving finer details such as eye shape and eyelashes, and we acknowledge these as areas for future research in this domain.

Moreover, note that the ID score reported in our paper is multi-view identity consistency, which is calculated between the novel views after editing and the input view.
Like EG3D, the state-of-the-art 3D face generation method, we also uniformly rendered novel views from yaw angles between $[-30^{\circ},30^{\circ}]$ and pitch angles between $[-20^{\circ},20^{\circ}]$ for an input image.
EG3D reported a score of 0.77 for identity similarity in synthetic faces in their paper.
Although we are working with real-world faces, not synthetic ones, and with editing, we still achieve a score of 0.6 or above.
**To the best of our knowledge, our method is the current state-of-the-art in preserving identity in 3D-aware face editing.
Therefore, We highlight our ability to preserve identity after editing in different poses.**

Reviewer comment (Experiments):
Instead of an Encoder, if direct optimization of the W+ vector was used (assuming much larger compute), would it preserve the identity better? What if this is only done during inference and not training?

Author response:
Your question is indeed intriguing, and we appreciate your curiosity regarding the potential benefits of direct optimization of the $\mathcal{W+}$ vector for identity preservation, especially if conducted during inference. We conducted experiments to explore this scenario and provide insights into its causes.

**In our experiments, we considered two configurations: Direct $\mathcal{W+}$ optimization and PTI optimization.** Direct $\mathcal{W+}$ optimization involves optimizing the $\mathcal{W+}$ vector while keeping the generator fixed. PTI (Pivotal Tuning Inversion) technique fine-tunes the generator based on the initial value provided by direct optimization. We conducted 500 steps of optimization on the $\mathcal{W+}$ vector, and PTI added 100 steps of fine-tuning the generator.

The results of these experiments are presented in Figure 10, where we compare the outcomes of direct $\mathcal{W+}$ optimization, PTI, and the encoder-based method. The results show that directly replacing the encoder with an optimization method during inference will lead to a severe decrease in both editing effect and identity consistency.

**We attribute this issue to the deviation between the model and data distribution.** The model learns a conditional distribution within the encoder's inversion space during training. When the encoder is replaced by an optimization method during inference, the data distribution used for inference mismatches the learned model distribution. This mismatch results in greater identity drift and undesirable editing outcomes.

While conducting $\mathcal{W+}$ optimization during training (much larger compute) could potentially address the distribution deviation problem, it may introduce artifacts in novel views, as pointed out by PREIM3D. This is due to optimization being performed on a single image during training.

In summary, while direct optimization of the $\mathcal{W+}$ vector is an interesting concept, our experiments suggest that it may not necessarily lead to improved identity preservation and editing results compared to the encoder-based approach.

Reviewer comment (Evaluation):
Through the visualization in Figure 1, I find that the original identity and RGB image attributes are not well preserved. Large differences can still be observed in the areas that are not supposed to be edited.

Author response:
Thanks. Identity preservation and attribute disentanglement have posed challenges for face editing. 3D-aware editing becomes more difficult than 2D editing in this regard since only one input image is visible. 

The ID score for 2D editing(e4e, HFGI) is about 0.8, and for structured 3D-aware editing(PREIM3D, IDE-3D) is 0.5~0.6. We have added quantitative results of InstructPix2Pix and img2img on 30 attribute editings, highlighted in red, in Table 5. **Tables 2, 3, and 5 show that our ID score is above 0.6, higher than the previous work. Additionally, the results of our user study (Table 6) support the notion that our method achieves superior identity preservation and attribute disentanglement.**

Furthermore, we introduced attribute dependency (AD) score to measure the change in other attributes when editing certain attributes. **Figure 3 shows that our method preserves the other attributes better when meeting the editing requirements.** For example, in the third row, first column of Figure 3, InstructPix2Pix alters the expression when adding bangs, while our method maintains expression consistency. InstructPix2Pix and img2img often lead to undesirable changes in skin color. 
We acknowledge that there are some limitations in preserving fine details such as eye shape and eyelashes, and we consider these areas for future improvement.

Moreover, note that the ID score reported in our paper is multi-view identity consistency, which is calculated between the novel views after editing and the input view.
Like EG3D, the state-of-the-art 3D face generation method, we also uniformly rendered novel views from yaw angles between $[-30^{\\circ},30^{\\circ}]$ and pitch angles between $[-20^{\\circ},20^{\\circ}]$ for an input image.
EG3D reported a score of 0.77 for identity similarity in synthetic faces in their paper.
Although we are working with real-world faces, not synthetic ones, and with editing, we still achieve a score of 0.6 or above.
**To the best of our knowledge, our method is the current state-of-the-art in preserving identity in 3D-aware face editing.
Therefore, We highlight our ability to preserve identity after editing in different poses.**

Reviewer comment (Experiments):
Qualitative comparisons. (1) The proposed method seems to struggle with expression editing, e.g., it fails to make the head smiling; The instruct-pix2pix model doesn't encounter this problem; (2) Regarding the "bangs" example, I would prefer the instruct-pix2pix as it contains real bangs; (3) There is no comparisons with Instruct-NeRF2NeRF, AvatarStudio, and HeadSculpt, considering they are more similar works than the compared Talk-to-Edit and img2img; (4) More examples and more scenarios will largely improve the validation. Currently, there are only three types presented.

Author response:
Regarding expression editing, it's important to note that our method allows for controlled adjustments of attributes, including expressions. In the case you mentioned, the head does exhibit a smile, albeit to a lesser degree. The degree of the smile can be controlled through the guidance scale, which also affects identity preservation and attribute dependency. 
**As shown in Tables 2, 3, and 5, our method has better ID preservation and less impact on other attributes when obtaining the same editing degree.**
The ID score of about 0.49 and AD score of above 0.7 indicate that InstructPix2Pix is prone to identity drift and other attribute changes when performing editing. The results of the user study (Table 6) also illustrate this point. For example, in the third row, first column of Figure 3, InstructPix2Pix alters the expression when adding bangs, while our method maintains expression consistency.

For smile editing, you can refer to Figure 11 (first row) and Figure 14 (fourth row) in our paper.
As for the "bangs" example, you can refer to Figure 15 (third row) and Figure 18 (fourth column). It is true that different methods may have their strengths in specific scenarios, and user preferences may vary. We appreciate your input and understand the appeal of InstructPix2Pix's real bangs. Our method aims to provide versatility for a wide range of editing tasks, particularly those requiring attribute preservation and control.

> *W2 (3): There is no comparisons with Instruct-NeRF2NeRF, AvatarStudio, and HeadSculpt.*

**A**: We appreciate your awareness of these methods and discuss and cite these methods in Section 2. **However, it's crucial to note that these methods have different tasks from our method and cannot handle 3D-aware editing from a single image. While these methods take tens of minutes to optimize a single scene, our method generates 3D-aware editing in seconds for different faces.**
The input of our method is a single image and user instruction, and the output is edited 3D-aware images and geometry.

- **Instruct-NeRF2NeRF works with reconstructed NeRF scenes, taking as input a set of captured images, their camera poses, and calibration data, while our method operates on a single image with user instructions.**
Instruct-NeRF2NeRF uses InstructPix2Pix to iteratively edit the set of captured images while optimizing the underlying scene. 

- **AvatarStudio utilizes dynamic full-head avatars reconstructed from head videos, which is distinct from our single-image and instruction-based input.**
AvatarStudio uses their proposed view-and-time-aware Score Distillation Sampling (VT-SDS) with a personalized diffusion model to edit the dynamic NeRF.

- HeadSculpt primarily focuses on coarse-to-fine text-to-3D generation and can achieve editing in the fine stage by blending scores predicted by ControlNet-based InstructPix2Pix and landmark-based ControlNet for the editing instruction and the original description. **However, HeadSculpt is only capable of editing the trained scene model generated in the coarse stage, rather than a single input image.** 

Due to these differences in tasks, inputs, and methodologies, a direct comparison with these methods may not provide meaningful insights for evaluating our work. We hope this clarification helps to address your concern.

> *W2 (4): More examples and more scenarios will largely improve the validation.*

**A**: Thank you for the suggestion. We have taken steps to enhance the validation of our work by including more examples and scenarios. In our previous version, we provided examples in Appendix A.3, covering attributes such as beards, hair color, vampires, zombies, old people, children, long hair, etc. in Figures 12 and 13.
Per your suggestion, **we have further expanded our results to include novel characters from movies (Figure 16), Asians (Figure 14 and 15), and black men and women (Figure 14 and 15).** We believe that these additional results strengthen the breadth and depth of our validation, showing the diversity of our method.

Reviewer comment (Evaluation):
Quantitative comparisons. (1) The evaluations are not comprehensive. Still, only three examples are presented; (2) More quantitative evaluations, e.g., user studies, would be beneficial.

Author response:
We appreciate your feedback on the comprehensiveness of our evaluations. To address your concern, we would like to clarify that we indeed provide a comprehensive evaluation of our method, which includes a broader set of attributes beyond the three examples presented in the main text.

In our previous version, we offered quantitative results for a total of 30 attribute editings of our method, as detailed in Table 5. These results provide a comprehensive assessment of our method's performance across a wide range of attributes. per your suggestion, **we have added quantitative results of InstructPix2Pix and img2img on 30 attribute editings, highlighted in red, in Table 5.**
The results reveal that when the attribute editing degree AA is comparable to or surpasses the baseline, our method exhibits remarkable enhancement in ID and AD metrics.

> *W3 (2): More quantitative evaluations, e.g., user studies.*

**A**: We apologize for any confusion regarding the presentation of quantitative evaluations, particularly the user study. **In our previous version, we conducted a user study to comprehensively evaluate our method's performance. You can find the user study results in Appendix A.2.4 (Table 6), which provides valuable insights into the quality of our edits.**

The user study involved 1,440 votes from 30 volunteers who evaluated the text instruction correspondence and multi-view identity consistency of editing results. Participants were asked to choose the better result between our method and baseline for various attributes and multiple instructions. The results, presented in Table 6, clearly demonstrate that our method outperforms the baselines, further substantiating the effectiveness of our approach.

We appreciate your feedback and have clarified the location of the user study within the paper for your reference. We copy Table 6 here.

Table 6. The result of our user study. The value represents the rate of Ours > others. Multiple instructions indicate editing with the combinations of the above three attributes.
|Method|bang|eyeglasses|smile|multiple instructions|
|:--------|:-------:|:-------:|:------:|:------:|
|Talk-to-Edit|0.742|0.958|0.817|0.875|
|InstructPix2Pix|0.833|0.667|0.725|0.683|
|img2img|0.733|0.758|0.750|0.783|

Reviewer comment (Experiments):
There lack the reason for generating and using 20-30 instruction prompts for one single paired image. Will the number of instructions affect the training?

Author response:
Your question is indeed valuable, and we appreciate your curiosity regarding the impact of the number of instructions on our training process. Up until now, our approach has primarily drawn inspiration from the success of InstructPix2Pix, employing a range of 20-30 instructions during training. This choice was primarily driven by empirical observations and the composition of the InstructPix2Pix dataset.

Specifically, we performed preprocessing steps where we filtered and removed duplicate face instructions from the InstructPix2Pix dataset. This analysis revealed that, for a single attribute, there were often tens of valid instructions available. By leveraging these insights and filtering out semantically non-compliant instructions generated by ChatGPT, we arrived at the use of 20-30 instructions for training.

We acknowledge the importance of exploring the impact of instruction quantity further. To address this, we are currently conducting training experiments with a smaller or bigger number of instructions. We hope to obtain preliminary results in the near future and look forward to sharing these findings.

Reviewer comment (Experiments):
How will the model perform when it deals with novel characters as in the movie, long hair examples, black men/women, and humans of different ages?

Author response:
**Our models are designed to handle a wide range of attributes, including novel characters from movies, varied hairstyles, different races, and varying ages.** As outlined in our experimental settings (Appendix A.1.1), our model is trained on the FFHQ dataset, which comprises 70,000 faces of diverse individuals. The evaluation is conducted on a separate dataset, CelebA-HQ.

In Figure 16, we show the results of editing the characters in the movie's background and lighting conditions from this year's movie "Mission: Impossible – Dead Reckoning Part One."

Additionally, we provide more examples to illustrate the diversity of our method. These examples include long hairs (Figure 12, 13), black men and women (Figure 14, 15), olds (first row of Figure 12, last row of Figure 15), and children (first and fourth row of Figure 12, last row of Figure 14).

We believe that these examples underscore the robustness and applicability of our method across a wide range of faces and scenarios.

Reviewer comment (Experiments):
Will the background affect the edited results? It would be interesting to see the outcomes obtained when editing the same subject against various backgrounds.

Author response:
Thank you for raising this interesting question. To investigate this, we conducted experiments where we edited images of the same subject in different backgrounds.

We show the results in Figure 11, where the first two rows are the same subject, the middle two rows are the same subject and the last two rows are the same subject. The results show that the background has no obvious impact on the editing results.

However, note that when editing colors, particularly when the color being edited is close to the background color, there can be some blending between the foreground and background elements. 

We hope this addresses your query and provides insights into the background's role in our editing process.