Paper: Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR-2024)

Reviewer comment (Reproducibility):
Can authors clarify which 3DMM they use? Is it BFM2009 as referred from section 3.4, or is it [Blanz and Vetter 1999] as referred from B.2? Note that while they both use PCA, their bases are not compatible because they are computed from different data.

Author response:
>  Q1: Can authors clarify which 3DMM they use? In 3.4, authors say: "3DMM basis is a nonhomogeneous linear equation and has multiple solutions". At least for PCA-based 3DMMs like BFM2009 and [Blanz and Vetter 1999], this is incorrect.  Because of this misunderstanding, the paper may have an over-complex setup in its audio-to-motion model.

- A1: We use BFM2009 in all our experiments, though we have also cited [Blanz and Vetter 1999] in B.2. Following your comment, we checked the details in BFM2009 and found that our previous statement "3DMM basis is a nonhomogeneous linear equation and multiple solutions" is incorrect since the expression bases are orthogonal. Due to this problem, it is not necessary for the audio-to-motion model to avoid the MSE loss of expression (in the previous version to prevent over-smoothness, we designed an MSE loss of landmark). To handle this mistake, we have updated the discussion in the corresponding part of the revised manuscript. Specifically, **in the blue-colored part in Section 3.4 of the revised manuscript**, we replace the incorrect statement with:

  - >  [Revised Part 1] We choose BFM 2009 \citep{bfm09} as the 3DMM model. Since all expression bases are orthogonal, given the same identity code, the reconstructed 3D face meshes in a video are uniquely determined by the expression code. Hence an L2 error on the expression code, L_ExpRecon, is feasible to be the reconstruction term in training the VAE. To encourage the model to better reconstruct the facial landmark (instead of only the 3DMM parameters), we additionally introduce the L2 reconstruction error of 468 key points of the reconstructed 3DMM vertex, L_LdmRecon, as an auxiliary supervision signal. The training loss of the generic audio-to-motion model is as follows:

- Further, to fully investigate the influence of using MSE loss on expression and landmark, we compare three settings: (1) only use Expression MSE, (2) only use Landmark MSE, and (3) use both Expression / Landmark MSE. We compute the reconstructed landmark L1 error and SyncScore, and the results are shown as follows, we found that using Expression MSE loss alone achieves slightly better reconstruction performance than only using landmark MSE loss but leads to a slightly degraded sync score. we suspect that it is because for our model that predicts the expression code given audio, updating on an Expression MSE is straightforward and easier to optimize than on a Landmark MSE so the model would more focus on the reconstruction. Another finding is that combined usage of these two losses achieves the best performance in terms of reconstruction and sync score. Therefore, in the revised manuscript, we turn to using a combination of Expression MSE and Landmark MSE as the reconstruction loss of the VAE. We sincerely thank you for your careful reading and in-depth understanding of 3DMM which helps improve the correctness and soundness of this paper.
   - | Loss Setting                                    | L1 error on pred. Ldms $\downarrow$ | Sync Score  $\uparrow$|
| ----------------------------------------------- | --------------------------- | ---------- |
| MSE_Landmark (in the original manuscript)       | 0.0113                      | 6.565      |
| MSE_Expression                                  | 0.0104                      | 6.438      |
| MSE_Expression + MSE_Landmark (revised version) | $\textbf{0.0102}$                     | $\textbf{6.617} $     |

Reviewer comment (Theory):
In 3.4, authors say:  
> 3DMM basis is a nonhomogeneous linear equation and has multiple solutions (e.g., BFM2009)...  
At least for PCA-based 3DMMs like BFM2009 and [Blanz and Vetter 1999], this is incorrect. Because PCA bases are orthogonal, a unique least squares solution can be found by simply multiplying the principal components and the data (target 3D mesh vertex positions in this case). Because of this misunderstanding, the paper may have an over-complex setup in its audio-to-motion model.

Author response:
>  Q1: Can authors clarify which 3DMM they use? In 3.4, authors say: "3DMM basis is a nonhomogeneous linear equation and has multiple solutions". At least for PCA-based 3DMMs like BFM2009 and [Blanz and Vetter 1999], this is incorrect.  Because of this misunderstanding, the paper may have an over-complex setup in its audio-to-motion model.

- A1: We use BFM2009 in all our experiments, though we have also cited [Blanz and Vetter 1999] in B.2. Following your comment, we checked the details in BFM2009 and found that our previous statement "3DMM basis is a nonhomogeneous linear equation and multiple solutions" is incorrect since the expression bases are orthogonal. Due to this problem, it is not necessary for the audio-to-motion model to avoid the MSE loss of expression (in the previous version to prevent over-smoothness, we designed an MSE loss of landmark). To handle this mistake, we have updated the discussion in the corresponding part of the revised manuscript. Specifically, **in the blue-colored part in Section 3.4 of the revised manuscript**, we replace the incorrect statement with:

  - >  [Revised Part 1] We choose BFM 2009 \citep{bfm09} as the 3DMM model. Since all expression bases are orthogonal, given the same identity code, the reconstructed 3D face meshes in a video are uniquely determined by the expression code. Hence an L2 error on the expression code, L_ExpRecon, is feasible to be the reconstruction term in training the VAE. To encourage the model to better reconstruct the facial landmark (instead of only the 3DMM parameters), we additionally introduce the L2 reconstruction error of 468 key points of the reconstructed 3DMM vertex, L_LdmRecon, as an auxiliary supervision signal. The training loss of the generic audio-to-motion model is as follows:

- Further, to fully investigate the influence of using MSE loss on expression and landmark, we compare three settings: (1) only use Expression MSE, (2) only use Landmark MSE, and (3) use both Expression / Landmark MSE. We compute the reconstructed landmark L1 error and SyncScore, and the results are shown as follows, we found that using Expression MSE loss alone achieves slightly better reconstruction performance than only using landmark MSE loss but leads to a slightly degraded sync score. we suspect that it is because for our model that predicts the expression code given audio, updating on an Expression MSE is straightforward and easier to optimize than on a Landmark MSE so the model would more focus on the reconstruction. Another finding is that combined usage of these two losses achieves the best performance in terms of reconstruction and sync score. Therefore, in the revised manuscript, we turn to using a combination of Expression MSE and Landmark MSE as the reconstruction loss of the VAE. We sincerely thank you for your careful reading and in-depth understanding of 3DMM which helps improve the correctness and soundness of this paper.
   - | Loss Setting                                    | L1 error on pred. Ldms $\downarrow$ | Sync Score  $\uparrow$|
| ----------------------------------------------- | --------------------------- | ---------- |
| MSE_Landmark (in the original manuscript)       | 0.0113                      | 6.565      |
| MSE_Expression                                  | 0.0104                      | 6.438      |
| MSE_Expression + MSE_Landmark (revised version) | $\textbf{0.0102}$                     | $\textbf{6.617} $     |

Reviewer comment (Experiments):
How does the I2P model handle an input image with a non-neutral face? I.e., if the model is given an image with an extreme facial expression as the source image, what happens? Have authors thought about canonicalization in this respect? I believe a concurrent work, "Generalizable One-shot Neural Head Avatar," canonicalizes the input expression as well. Perhaps this should be clarified as a limitation.

Author response:
> Q2: (1) How does the I2P model handle an input image with a non-neutral face? (2)  Have authors thought about canonicalization in this respect? Discuss with a concurrent work, "Generalizable One-shot Neural Head Avatar," which canonicalizes the input expression.

- A2: (1) Thanks for pointing out this problem! We acknowledge your concerns that if the motion adapter is only provided with the target PNCC, it cannot work well with the I2P model which could have arbitrary expression in the cano_planes. We apologize for not clarifying this issue in the previous manuscript. Actually, our key motivation in inventing the motion adapter is to directly perform the source-to-target transform in the cano_planes (skipping an intermediate expression canonicalization process that is required by GOS_Avatar's source-neutral-target pipeline). In practice, instead of only feeding the target PNCC into the motion adapter, the input is a channel-wise concated source-target PNCC image of shape $(C_{src}+C_{tgt})\times H\times W$. We sincerely thank for your insightful comments and we have updated the missed technical details of the motion adapter in the revised version (**Please refer to the blue-colored part in Section 3.2 of the revised manuscript**). In QA3 and QA4, we will provide two demos that are helpful for a better demonstration that our I2P+MA design could well handle source images of arbitrary expression. See [this link](https://real3dportrait.github.io/static/videos/Comparison_with_deformation.mp4) for the effectiveness of the motion adapter and [this link](https://real3dportrait.github.io/static/videos/I2P_multi_view_synthesis.mp4) for the performance of the I2P model.
- (2) As for "think about canonicalization in this process" and discussion on the difference with "Generalizable One-shot Neural Head Avatar" (GOS-Avatar).  To edit the expression in a non-neutral face image, it is a common practice to convert it into a neutral expression first, then perform a neutral-to-target expression (like os_avatar does). However, this two-stage pipeline (1) requires additional supervision signals (e.g., os_avatar adopts a 3DMM rendered neutral face as the intermediate target for the network to predict) and (2) may suffer from information loss due to the source-to-neutral-to-target process (e.g, for some OOD face, the expression neutralization cannot produce satisfactory neutral face and there may be texture detail information loss). Based on these insights, during the system designing process, we think a straightforward source-to-target method should perform more robustly and efficiently, which is exactly what our motion adapter does, and the demo in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_deformation.mp4) will demonstrate our insight.  Thanks for your valuable comments that helped us better highlight the novelty, we have additionally discussed the difference between our method and GOS-Avatar in the revised version. (Please refer to the blue-colored part in Appendix A of the revised manuscript).

Reviewer comment (Experiments):
Can authors actually show results if the deformation field is used instead of the residual motion diff-plane? The paper just says "bad quality," but this is never properly compared. The comparison video says the main issue with HiDe-NeRF (which uses the deformation field) is the temporal jittering. However, the temporal jittering is mitigated in this paper by the temporal Laplacian loss, not by the residual motion diff-plane.

Author response:
> Q3: Can authors actually show results if the deformation field is used instead of the residual motion diff-plane? The paper just says "bad quality," but this is never properly compared.
- A3: Thanks for pointing out this problem! Actually, "bad quality" denotes "bad quality of the reconstructed 3D geometry", which results in worse visual quality. Following your suggestion, we trained a model that only replace the real3d-portrait's Motion adapter with a deformation field in HiDe-NeRF to achieve face animation. To be more intuitive, we provide a [demo](https://real3dportrait.github.io/static/videos/Comparison_with_deformation.mp4) that visualizes the depth and color image of this deform-based model and our motion adapter-based model when driven by audio. Please kindly refer to [this link](https://real3dportrait.github.io/static/videos/Comparison_with_deformation.mp4) for viewing. As shown in the depth image, we can observe that instead of learning a 3D head mesh, the reconstructed 3D structure of the deformation field is more like a surface (i.e., only the most-frontend voxels contribute to final results), which is ill-posed and should account for the worse visual quality of this deformation-based system. By contrast, we can see that our I2P could normally produce a 3d head mesh, and then the MA only morphs the minimal parts of geometry corresponding to the input PNCC. As discussed above, we have shown the superiority of our MA over the deformation field for face animation. We also add this discussion in the blue-colored part in Appendix A of the revised manuscript. Thanks for your helpful suggestion that improves the soundness of this paper.

Reviewer comment (Experiments):
Can authors provide results from multiple views to show how good the 3D reconstruction is?

Author response:
> Q4: Can authors provide results from multiple views to show how good the 3D reconstruction is?
- A4: Sure, please refer to [this link](https://real3dportrait.github.io/static/videos/I2P_multi_view_synthesis.mp4) for a demo of multi-view synthesis produced by our I2P model (Which is extracted from the Real3D-Portrait's final checkpoint). We can observe that the I2P could reconstruct accurate 3D shape and texture given the input image.

Reviewer comment (Experiments):
Can authors consider comparisons to Next3D [Sun et al. 2023]? I understand that Next3D is an animatable 3D GAN model requiring a GAN inversion to reproduce an identity. A discussion on a concurrent work, "Generalizable One-shot Neural Head Avatar," is not required but much appreciated to understand the novelty of this work.

Author response:
> Q5: (1) Comparison to Next3D. (2) Also, add discussion on "Generalizable One-shot Neural Head Avatar" to help understand the novelty of this work.

- A5: (1) As for the Next3D, we provide a comparison video in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_Next3D.mp4). Note that since Next3D has not released their GAN inversion code, we reproduce a Next3D + PTI to achieve face reenactment by combining the official Next3D repo and an unofficial PTI repository. In the demo video, we can see that our method has the following advantages: rendering uncropped full frames, natural torso, better identity preservation, more accurate expression control, and more efficient (ours is one-shot while PTI requires 1000 iterations of tuning). As for qualitative results, we found Next3D+PTI is worse than ours in terms of CSIM, FID, AED, and APD.

  -  | Method     | CSIM$\uparrow$   | FID$\downarrow$  | AED$\downarrow$  | APD$\downarrow$  |
  | ---------- | ---------------- | ---------------- | ---------------- | ---------------- |
  | Next3D+PTI | 0.645            | 58.66            | 0.183            | 0.039            |
  | Ours       | $\mathbf{0.758}$ | $\mathbf{42.37}$ | $\mathbf{0.138}$ | $\mathbf{0.022}$ |

- (2) As for the discussion on GOS-Avatar, as we discussed in $\mathbf{A2}$, our most related improvement is to design a motion adapter that efficiently performs the "source-to-target" transform, instead of GOS-Avatar's "source-to-neutral-to-target" transform that requires additional supervision signals and may bring information loss to the model (We suspect this might be reason that GOS-Avatar has to train an additional appearance branch to help the model to "remind" these texture details). Besides, we adopt a pretraining process to the I2P model, which is not performed by GOS-Avatar. Further, we are the first to consider the head-torso-background individual modeling, which is necessary to achieve realistic video results. Finally, we are also the first work that supports the end-to-end audio-driven task. We have added a discussion on the difference between our method and GOS-Avatar in the blue-colored part in Appendix A of the revised manuscript.

Reviewer comment (Writing):
(minor typo)  
3.1 Network Design  
> comprises a stack of SegFormer bock  
bock -> block

Author response:
> Q6: (minor typo) "3.1 Network Design comprises a stack of SegFormer bock."  bock -> block

- A6: Thanks for your careful reading of this manuscript! We have fixed this typo in the revised manuscript accordingly.s

Reviewer comment (Novelty):
The paper has omitted citing some important related methods of the field: J. S. Chung, A. Jamaludin, and A. Zisserman, “You said that?” in BMVC, 2017. Ye, Z., Xia, M., Yi, R., Zhang, J., Lai, Y.-K., Huang, X., et al. (2022). Audio-driven talking face video generation with dynamic convolution kernels. IEEE Transactions on Multimedia.

Author response:
- A1: Thanks for your insightful comment, we have cited the mentioned related works in the revised manuscript, which we found has improved the soundness of the manuscript. We also compare with DFA-NeRF in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_DFANeRF.mp4) and AUD in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_DFANeRF.mp4). Note that both DFA-NeRF and AUD are person-specific methods that over-fit a target identity with a two-minute-long video. We tried our best to reproduce these results, but since DFA-NeRF seems incomplete, it could only produce reasonable results when driven by the ground truth expression. So the comparison is mainly performed with AUD. Firstly, we can perform a qualitative comparison from the demo videos. We can see that both DFA-NeRF and AUD produce temporal jittering results, and AUD has a slightly worse identity-preserving ability. By contrast, our method is one-shot, temporally stable, identity-preserving, and lip-synchronized. As for the quantitative comparison with AUD, for a fair comparison, we crop and downsample our method into 256x256 resolution to align with AUD. The results are as follows, we can observe better Sync Score, AED, CSIM, and FID over AUD.
   - | Method | CSIM$\uparrow$   | FID$\downarrow$  | AED$\downarrow$ | Sync$\uparrow$   |
| ------ | ---------------- | ---------------- | --------------------------- | ---------------- |
| AUD    | 0.742            | 47.59            | 0.160                       | 5.863            |
| Ours   | $\mathbf{0.775}$ | $\mathbf{42.63}$ | $\mathbf{0.141}$            | $\mathbf{6.630}$ |

Reviewer comment (Novelty):
The method is based on the projected normalized coordinate code (PNCC) representation but it has not cited one of the most important works of the field that is also based on the same representation: Kim, H., Garrido, P., Tewari, A., Xu, W., Thies, J., Niessner, M., Pérez, P., Richardt, C., Zollhöfer, M. and Theobalt, C., 2018. Deep video portraits. ACM transactions on graphics (TOG), 37(4), pp.1-14.

Author response:
- A1: Thanks for your insightful comment, we have cited the mentioned related works in the revised manuscript, which we found has improved the soundness of the manuscript. We also compare with DFA-NeRF in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_DFANeRF.mp4) and AUD in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_DFANeRF.mp4). Note that both DFA-NeRF and AUD are person-specific methods that over-fit a target identity with a two-minute-long video. We tried our best to reproduce these results, but since DFA-NeRF seems incomplete, it could only produce reasonable results when driven by the ground truth expression. So the comparison is mainly performed with AUD. Firstly, we can perform a qualitative comparison from the demo videos. We can see that both DFA-NeRF and AUD produce temporal jittering results, and AUD has a slightly worse identity-preserving ability. By contrast, our method is one-shot, temporally stable, identity-preserving, and lip-synchronized. As for the quantitative comparison with AUD, for a fair comparison, we crop and downsample our method into 256x256 resolution to align with AUD. The results are as follows, we can observe better Sync Score, AED, CSIM, and FID over AUD.
   - | Method | CSIM$\uparrow$   | FID$\downarrow$  | AED$\downarrow$ | Sync$\uparrow$   |
| ------ | ---------------- | ---------------- | --------------------------- | ---------------- |
| AUD    | 0.742            | 47.59            | 0.160                       | 5.863            |
| Ours   | $\mathbf{0.775}$ | $\mathbf{42.63}$ | $\mathbf{0.141}$            | $\mathbf{6.630}$ |

Reviewer comment (Novelty):
The method supports the conditioning of the animation based on a speech audio signal but it does not cite another important and seminal work of the field that does something similar: Kim, H., Elgharib, M., Zollhöfer, M., Seidel, H.P., Beeler, T., Richardt, C. and Theobalt, C., 2019. Neural style-preserving visual dubbing. ACM Transactions on Graphics (TOG), 38(6), pp.1-13.

Author response:
- A1: Thanks for your insightful comment, we have cited the mentioned related works in the revised manuscript, which we found has improved the soundness of the manuscript. We also compare with DFA-NeRF in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_DFANeRF.mp4) and AUD in [this link](https://real3dportrait.github.io/static/videos/Comparison_with_DFANeRF.mp4). Note that both DFA-NeRF and AUD are person-specific methods that over-fit a target identity with a two-minute-long video. We tried our best to reproduce these results, but since DFA-NeRF seems incomplete, it could only produce reasonable results when driven by the ground truth expression. So the comparison is mainly performed with AUD. Firstly, we can perform a qualitative comparison from the demo videos. We can see that both DFA-NeRF and AUD produce temporal jittering results, and AUD has a slightly worse identity-preserving ability. By contrast, our method is one-shot, temporally stable, identity-preserving, and lip-synchronized. As for the quantitative comparison with AUD, for a fair comparison, we crop and downsample our method into 256x256 resolution to align with AUD. The results are as follows, we can observe better Sync Score, AED, CSIM, and FID over AUD.
   - | Method | CSIM$\uparrow$   | FID$\downarrow$  | AED$\downarrow$ | Sync$\uparrow$   |
| ------ | ---------------- | ---------------- | --------------------------- | ---------------- |
| AUD    | 0.742            | 47.59            | 0.160                       | 5.863            |
| Ours   | $\mathbf{0.775}$ | $\mathbf{42.63}$ | $\mathbf{0.141}$            | $\mathbf{6.630}$ |

Reviewer comment (Reproducibility):
In the methodology, the paper does not clarify how the per-frame head pose is predicted from the audio, in the case of audio-driven talking face generation. More details and clarifications about that should be provided.

Author response:
- A2: Thanks for pointing out this problem. In this paper, since the prediction of head pose is not our main interest, we devise a naive strategy to obtain head pose sequences from GT videos of CelebV-HQ. To be specific, (1) in the same-identity setting, the driving head pose is exactly the same as the GT video of the source image (source image is the first frame of the test video); (2) in the cross-identity setting, the driving head pose is extracted from the driving video; (3) then, in the audio-driven setting, most importantly, we first estimate the head pose in the source image, then we query 10 videos in the CelebV-HQ that has nearest distance between its first frame's head pose and the source image. Once the candidate videos that provide head pose are sampled, we randomly choose one of them to drive the source image. We acknowledge that an explanation of the head pose selection strategy is beneficial for the soundness of the manuscript, and have added the discussion above **in the blue-colored part in Appendix D.1 of the revised manuscript**. Thanks again for improving the soundness of this paper!

Reviewer comment (Evaluation):
In terms of the realism of the final synthetic video, one can observe that the method has some limitations regarding generating realistic background. Considering the general quality and realism of the results, this is an acceptable limitation, but the authors should openly discuss these limitations and link them with potential directions of future works.

Author response:
- A3: We admit that the quality of the inpainted background is important for the realism of the final synthetic video. Since the background inpainting process can be performed by arbitrary off-the-shelf inpainting methods, we didn't focus on this topic in the original manuscript. We also notice that the proposed KNN-based inpainting method is relatively naive and can be replaced with a more advanced neural network-based inpainting method. We truly agree with your idea that an open discussion on the limitation of the background inpainting part is necessary and could motivate future improvements. Specifically, in the revised manuscript, we have added the following paragraph **to the blue-colored part in Appendix E.1, Limitation and future works, of the revised manuscript**:

  > Though we have achieved generally high-quality realistic talking portrait results, one of the limitations is that the inpainted background image could leak unnaturalness when the talking person is doing a large pose motion. We believe the introduced naive KNN-based background inpainting method is to be blamed. Since generating a realistic background is of high importance to ensure the realism of the final video, we plan to upgrade the background inpainting method with a more advanced neural network-based system, such as MAT (MAT: Mask-Aware Transformer for Large Hole Image Inpainting) and DDNM (ero-shot image restoration using denoising diffusion null-space model). An alternative might be learning the background inpainting module within the HTB-SR model in an end-to-end manner. We leave this for future work.

Reviewer comment (Experiments):
Given that the overall architecture is very similar to HiDe-NeRF, it is unclear where the improvement of the proposed method is coming from. Is it because a pretrained Tri-plane is a better representation than the multi-resolution tri-plane features of HiDe-NeRF? It would be great if the authors could clarify this.

Author response:
We thank the reviewer for the constructive feedback and the positive remarks on our qualitative/quantitative results. We acknowledge that your concerns are mainly about the novelty of this paper, and hope our response resolves your concerns fully.

> Q1: (1) Given that the overall architecture is very similar to HiDe-NeRF, it is unclear where the improvement of the proposed method is coming from. (2) Is it because a pretrained Tri-plane is a better representation than the multi-resolution tri-plane features of HiDe-NeRF? It would be great if the authors could clarify this 

- A1: (1) Actually, there are several significant differences between our method and HiDe-NeRF. Firstly, we pretrain the I2P model while HiDe-NeRF doesn't; secondly, we use the motion adapter to predict a motion diff-plane while HiDe-NeRF resorts to the deformation field; thirdly, we first propose to individually model the head-torso-background parts; fourthly, we propose a generic audio-to-motion model to additionally support audio-driven tasks while HiDe-NeRF only support video-driven. 
- (2) As for the performance improvement of ours over the HiDe-NeRF, we think the reasons are three-fold. Firstly, as pointed out by the reviewer, pre-training the I2P model is an important reason since it utilizes the prior knowledge about 3D geometry from the 3D GAN. We have ablated this in Figure 15 of the original manuscript. **Secondly, and most importantly, we think replacing the deformation field with our proposed motion adapter is the key reason.** The deformation field aims to learn to deform the ray in the canonical space during inference to achieve dynamic scene synthesis. However, based on our experiment (please kindly refer to [this link](https://real3dportrait.github.io/static/videos/Comparison_with_deformation.mp4), in which we compare our motion adapter versus the deformation field by HiDe-NeRF), we find deformation field destroys the good 3D mesh reconstructed by our I2P model and tends to predict an ill-posed face surface (instead of a 3D mesh), which results in bad robustness and bad visual quality. By contrast, our motion adapter only edits the minimal part of the 3D face representation corresponding to the input PNCC, hence maintaining the most of geometry and texture information stored in the reconstructed canonical tri-plane. As a result, our method achieves better identity preserving and geometry realness than HiDe-NeRF. The third reason for our better performance over HiDe-NeRF is that we individually model the torso part. By contrast, in HiDe-NeRF the torso part will rotate along with the head part, which is unrealistic. We sincerely thank the reviewer for sharing the valuable feedback, and we have added the discussion above in the appendix of the revised manuscript (**Please refer to the blue-colored part in Appendix A of the revised manuscript for more details**), which we suspect could better clarify our novelty and the intrinsic mechanism of our better quality than HiDe-NeRF.

Reviewer comment (Novelty):
The addition of background and torso modelling, while important, is relatively incremental.

Author response:
We appreciate it for acknowledging the importance of individually modeling the torso and background segment in the field of one-shot 3D talking generation. We believe that the proposed head-torso-background individual modeling paradigm could pave the way for future works to achieve more realistic results (since most existing works only focus on the head part or jointly model the head and torso). As for the "incremental", we admit that our HTB-SR has adopted many known techniques to guarantee robustness and performance, e.g., the Torso branch is based on the warping-based method. However, we have also come up with several necessary modifications to achieve more realistic results. For instance, in the warping-based torso branch, we propose to replace the implicit key points in the warping-based method with our predefined projected facial landmark, which is necessary to guarantee temporal stability. Also, it is the first attempt that only warps a specific torso segment of the talking person frame, as the previous methods typically warp the whole frame, which results in distortion artifacts in the background part. Besides, when integrating the head-torso-background segments, we propose an alpha-blending-style fusion mechanism, which could handle the robustness issue caused by the naive channel-wise concatenation. We hope the above clarification could better highlight our contribution to the addition of background and torso modeling.

Reviewer comment (Experiments):
It would be great if the authors could clarify why the proposed method works better than HiDe-NeRF despite having a very similar architecture and set-up. Is pretraining the only reason?

Author response:
We thank the reviewer for the constructive feedback and the positive remarks on our qualitative/quantitative results. We acknowledge that your concerns are mainly about the novelty of this paper, and hope our response resolves your concerns fully.

> Q1: (1) Given that the overall architecture is very similar to HiDe-NeRF, it is unclear where the improvement of the proposed method is coming from. (2) Is it because a pretrained Tri-plane is a better representation than the multi-resolution tri-plane features of HiDe-NeRF? It would be great if the authors could clarify this 

- A1: (1) Actually, there are several significant differences between our method and HiDe-NeRF. Firstly, we pretrain the I2P model while HiDe-NeRF doesn't; secondly, we use the motion adapter to predict a motion diff-plane while HiDe-NeRF resorts to the deformation field; thirdly, we first propose to individually model the head-torso-background parts; fourthly, we propose a generic audio-to-motion model to additionally support audio-driven tasks while HiDe-NeRF only support video-driven. 
- (2) As for the performance improvement of ours over the HiDe-NeRF, we think the reasons are three-fold. Firstly, as pointed out by the reviewer, pre-training the I2P model is an important reason since it utilizes the prior knowledge about 3D geometry from the 3D GAN. We have ablated this in Figure 15 of the original manuscript. **Secondly, and most importantly, we think replacing the deformation field with our proposed motion adapter is the key reason.** The deformation field aims to learn to deform the ray in the canonical space during inference to achieve dynamic scene synthesis. However, based on our experiment (please kindly refer to [this link](https://real3dportrait.github.io/static/videos/Comparison_with_deformation.mp4), in which we compare our motion adapter versus the deformation field by HiDe-NeRF), we find deformation field destroys the good 3D mesh reconstructed by our I2P model and tends to predict an ill-posed face surface (instead of a 3D mesh), which results in bad robustness and bad visual quality. By contrast, our motion adapter only edits the minimal part of the 3D face representation corresponding to the input PNCC, hence maintaining the most of geometry and texture information stored in the reconstructed canonical tri-plane. As a result, our method achieves better identity preserving and geometry realness than HiDe-NeRF. The third reason for our better performance over HiDe-NeRF is that we individually model the torso part. By contrast, in HiDe-NeRF the torso part will rotate along with the head part, which is unrealistic. We sincerely thank the reviewer for sharing the valuable feedback, and we have added the discussion above in the appendix of the revised manuscript (**Please refer to the blue-colored part in Appendix A of the revised manuscript for more details**), which we suspect could better clarify our novelty and the intrinsic mechanism of our better quality than HiDe-NeRF.

Reviewer comment (Experiments):
The method by design requires the canonicalization, i.e., removal of the source image's facial expression, for the driving expression to be successfully applied to it. This is because the PNCC code is derived purely from the target driving video/audio's expression and hence cannot contain the information to erase/neutralize the source image's facial expression. I think the proposed method achieves this canonicalization during the fine-tuning phase with the Celeb-V-HQ video dataset. However, in my experience without any further stronger constraints, the proposed method cannot fully remove the source image's original facial expression. Can the author show examples of these canonicalized 3D face reconstructions of the input images without applying the target expression? Related to this is the question of how robust is the proposed method to the presence of large facial expressions in the source image. Does it work for expressions where the source image has a wide open mouth and lowered jaw or nearly closed eyes, for example?

Author response:
> Q1: (1) Doubt about the motion adapter's capability of expression canonicalization (or say, neutralization), as without any further stronger constraints, the proposed method cannot fully remove the source image's original facial expression. (2) Can the author show examples of these canonicalized 3D face reconstructions of the input images without applying the target expression? (3) Does it work for expressions where the source image has a wide open mouth and lowered jaw or nearly closed eyes, for example?

- A1: (1) Thanks for pointing out this problem! We acknowledge your concerns that if the motion adapter is only provided with the target PNCC, it cannot work well with the I2P model which could have arbitrary expression in the cano_planes. We apologize for not clarifying this issue in the previous manuscript. Actually, our key motivation in inventing the motion adapter is to directly perform the source-to-target transform in the cano_planes (skipping an intermediate expression canonicalization process that is required by a traditional source-to-neutral-to-target pipeline). In practice, instead of only feeding the target PNCC into the motion adapter, the input is a channel-wise concatenated source-target PNCC image of shape $(C_{src}+C_{tgt})\times H\times W$. So the motion adapter could learn the source-to-target transform only related to the 3DMM geometry prior while eliminating any other texture-related information. We sincerely thank you for your insightful comments and we have added the technical details above to the revised manuscript (**Please refer to the blue-colored part in Section 3.2 of the revised manuscript for more details**). 
- (2) Sure, in [this link](https://real3dportrait.github.io/static/videos/I2P_multi_view_synthesis.mp4) we provide a demo in which the I2P model is extracted from Real3D-Portrait's final checkpoint and perform the multi-view synthesis task. We can observe that the rendered canonical plane indeed is of the source expression. (3) Yes, as we discussed in the first point, although without an extra expression neutralization process since our motion adapter learns the source-to-target transform by utilizing the input source-target-concatenated PNCC, it could support arbitrary target expression given a source image of arbitrary source expression. To demonstrate this, we provide a demo video in [this link](https://real3dportrait.github.io/static/videos/Demo_critical_source_images.mp4), in which we show our method performs well given the critical source images that have a wide open mouth, lowered jaw, or totally closed eyes. To summarize, in this QA, we discussed Real3D-Portrait's generalizability of 3D reconstruction and animation. We'd like to thank you for providing constructive comments to help us enrich the technical details in the manuscript and motivate us to produce these intuitive demos, which we believe are helpful for new readers to perceive the intrinsic mechanism of this paper.

Reviewer comment (Novelty):
Overall the proposed methods is an obvious combination of several existing ideas from prior works on the topic to culminate in a successful large engineered end-to-end solution. For example, the idea of using I2P was previously proposed in LP3D; the idea of using rendered target expression images from a 3DMM was proposed in Li et al., 2023b, the audio to PNCC code predictor is borrowed from prior work; and the torso warping module is also borrowed from Wang et al., 2021. While the proposed combination of existing ideas results in an effective solution, from the research perspective the overall solution is light on significant novel or surprising insights.

Author response:
> Q2: About novelty. The proposed methods combines several existing ideas from prior works to culminate in a successful large engineered end-to-end solution.

- A2: Thanks for your comments. We'd like to explain the major novelty of this paper as follows. We first propose an efficient motion adapter that achieves high-quality and geometry-preserving 3D face source-to-target animation, without an extra expression neutralization process. Secondly, we are the first work that points out the necessity of head-torso-background individual modeling and come up with an effective HTB-SR model (with several necessary modifications, such as explicit keypoint in the torso branch and alpha-blending-style fusion mechanism in combining the three segments) to handle this issue. Thirdly, we are the first work that achieves audio-driven ont-shot 3D talking face. Besides, the proposed four-stage framework, which supports audio/video-driven applications,  is easy to be expanded to support other sub-topics, such as stylized talking faces, etc. We expect Real3D-Portrait to provide a new paradigm for the new generation of one-shot 3D talking face methods to produce realistic video results.

Reviewer comment (Reproducibility):
The authors don't specify the dataset/protocol that they used for evaluation.

Author response:
> Q3: Specify the dataset/protocol that are used for evaluation.

- A3: Thanks for pointing out this issue. Due to space limitations, in the original manuscript, we only briefly explain the dataset/protocol used for evaluation. To be specific, there are three groups of evaluated data: (1) for the Same-Identity Reenactment, we randomly chose 100 videos in our preserved validation split of CelebV-HQ. (2) for the Cross-Identity Reenactment, we use the first frame of the previously selected videos to obtain 100 identities, then use the expression-pose sequence from a randomly selected video to construct the cross-identity reenactment data pair. (3) for the audio-driven scenario, we use 10 out-of-domain images downloaded from the internet (which are exactly the 10 identities shown in the demo video) and choose 10 audios from different languages to form the data pair (so each method has 100 videos as the test samples).  We apologize for the missed dataset/protocol specification in the original manuscript, and **we have added it to the blue-colored part in Appendix D.1 of the revised version**. Thanks for your suggestion that improves the technical soundness of the manuscript.