Paper: GPAvatar: Generalizable and Precise Head Avatar from Image(s) (ICLR-2024)

Reviewer comment (Evaluation):
the PEF could well handle the segment modeled by FLAME (such as head and torso), but it cannot handles other parts, such as hair and clothes. See question 1.

Author response:
Our feature space mainly consists of two parts, the canonical tri-planes feature space and the expression feature space based on the FLAME point cloud (PEF). Non-face areas are overall modeled by the tri-planes space, which is essentially a rigid body. If we limit the feature search of the expression field to the FLAME area, then parts such as hair and torso will be completely modeled as rigid bodies that change as the camera perspective rotates. In order to be able to solve some non-flame modeling areas related to the expressions, we relax the nearest neighbor search distance of the expression feature field, so that some hints of the current expression can be obtained in the entire tri-planes feature space, which makes some non-FLAME areas more flexible. Especially areas such as hair and glasses that are close to the face but are not modeled by FLAME. The quantitative ablation experiment on global sampling also confirmed that this can improve various metrics including reconstruction quality and expression control fineness.

Reviewer comment (Evaluation):
The identity similarity in the demo video is worse than some baseline (HideNeRF).

Author response:
We examined the technology and implementation used by HideNeRF in detail, and found that Arcface was used to establish identity loss to improve identity consistency. We also adopted the same method and conducted experiments, and we can observe that our identity consistency has also been improved, but at the same time, the reconstruction quality and expression control accuracy have decreased. We believe that we may need to conduct more experiments to find a better balance between multiple losses to obtain better results. We will continue to work on this part. But it is worth noting that the fine control of expressions and the substantial improvement in reconstruction quality are our main contributions and the features we most want to retain.

In the following is the result when we add Arcface-based identity loss L_ID. Among them, ours one-in w/o L_ID is the result in the original paper version, and w L_ID is the result with identity loss added.
|   Exp on VFHQ              | PSNR↑ |  SSIM↑| LPIPS↓|CSIM↑  |   L1↓ | AED↓  | APD↓  | AKD↓ | CSIM↑ | AED↓  | APD↓  |
|---                    |---    |---    |---    |---    |---    |---    |---    |---   |---    |---    |---    |
| HideNeRF              | 20.07 | 0.745 | 0.204 | 0.794 | 0.056 | 0.521 | 0.031 | 5.33 | 0.558 | 1.024 | 0.044 |
| Ours One-in w/o L_ID  | **22.08** | **0.765** | **0.177** | 0.789 | **0.039** | **0.434** | **0.017**| **3.53** | 0.558 | **0.910** | **0.034** |
| Ours One-in w L_ID    | 21.48 | 0.759 | 0.187 | **0.797** | 0.043 | 0.464 | 0.017 | 3.84 | **0.574** | 0.932 | 0.035 |

|     Exp on HDTF         | PSNR↑ |  SSIM↑| LPIPS↓|CSIM↑  |   L1↓ | AED↓  | APD↓  | AKD↓ | CSIM↑ | AED↓  | APD↓  |
|---                    |---    |---    |---    |---    |---    |---    |---    |---   |---    |---    |---    |
| HideNeRF              | 21.38 | 0.803 | 0.147 | 0.907 | 0.038 | 0.499 | 0.027 | 4.33 | 0.803 | 1.031 | 0.032 |
| Ours One-in w/o L_ID  | **24.21** | **0.834** | **0.131** | 0.871 | **0.029** | **0.427** | **0.012** | **3.06**| 0.790 | **0.869** | **0.020** |
| Ours One-in w L_ID    | 23.67 | 0.821 | 0.136 | **0.910** | 0.034 | 0.443 | 0.013 | 3.45 | **0.812** | 0.892 | 0.020 |

Reviewer comment (Presentation):
The image quality can be improved. For instance, in Figure 1, the predicted images in the second column seems blurry.

Author response:
Since our model is only trained on VFHQ data (characters from TV interviews) and inferred on unseen IDs, for some inputs like CG characters, there are some cases where we cannot generate clear reconstruction. We hope that we can solve this problem by enhancing data or improving cross-domain robustness in future work. It is worth noting that even though we have not added these improvements, our model still has amazing generalization ability and is able to handle many cross-domain cases.

Reviewer comment (Reproducibility):
In PEF, the expression feature of facial part can be queried from the FLAME mesh, but the non-facial part, such as hair, clothes, and background is not modeled by FLAME. The authors said "we instead search for neighboring points in the entire space", but it is not clear how it bundles the non-facial part in the 3D space with the learnable features.

Author response:
Our feature space mainly consists of two parts, the canonical tri-planes feature space and the expression feature space based on the FLAME point cloud (PEF). Non-face areas are overall modeled by the tri-planes space, which is essentially a rigid body. If we limit the feature search of the expression field to the FLAME area, then parts such as hair and torso will be completely modeled as rigid bodies that change as the camera perspective rotates. In order to be able to solve some non-flame modeling areas related to the expressions, we relax the nearest neighbor search distance of the expression feature field, so that some hints of the current expression can be obtained in the entire tri-planes feature space, which makes some non-FLAME areas more flexible. Especially areas such as hair and glasses that are close to the face but are not modeled by FLAME. The quantitative ablation experiment on global sampling also confirmed that this can improve various metrics including reconstruction quality and expression control fineness.

Reviewer comment (Evaluation):
The identity similarity in the demo video is not as good as previous one-shot 3D talking face methods (such as HiDe-NeRF), what's the cause?

Author response:
We examined the technology and implementation used by HideNeRF in detail, and found that Arcface was used to establish identity loss to improve identity consistency. We also adopted the same method and conducted experiments, and we can observe that our identity consistency has also been improved, but at the same time, the reconstruction quality and expression control accuracy have decreased. We believe that we may need to conduct more experiments to find a better balance between multiple losses to obtain better results. We will continue to work on this part. But it is worth noting that the fine control of expressions and the substantial improvement in reconstruction quality are our main contributions and the features we most want to retain.

In the following is the result when we add Arcface-based identity loss L_ID. Among them, ours one-in w/o L_ID is the result in the original paper version, and w L_ID is the result with identity loss added.
|   Exp on VFHQ              | PSNR↑ |  SSIM↑| LPIPS↓|CSIM↑  |   L1↓ | AED↓  | APD↓  | AKD↓ | CSIM↑ | AED↓  | APD↓  |
|---                    |---    |---    |---    |---    |---    |---    |---    |---   |---    |---    |---    |
| HideNeRF              | 20.07 | 0.745 | 0.204 | 0.794 | 0.056 | 0.521 | 0.031 | 5.33 | 0.558 | 1.024 | 0.044 |
| Ours One-in w/o L_ID  | **22.08** | **0.765** | **0.177** | 0.789 | **0.039** | **0.434** | **0.017**| **3.53** | 0.558 | **0.910** | **0.034** |
| Ours One-in w L_ID    | 21.48 | 0.759 | 0.187 | **0.797** | 0.043 | 0.464 | 0.017 | 3.84 | **0.574** | 0.932 | 0.035 |

|     Exp on HDTF         | PSNR↑ |  SSIM↑| LPIPS↓|CSIM↑  |   L1↓ | AED↓  | APD↓  | AKD↓ | CSIM↑ | AED↓  | APD↓  |
|---                    |---    |---    |---    |---    |---    |---    |---    |---   |---    |---    |---    |
| HideNeRF              | 21.38 | 0.803 | 0.147 | 0.907 | 0.038 | 0.499 | 0.027 | 4.33 | 0.803 | 1.031 | 0.032 |
| Ours One-in w/o L_ID  | **24.21** | **0.834** | **0.131** | 0.871 | **0.029** | **0.427** | **0.012** | **3.06**| 0.790 | **0.869** | **0.020** |
| Ours One-in w L_ID    | 23.67 | 0.821 | 0.136 | **0.910** | 0.034 | 0.443 | 0.013 | 3.45 | **0.812** | 0.892 | 0.020 |

Reviewer comment (Experiments):
The head movement in the provided video is quite gentle. Is there any demo where the head pose is larger (such as side view)? Since one of the biggest advantage of 3D methods over the traditional 2D methods is the good quality under a large view angle, I think this is necessary for the reviewer to assess the performance of a 3D-based work.

Author response:
We agreed that a large change in the view angle is crucial for evaluating a 3D-based method. The animation sequence in our audio-driven demonstration is extracted from SadTalker, which always has smooth and gentle head movements. But in our paper numerous results show significant head movements, even in side views (as illustrated in Figure 7). These results show that our method effectively inherits the advantages of 3D methods in view angle changes.

Reviewer comment (Experiments):
Could you provide the visualization of the attention weights in the multi-reference setting? I'm also curious about the scalability of the MTA, how is the attention map looks like under the two-in/five-in/ten-in ?

Author response:
We appreciate the reviewer's insightful suggestion to show attention visualizations, which can better show how our MTA module works. For now, we have included these visualizations in the revised version (Sec.C and Figure.9). These attention visualization results are very interesting and show that our method can indeed pay attention to and combine different angles of people in different images.

Reviewer comment (Presentation):
In the demo video, the avatars are driven by audio, it is better to illustrate the way used to obtain the facial expression.

Author response:
We employ a widely used audio-driven motion generator: SadTalker to generate head motion based on audio input, although other similar methods can also be used. SadTalker uses an image and a piece of audio to generate a video, and then we track the 3DMM parameters in that video. We then use these parameters to animate the GPAvatar. We will add descriptions in future demos.

Reviewer comment (Novelty):
The following paper is not cited, despite the fact that it is very closely related in terms of methodology: Athar, S., Shu, Z. and Samaras, D., 2023, January. Flame-in-nerf: Neural control of radiance fields for free view face animation. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1-8). IEEE. The above un-cited paper also uses a FLAME-based representation of the 3D face and extends the formulation of NERFS to achieve realistic face animation with expression control. The similarities and differences with the proposed method are not discussed and the Flame-in-nerf is not included in the comparisons of the experimental section. This raises concerns in terms of the real novelty and contributions of the proposed method.

Author response:
Thanks for pointing this out. We acknowledge the relevance of this paper and include it in our updated related work section.
However, it is worth noting that our method is not comparable to these methods. Our approach focuses on oneshot and fewshot settings, meaning the input is only one or a few images, no training on new identities is required, and our model is driven by motion sequences based on FLAME parameters.
Among these papers, the paper [1] requires videos as input (may have thousands of images), and each ID-specific model needs to be trained based on the videos. This prevents the model from making inferences on unseen IDs, but requires data from that ID for training. This is different from our setting, where we use unseen IDs during inference to evaluate our method, which is not possible in paper [1]. At the same time, the way it uses FLAME is also different from us. Our motivation is to avoid the loss of expression information, but the paper [1] adopts a method similar to Figure.2(c), by directly inputting the expression vector to control the expression, which will obviously lead to expression Loss of information.

Reviewer comment (Novelty):
There are also several other closely related works that are not cited. For example: Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Gang Zeng, and Jingdong Wang. Real-time neural radiance talking portrait synthesis via audio-spatial decomposition. arXiv preprint arXiv:2211.12368, 2022. Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In ICCV, pp. 5784–5794, 2021. Yu, H., Niinuma, K. and Jeni, L.A., 2023, January. CoNFies: Controllable Neural Face Avatars. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG) (pp. 1-8). IEEE.

Author response:
Thanks for pointing this out. We acknowledge the relevance of this paper and include it in our updated related work section.
However, it is worth noting that our method is not comparable to these methods. Our approach focuses on oneshot and fewshot settings, meaning the input is only one or a few images, no training on new identities is required, and our model is driven by motion sequences based on FLAME parameters.
Among these papers, the paper [1] requires videos as input (may have thousands of images), and each ID-specific model needs to be trained based on the videos. This prevents the model from making inferences on unseen IDs, but requires data from that ID for training. This is different from our setting, where we use unseen IDs during inference to evaluate our method, which is not possible in paper [1]. At the same time, the way it uses FLAME is also different from us. Our motivation is to avoid the loss of expression information, but the paper [1] adopts a method similar to Figure.2(c), by directly inputting the expression vector to control the expression, which will obviously lead to expression Loss of information.

Papers [2] and [3] use audio signals as the driving method, which makes it difficult to accurately control the avatar to make the desired expression, and since we use FLAME parameters to control the avatar to make expressions, the difference in control signals makes we differ from papers [2] and [3] and cannot evaluate in the same setting.

The setting of paper [4] is similar to paper [1], requiring the use of video as training data and cannot infer with an unseen ID. This makes paper [4] not comparable to our method.

Reviewer comment (Writing):
Section 3.2: the paper fails to clearly explain how the expression information affects the process of building a point-based expression field.

Author response:
Our feature space mainly consists of two parts, the canonical tri-planes feature space and the expression feature space based on FLAME point cloud (PEF). Expression information is given in the form of FLAME parameters, and the FLAME model dynamically forms a point cloud based on these parameters. That is, different FLAME parameters will generate point clouds with different positions. Since our PEF is based on these point clouds, the feature queried from PEF will also be changed with different expression parameters.

Reviewer comment (Reproducibility):
Section 3.3: the paper provides insufficient details about about how the canonical encoder is defined and built.

Author response:
We adopt the same Style-UNet structure as in GFPGAN with only modification of the input and output size. 
In short，we obtain the style code through 4 groups of ResBlock down-sampling, use 3 groups of ResBlock up-sampling to obtain the conditions, and then use StyleGAN to output the 3 × 32 × 256 × 256 tri-planes based on style code and conditions. For more implementration details please refer to Sec.A.1, and we will also release the code and checkpoints to facilitate reproducibility and further research.

Reviewer comment (Presentation):
Figure 4: For several columns with results, it is unclear which is the corresponding method. There is one column more than the number of methods in the caption and one column more than the columns of Figure 5. This creates confusions.

Author response:
The last two columns actually correspond to "Ours One-in" and "Ours Two-in". For a fair comparison, we decided to only compare here the driving results of a single image as input. In the revised version we have removed "Ours Two-in" in Figure 4. Results of Ours Two-in or more can be referenced from other Figures (like Figure 6,7,8,9).

Reviewer comment (Writing):
After Equation (2): N is refereed to as the "input number", but apparently it should be referred to as the number of input images.

Author response:
In Equation (2): $N$ should be referred to the number of input images and we also correct it in the revised version.

Reviewer comment (Evaluation):
No discussion about the limitations of the approach?

Author response:
We appreciate the reviewer for pointing out this missing part. Specifically, our current FLAME-based model lacks a module to control the shoulders and body, resulting in limited control below the neck (the shoulder position generally aligns with the input image for now). Additionally, for other areas not modeled by FLAME, such as hair and tongue, explicit control is not feasible. Furthermore, while our aspiration is to achieve real-time reenactment of more than 30 fps, our current performance is pre-real-time for now (approximately 15 fps on the A100 GPU). We added the discussion about limitations in our revised paper (Sec.B), and will solve these problems as future work.

Reviewer comment (Theory):
It is not clear to me how the model captures the subtle information such as closed eyes?

Author response:
We avoid the information loss resulting from over-processing using our PEF (illustrated in Figure 2). Consequently, any subtle details captured by FLAME directly influence the point positions in the PEF, thereby contributing to the rendering results.

Reviewer comment (Theory):
Why the normalization of the weight wi in the equation is required?

Author response:
We appreciate the reviewers for identifying sign errors in our formulas. These mistakes have been rectified in the revised version. The corrected equation is
$$f_{exp,x} = \sum_{i}^{K} \frac{w_i}{\sum_{j}^{K} w_j} L_p(f_{i}, F_{pos}(p_i-x)), \text{where}~w_i=\frac{1}{p_i-x}, $$ where $x$ is the coordinate to be queried, $K$ is the number of neighbors, $f_i$ is the corresponding feature, $p_i$ is the point coordinate, $L_p$ is the linear layers and $F_{pos}$ is the frequency positional encoding function. In this equation,  the  $w_i$ ensures that the nearest point has the highest weight when contributing to the feature. While for the $R$ in equation (page 3), it refers to the volume rendering process in NeRF, which renders the merged feature space into the result image.

Reviewer comment (Writing):
Several terms in the equation (page 3) I_t= R (....) are not defined. What R means?

Author response:
We appreciate the reviewers for identifying sign errors in our formulas. These mistakes have been rectified in the revised version. The corrected equation is
$$f_{exp,x} = \sum_{i}^{K} \frac{w_i}{\sum_{j}^{K} w_j} L_p(f_{i}, F_{pos}(p_i-x)), \text{where}~w_i=\frac{1}{p_i-x}, $$ where $x$ is the coordinate to be queried, $K$ is the number of neighbors, $f_i$ is the corresponding feature, $p_i$ is the point coordinate, $L_p$ is the linear layers and $F_{pos}$ is the frequency positional encoding function. In this equation,  the  $w_i$ ensures that the nearest point has the highest weight when contributing to the feature. While for the $R$ in equation (page 3), it refers to the volume rendering process in NeRF, which renders the merged feature space into the result image.