Paper: MI-NeRF: Learning a Single Face NeRF from Multiple Identities (ICLR-2024)

Reviewer comment (Experiments):
The paper utilizes learnable latent codes to capture time-varying information. However, there's a potential ambiguity regarding how the model ensures these codes don't inadvertently encode identity or expression information. A more rigorous analysis or mechanism would have been beneficial to validate that these latent codes truly only represent unique, time-varying elements without overlapping with identity and expression descriptors.

Author response:
> Q1: Learned latent codes.

We included in the appendix C a qualitative comparison of our model trained with and without the latent codes (see [Fig. 10](https://imgur.com/a/FkYTlf0)).  In the revised manuscript, we denote with red color the changes for visual distinction. As the indicative frame illustrates, our learned latent codes only capture very small variations in appearance and high-frequency details, while the expression and the identity remain the same (see also ablation study in Sec. 4.2). Thus, our experimental analysis exhibits that there is no overlap of these codes with identity and expression. 

In practice, we ensure this by adding a higher L2 regularization on the latent codes (see $\lambda_{l}$ in Sec. 3.3). This results in latent codes only obtaining small values and capturing very small per-frame variations, that are not captured by the powerful representation of the expression parameters in the PCA space, or the time-invariant identity code.

Reviewer comment (Reproducibility):
The collected videos lead to concerns about data privacy and the feasibility of public release. Ethical considerations surrounding personal data must be paramount. And will the authors be able or allowed to release the data?

Author response:
> Q3: Dataset details.

We included in the pdf an additional section in the appendix, Sec. F, where we provide additional details for the dataset. As Sec. F clarifies, we collected talking face videos from publicly available datasets, commonly used by other published works. We provide a detailed list of the identities and videos used, as well as the corresponding links that include the origin of the datasets and the license. A visualization of the learned identity codes is given in Fig. 8. 

As mentioned in Sec. 3.1 and Sec. F, we run standard pre-processing techniques to fit a 3DMM per video frame and extract the corresponding head pose and expression parameters. We refer the reviewer to [6] for more details in the data pre-processing.

If the reviewer has any remaining concerns, we would be happy to address them.

Reviewer comment (Experiments):
Within 140 videos, it's unclear how many unique individuals are represented and whether multiple videos pertain to the same person. Additionally, questions arise regarding the consistency of identity codes across different videos during training.

Author response:
> Q3: Dataset details.

We included in the pdf an additional section in the appendix, Sec. F, where we provide additional details for the dataset. As Sec. F clarifies, we collected talking face videos from publicly available datasets, commonly used by other published works. We provide a detailed list of the identities and videos used, as well as the corresponding links that include the origin of the datasets and the license. A visualization of the learned identity codes is given in Fig. 8. 

As mentioned in Sec. 3.1 and Sec. F, we run standard pre-processing techniques to fit a 3DMM per video frame and extract the corresponding head pose and expression parameters. We refer the reviewer to [6] for more details in the data pre-processing.

If the reviewer has any remaining concerns, we would be happy to address them.

Reviewer comment (Reproducibility):
The steps taken to prepare the data before its integration into MI-NeRF are essential. Without clear details about preprocessing, it's challenging to replicate results or trust the dataset's integrity.

Author response:
> Q3: Dataset details.

We included in the pdf an additional section in the appendix, Sec. F, where we provide additional details for the dataset. As Sec. F clarifies, we collected talking face videos from publicly available datasets, commonly used by other published works. We provide a detailed list of the identities and videos used, as well as the corresponding links that include the origin of the datasets and the license. A visualization of the learned identity codes is given in Fig. 8. 

As mentioned in Sec. 3.1 and Sec. F, we run standard pre-processing techniques to fit a 3DMM per video frame and extract the corresponding head pose and expression parameters. We refer the reviewer to [6] for more details in the data pre-processing.

If the reviewer has any remaining concerns, we would be happy to address them.

Reviewer comment (Experiments):
While the research highlights the limitations of the GAN-based method, Wav2Lip, it doesn't delve deeply into how MI-NeRF stands compared to other recent state-of-the-art GAN approaches.

Author response:
> Q2: Comparison with GAN-based methods.

Earlier works that propose GAN-based methods for expression transfer, like GANimation [1], operate in the 2D image space. Similarly with Wav2Lip, these 2D-based approaches frequently produce artifacts, since they cannot handle 3D motion (see Fig. 4, Sec. 4.4, and appendix C).

Recent state-of-the-art GANs propose 3D-aware approaches, like Next3D [2], that are based on EG3D [3] and use StyleGAN2 [4] priors. The output identity depends on the input latent code and therefore on the random seed. In order to render a specific person, they require to run a time-consuming GAN inversion (e.g. PTI [5]). This process is required in order to project the true identity to the closest identity the model can express. To achieve that, they use the closest latent codes and fine-tune the generator weights, in order to approximate an identity, given their image. In contrast, in our case, we can directly learn a code for each identity and render them with a high fidelity. In addition, these 3D-aware GANs *require very long training times* (e.g. training EG3D for 8.5 days on 8 Tesla V100 GPUs [3] and another 4 days training for Next3D on 4 3090 GPUs [2]).

Reviewer comment (Evaluation):
MI-NeRF's uniqueness largely stems from its multiplicative module. However, this also means the model's overall success heavily depends on this single component. If the module faces challenges in complex real-world scenarios, the entire model's performance could be compromised. A deeper discussion about the potential limitations of this module would be beneficial for a comprehensive understanding of its robustness.

Author response:
> Q4. Limitations of the multiplicative module.

Our proposed multiplicative module learns non-linear interactions between identity and expression. To learn these interactions, we use talking face videos that include a variety of facial expressions from different identities. We believe that a limitation would arise if there is no overlap between the expressions of different identities. As shown in [7], multiplicative interactions can be successfully captured if there is some overlap between different attributes, in order to learn all possible combinations.  However, the case of no overlap would not easily appear in a real-world scenario, since it is very probable that different subjects will show some similar expressions, e.g. neutral expression or a smile, while talking.

Reviewer comment (Theory):
Could the authors shed light on the exact nature of the per-frame latent codes and how they ensure these codes are devoid of identity or expression specifics?

Author response:
> Q1: Learned latent codes.

We included in the appendix C a qualitative comparison of our model trained with and without the latent codes (see [Fig. 10](https://imgur.com/a/FkYTlf0)).  In the revised manuscript, we denote with red color the changes for visual distinction. As the indicative frame illustrates, our learned latent codes only capture very small variations in appearance and high-frequency details, while the expression and the identity remain the same (see also ablation study in Sec. 4.2). Thus, our experimental analysis exhibits that there is no overlap of these codes with identity and expression. 

In practice, we ensure this by adding a higher L2 regularization on the latent codes (see $\lambda_{l}$ in Sec. 3.3). This results in latent codes only obtaining small values and capturing very small per-frame variations, that are not captured by the powerful representation of the expression parameters in the PCA space, or the time-invariant identity code.

Reviewer comment (Experiments):
Could there be a more detailed comparison between MI-NeRF and advanced GAN-based methods, particularly in the context of face video synthesis?

Author response:
> Q2: Comparison with GAN-based methods.

Earlier works that propose GAN-based methods for expression transfer, like GANimation [1], operate in the 2D image space. Similarly with Wav2Lip, these 2D-based approaches frequently produce artifacts, since they cannot handle 3D motion (see Fig. 4, Sec. 4.4, and appendix C).

Recent state-of-the-art GANs propose 3D-aware approaches, like Next3D [2], that are based on EG3D [3] and use StyleGAN2 [4] priors. The output identity depends on the input latent code and therefore on the random seed. In order to render a specific person, they require to run a time-consuming GAN inversion (e.g. PTI [5]). This process is required in order to project the true identity to the closest identity the model can express. To achieve that, they use the closest latent codes and fine-tune the generator weights, in order to approximate an identity, given their image. In contrast, in our case, we can directly learn a code for each identity and render them with a high fidelity. In addition, these 3D-aware GANs *require very long training times* (e.g. training EG3D for 8.5 days on 8 Tesla V100 GPUs [3] and another 4 days training for Next3D on 4 3090 GPUs [2]).

Reviewer comment (Experiments):
There is no experimental analysis about the proposed high-degree interaction module.

Author response:
We are thankful to the reviewer TTon for their time and effort to review our paper. We address their concern below: 


> Q1: High-degree module.

We conduct experiments using the high-degree module H, please see Table 1. The results indicate that it performs on par with the module M that we propose and visually we also observe a similar visual quality to the module M. 

In addition, this module can be even more useful when interactions between additional variables should be captured, e.g., if we include lighting conditions in the future. Therefore, we do believe that this method is useful in our methodological part.

Reviewer comment (Experiments):
The question section refers back to the weakness, implying that the concern is specifically about the absence of usage and experimental analysis for the high-degree interaction module.

Author response:
We are thankful to the reviewer TTon for their time and effort to review our paper. We address their concern below: 


> Q1: High-degree module.

We conduct experiments using the high-degree module H, please see Table 1. The results indicate that it performs on par with the module M that we propose and visually we also observe a similar visual quality to the module M. 

In addition, this module can be even more useful when interactions between additional variables should be captured, e.g., if we include lighting conditions in the future. Therefore, we do believe that this method is useful in our methodological part.