Paper: A Large-Scale 3D Face Mesh Video Dataset via Neural Re-parameterized Optimization (ICLR-2024)

Reviewer comment (Novelty):
I agree that the paper proposes the first method to introduce the neural re-parameterized technique to optimize FLAME parameters. As I know, previous work [1] has already introduced this technique to solve very relevant (actually more challenging) tasks; they adopt a neural network to re-parameterize the per-vertex displacement of a FLAME mesh. However, the submission does not discuss these very relevant works.

Author response:
We appreciate the reviewer for recommending the reference. We have revised the paper to include the discussion about the suggested reference in the 4th paragraph of Sec. 2 of the revision, highlighted in pink.

However, please note that our work and [C1] are focus on very different perspectives as:

1. We first remark that, in contrast to ours, Neural Head Avatars [C1] does not re-parameterize FLAME parameters with neural networks, but directly optimizes FLAME parameters which corresponds to the baseline we presented.
2. In contrast, we provided the empirical analysis about the gradient densification (Sec. 3.3, p. 5 in the initial submission and Sec. A.1, p. 17 in the Appendix), which can be attributed to the proposed neural re-parameterization, and hasn’t been explored in the fields. While [C2] also tries to make the dense gradient for geometry optimization, it uses a different technique, pre-conditioning, which is different from our neural re-parameterization. Also, we briefly showed the robustness and high-probability convergence guarantee of the optimization to global optima in the theoretical proof sketch of the initial submission (Sec. A.2, p. 18 in the Appendix).
3. They parameterize residual fine-detail geometries of face using an MLP, which is not our scope for reconstructing details of face shapes. Our scope is to reconstruct facial geometries, well complying with input facial gestures and motions as discussed in the response to W1 of Reviewer axFo.

Given these, [C1,C2] do not harm our contribution. Thanks for suggesting references. 

[C1] Grassal et al., Neural head avatars from monocular rgb videos. In CVPR 2022.

[C2] Nicolet et al., Large Steps in Inverse Rendering of Geometry. In SIGGRAPH ASIA 2021.

Reviewer comment (Experiments):
I think the video FLAME fitting algorithm is not well-designed. One of the contributions of the paper is to use the proposed fitting algorithm to provide pseudo GT for the video dataset, so I think it is necessary to push the quality of the fitting method as high as possible. I list some questions about the design choice here.

Author response:
- **W2-1.** **Why not use a photometric loss term?** 
    
As requested, we report the effect of the photometric loss ($\mathcal{L}\_{photo}$) in the metrics for the MEAD dataset we used in the main paper. We compare the optimization that uses three different loss configurations as below:

| Loss configuration  | $\text{MSI}\_{3D}\^L (\uparrow)$ | $\text{MSI}\_{3D}\^V (\uparrow)$ | $\text{CVD} (\downarrow)$ | $\text{NME} (\downarrow)$ |
| --- | --- | --- | --- | --- |
| **NeuFace (Original)** | **0.206** | **0.305** | **0.103** | **2.58** |
| NeuFace + $2\cdot\mathcal{L}\_{photo} $ | 0.205 | 0.299 | 0.106 | 2.62 |
| NeuFace + $5\cdot\mathcal{L}\_{photo}$ | 0.195 | 0.282 | 0.112 | 3.69 |

We tried this in our preliminary experiment, and even with the tuning of balancing, we obtain negligible effects of using  $\mathcal{L}\_{photo}.$ Using $\mathcal{L}\_{photo}$ rather degrades the performance, and the larger $\mathcal{L}\_{photo}$ results in more degradation. 

We postulate that, since $\mathcal{L}\_{photo}$ is designed to enforce the consistency of colors, i.e., albedo assumption, thus it has been known to be very prone to self-shadows, and non-Lambertian reflection by lighting, and noise existing in videos, which distract the optimization. Furthermore, it takes more than twice the GPU memory than the original configuration, due to the requirement of the differentiable rendering. Thus, we claim our design choice is sufficient to reconstruct dynamic 3D faces despite its compact combination. 

We have newly added the results and discussion in the revision (Sec. A.4, p. 21 in the Appendix).

Reviewer comment (Experiments):
Why not use a photometric loss term? Many previous works have demonstrated that photometric loss can improve the geometry reconstruction quality, see Figure 6 in [2]. I'm curious when photometric loss is used, will the proposed neural re-parameterized technique still improve the results a lot?

Author response:
1. Our proposed neural re-parameterized optimization is indeed useful with the dense energy terms, e.g., the photometric loss.
    
    As we discussed, **the limitations lie within the photometric loss itself, assuming the albedo consistency, and not within our optimization method**. This point was clearly mentioned in the response of our discussion. Although it's a different context, it has been known that the limitations of the photometric loss, e.g., the recent work [C10] also pointed out its limitations due to Albedo consistency assumption, which does not hold for human facial skins that have the view points varying radiance property.
    
    [C10] Ge et al., Ref-NeuS: Ambiguity-Reduced Neural Implicit Surface Learning for Multi-View Reconstruction with Reflection. In ICCV 2023 (Oral paper).

Reviewer comment (Experiments):
Why not use a shared identity code?

Author response:
- **W2-2. Why not use a shared identity code?**
    
We tried this in our preliminary experiment, but found negligible differences in our early test. 
In our newly conducted investigation, we found that the standard deviation of the identity codes ($\boldsymbol{\beta}$) of all frames, gradually decreases and converges during NeuFace optimization.
    
| Optim. step | 0 | 25 | 50 | 75 | 100 (End) |
| --- | --- | --- | --- | --- | --- |
| $\boldsymbol{\beta}$ std. | 3.152 | 1.489 | 1.372 | 1.370 | 1.366 |
    
This means, the identity codes across video frames and views automatically converge, although we do not manually force the shared identity codes with the losses. Accordingly, in the visual results, we obtain identical results in both 1) original NeuFace optimization and 2) NeuFace optimization with consistent $\boldsymbol{\beta}$. 
    
We have revised the paper and added this discussion and results in the revision (Sec. A.4, p. 21 in the Appendix)

Reviewer comment (Experiments):
From the supp. video, I find that the fitting results are still jittering, although better than the naive baseline of applying DECA to each frame separately. More advanced strategies beyond the temporal moving average method should be exploited, like the commonly-used optical flow loss, or stabilize the detected landmarks [3].

Author response:
- **W2-3.** **More advanced strategies beyond the temporal moving average method should be exploited, like the common-used optical flow loss, or stabilize the detected landmarks [C3].**
    
We thank the reviewer for suggesting diverse methods for improving each component in our optimization. 
As discussed in the response to [W2-1], our design choice is sufficient to reconstruct dynamic 3D faces despite its compact combination. It is also sufficient to validate the idea of neural re-parameterization with bootstrapping. Thus, adding more losses does not change our message and finding of our work we conclude (This is the goal of a piece of the academic paper). 
    
On the other hand, we agree with the reviewer’s point that using advanced losses would increase the accuracy. Note that we can readily add more advanced losses for our proposed optimization. Given the limited rebuttal period, we’ll investigate other advanced combinations of methods later, and add the relevant discussion in the camera-ready version.
    

[C3] Naruniec et al., High-Resolution Neural Face Swapping for Visual Effects. In Eurographics Symposium on Rendering 2020.

Reviewer comment (Experiments):
Thus, it gives me the impression that the paper does not do its best to improve the video-fitting results. I would like to hear from the authors to change my impression.

Author response:
- **W2-3.** **More advanced strategies beyond the temporal moving average method should be exploited, like the common-used optical flow loss, or stabilize the detected landmarks [C3].**
    
We thank the reviewer for suggesting diverse methods for improving each component in our optimization. 
As discussed in the response to [W2-1], our design choice is sufficient to reconstruct dynamic 3D faces despite its compact combination. It is also sufficient to validate the idea of neural re-parameterization with bootstrapping. Thus, adding more losses does not change our message and finding of our work we conclude (This is the goal of a piece of the academic paper). 
    
On the other hand, we agree with the reviewer’s point that using advanced losses would increase the accuracy. Note that we can readily add more advanced losses for our proposed optimization. Given the limited rebuttal period, we’ll investigate other advanced combinations of methods later, and add the relevant discussion in the camera-ready version.
    

[C3] Naruniec et al., High-Resolution Neural Face Swapping for Visual Effects. In Eurographics Symposium on Rendering 2020.

Reviewer comment (Experiments):
Apply DECA to each frame is too weak to serve as a baseline for video fitting. There are many previous works [4-6] proposed to fit 3DMM into a video. Why not compare with these more strong baselines specialized in video fitting? In the supp. material, the method is compared to MICA-T, but I think more competitors are expected. At least they should be discussed in the paper. So I think the paper does not well evaluate the proposed technique for video fitting.

Author response:
We compared with MICA+Tracker as the representative competitor because of the following reasons:

1. MICA+Tracker [C4] is the most competitive video fitting work, which already demonstrate the surpassing performance over Face2Face [C5], and other previous baselines [C6,C7]. Ours better performs than MICA+Tracker, which implies ours performs also better than [C5,C6,C7]. 
2. For [C8,C9], their codes are not publicly available. During the discussion period, we reached out to the authors of [C8,C9] and requested to share the code, but we couldn’t get a reply. We cannot even quote their quantitative results, because they did not present the benchmarks shared with both MICA and ours.

Their quantitative results cannot be directly quoted, because the benchmarks they presented are not compatible with ours. Despite our efforts, we could not manage to add them. Instead, we have newly added the reference in the revision (Sec. C.2, p. 24 in the Appendix).

[C4] Zielonka et al., MICA - Towards Metrical Reconstruction of Human Faces. In ECCV 2022.

[C5] Thies et al., Face2Face: Real-time face capture and reenactment of rgb videos. In CVPR 2016.

[C6] Deng et al., Accurate 3d face reconstruction with weakly-supervised learning: From single image to image set. In CVPRW 2019.

[C7] Feng et al., Learning an animatable detailed 3D face model from in-the-wild images. In SIGGRAPH 2021.

[C8] Cao et al., 3D Shape Regression for Real-time Facial Animation. In SIGGRAPH 2013.

[C9] Cao et al., Real-time high-fidelity facial performance capture. In SIGGRAPH 2015.

Reviewer comment (Experiments):
Unfair Comparison: The comparison with methods like DECA and EMOCA, which operate on single-view data (DECA-dataset and EMOCA-dataset), cannot utilize multi-view information. It can be argued that the proposed method leverages more information by utilizing multi-view data. Therefore, comparing the proposed multi-view approach to these single-view reconstruction methods may not provide a fair evaluation.

Author response:
We confirm that the experiments were fair.

Our NeuFace optimization is a **test-time optimization method.** The test-time optimization is a method that updates the model to fit into the deployment environment at test time, without access to training data [C1]. Test-time optimization methods utilize the losses with self-supervision or pseudo supervision, available at test-time. Prior arts [C1,C2,C3,C4] also compared their proposed test-time optimization methods and test-time losses with the baseline methods that do not involve test-time optimization. It is regarded as fair and valid comparison.

When evaluating the test-time optimization methods, the configuration of the pre-trained backbone network is important. Our NeuFace optimization starts from the pre-trained DECA/EMOCA checkpoint same as the compared baselines, which makes our settings and comparison fair.

Also, we would have tried if there was a fair setting that the reviewer specifically thinks. Please leave the comment if there is a specific fair setting to compare with DECA/EMOCA. We’d like to further discuss and willing to include it. 

[C1] Yi et al., Temporal Coherent Test-Time Optimization for Robust Video Classification. In ICLR 2023.

[C2] Zeng et al., Test-Time Optimization for Video Depth Estimation Using Pseudo Reference Depth. In Computer Graphics Forum 2023.

[C3] Luo et al., Consistent Video Depth Estimation. In SIGGRAPH 2020.

[C4] Schneider et al., Improving robustness against common corruptions by covariate shift adaptation. In NeurIPS 2020.

Reviewer comment (Novelty):
The novelty is limited. The proposed temporal-consistency-loss and multi-view-consistency-loss seem more like separate regularizations (or averages) applied to pose, camera parameters, or face shape and expression coefficients to achieve reduced jitter in the reconstructed videos.

Author response:
We haven’t claimed the novelty of our losses. While our loss configurations are simple, we obtain a sufficiently good quality dataset, thanks to our neural re-parameterized optimization and its favorable properties. We can readily add more advanced losses for our proposed optimization.

Although our loss configuration is sufficient, we share the experimental result that reviewer B8S8 asked: NeuFace optimization + photometric loss ($\mathcal{L}_{photo}$).

Below we report the metrics for the MEAD dataset we used in the main paper. We compare the optimization that uses three different loss configurations as follows: 

| Loss configuration  | $\text{MSI}_{3D}^L (\uparrow)$ | $\text{MSI}_{3D}^V (\uparrow)$ | $\text{CVD} (\downarrow)$ | $\text{NME}(\downarrow)$ |
| --- | --- | --- | --- | --- |
| **NeuFace (Original)** | **0.206** | **0.305** | **0.103** | **2.58** |
| NeuFace + $2*\mathcal{L}_{photo} $ | 0.205 | 0.299 | 0.106 | 2.62 |
| NeuFace + $5*\mathcal{L}_{photo}$ | 0.195 | 0.282 | 0.112 | 3.69 |

We obtain negligible effects of using $\mathcal{L}_{photo}$. The results show our minimal loss combination is sufficient and essential to reconstruct dynamic 3D faces. We have added the results and discussion in the revision (Sec. A.4, p. 21 in the Appendix).

Reviewer comment (Experiments):
Doubts remain about the effectiveness of the multi-view-consistency-loss. In the training set, only the MEAD dataset consists of multi-view video data, while VoxCeleb2 and CelebV-HQ have only single-view video data. Consequently, it appears that only the MEAD dataset can effectively leverage the multi-view consistency loss. Table 1 illustrates that MEAD comprises a mere 1% of the total duration, suggesting that the majority of the proposed NeuFace-dataset primarily derives from VoxCeleb2 and CelebV-HQ.

Author response:
We’d like to remind that our work focuses on the effectiveness of our proposal, the neural re-parameterization with bootstrapping (temporal and multi-view consistency).  

Thus, we show that

1. Even with the temporal bootstrapping alone in the majority of data, we demonstrate the effectiveness of our NeuFace dataset by significantly enhancing the 3D reconstruction quality in the public benchmark (Table 3a, Sec.5.1. in the initial submission). 
2. Also, Table 3a, Sec.5.1. in the initial submission shows that the model trained with our NeuFace dataset significantly improves 3D reconstruction quality for extreme side view cases, where the temporal consistency alone is weak while the multi-view consistency effectively works. This suggests that even with the small portion of data involving multi-view consistency as the reviewer mentioned, the multi-view consistency helps.

Regardless of which bootstrapping (temporal or multi-view consistency) is, it does not alter our conclusion that the neural re-parameterization with bootstrapping is effective. Also, while the multi-view consistency is important, for scalability of the dataset, it is a practical consideration to leverage large-scale monocular video data.

We will consider processing and enlarging the portion of multi-view datasets, e.g., HUMBI [C5], to provide FLAME parameters when releasing the dataset.

[C5] Yu et al., HUMBI: A Large Multiview Dataset of Human Body Expressions. In CVPR 2020.

Reviewer comment (Experiments):
As mentioned in the paper, the proposed dataset contains a large amount of data, and the preliminary 3D mesh results generated based on DECA (EMOCA) may have errors. Have the authors considered how to filter out failed reconstruction results?

Author response:
Our NeuFace optimization is robust to the initial 3D face estimation so that small number of failure does not affect the performance. In the newly conducted experiment, we have compared the multi-view consistency of the final reconstruction on two scenarios: 1) Standard cases (use original images) and 2) Extreme cases (use perturbated images). For extreme cases, we randomly apply perturbation with large black boxes to the facial areas for 2~3 views in the multi-view videos to mimic significant corruption scenarios. 

| Loss configuration |  Standard cases (CVD)↓ | Extreme cases (CVD)↓ |
| --- | --- | --- |
| Weighted average (ours) | 0.103 | 0.113 |

The results show our optimization is robust to initial failure that occurs in the extreme cases.

Furthermore, to filter out the potentially remaining erroneous samples or outliers, we conducted 1) automatic filtering and 2) manual human verification when we constructed our NeuFace dataset. Please refer to Sec. B in the appendix of the initial submission.

Reviewer comment (Evaluation):
The quality of the reconstructed results for extreme facial expressions appears suboptimal. For instance, in Figure 5's top-left corner, where the open-mouth expression is depicted, the reconstruction of the mouth region does not seem consistent with the original input. Additionally, there appear to be imperfections in the reconstruction of closed-eye expressions.

Author response:
The failure cases stem from the limited performance of the off-the-shelf 2D landmark detection algorithm for extreme facial expressions. Since 2D landmark human annotations are relatively cheaper than any other signals, we think using better 2D landmarks can mitigate this limitation. We newly added this discussion in the revision (Sec. E, p. 24 in the Appendix).

Also note that these failure cases are rare, and we showed our annotation method’s effectiveness even in the presence of these rare cases, by enhancing the existing 3D reconstruction model with our dataset (Table 3. in Sec. 5.1.). We hope the reviewer acknowledges this perspective.

Reviewer comment (Experiments):
Given the analysis above, while the dataset's scale is certainly commendable, there seems to be room for improvement in terms of reconstruction accuracy. It might be worthwhile for the authors to consider utilizing such data as annotations for 3D landmarks rather than 3D mesh data. Additionally, have the authors explored the possibility of applying their proposed method to a different face model, such as the Basel Face Model (BFM), or investigating alternative pre-trained models instead of DECA or EMOCA?

Author response:
Thank you for the suggestion regarding the use of our dataset for improving the another task, 3D landmark reconstruction. As our dataset includes accurate pseudo 3D mesh data along with corresponding 3D landmarks, which indeed makes it a valuable resource for this purpose. Due to the limited time of the rebuttal, we aim to include the results of these experiments in the camera-ready version of our paper.

We think our losses and optimization are applicable to other 3DMMs and neural models. We chose FLAME since it is the most recent and popular 3D linear face model.

For the alternative pre-trained models, we can use our method and the same benefits would hold as long as the model predicts low dimensional parameters and the neural network is overparameterized (from Proposition 1 in Sec. 3.3.). Also, using a neural model with better accuracy would provide better initial points that can lead to the faster convergence of NeuFace optimization.

Reviewer comment (Novelty):
The theoretical novelty of face reconstruction using Neural network parameterization is incremental.

Author response:
We haven’t claimed the theoretical novelty itself. 

The theoretical context in Sec. A.2 helps the deep understanding of the analysis and algorithmic behavior of our proposed optimization system. We respectfully request the reviewer to consider our contributions below:

- Proposing the first and insightful concept of neural re-parameterized 3D face optimization, which mitigates the undesirable sparse gradient for face optimization (acknowledged by Reviewer B8S8).
- Providing a NeuFace-dataset, the first large-scale 3D face mesh pseudo-labels for existing large-scale 2D face video datasets. The dataset would benefit future research in this field (acknowledged by all the other reviewers).
- Extensive experiments to compare the quality and reliability of the proposed optimization and the dataset, and some empirical analysis.

Reviewer comment (Theory):
In the EM-like optimization, does the reconstruction always go to the right optional result?

Author response:
The NeuFace optimization and its losses are designed in a self-improving manner. Although the initial estimate of DECA could be noisy, the strong measurement of detected 2D landmarks and robust target supervision at each iteration gradually corrects the initial noisy predictions. The theoretical analysis shows that our neural re-parameterized optimization is highly likely to converge to global optima (Appendix Sec. A.2 of the initial submission). This hints that, with our neural parameterization, the optimization is robust to noisy initials and guarantees to exhibit at least stabler optimization behaviors than the compared baseline methods.

Reviewer comment (Theory):
The paper claims to investigate the reconstruction of image-aligned facial details on 3D meshes. However, the approach is based on optimizing the parameters of a 3DMM model, with the limitations of a linear model to represent fine facial details.

Author response:
We would like to clarify the scope of our approach. This is a misleading point due to the term we used, which we have revised and tone-downed the terms across the paper in this revision.

Our work aims to *facial details* of facial gestures and motions, not mesoscopic facial geometry of facial skin, which is the point the misleading happens at. Thus, we do not reconstruct the displacement-level facial details which is not the target of our work. We focus on 3D facial geometries complying with input facial gestures and motions. 

We revised the overall paper and tone-down our claim of “*reconstructing image-aligned facial details*” into “***reconstructing facial geometries, well complying with input facial gestures and motions***”, highlighted in pink. Thanks for the comment for specifying our scope better.

Reviewer comment (Experiments):
In the vertex accuracy evaluation experiments described in Sec. 3.4 and shown in Fig. 4, NeuFace optimization is compared with DECA and plain FLAME fitting. The paper does not describe the details of this experiment, specifically what are the train, validation and test data used for evaluating each algorithm. DECA results were produced by the plain pre-trained DECA model. It appears that NeuFace optimization, as described in Sec. 3.2, was trained with some part of VOCASET and evaluated on a different part of it, which does not seem like a fair comparison since DECA did not have the chance to see any part of VOCASET.

Author response:
We respectfully note that this concern misled by the reviewer's misunderstanding of the premise of our work. We hope the explanation below clearly addresses the points leading to the reviewer's misconcern.

First of all, the evaluation of our NeuFace test-time optimization is indeed fair, which we carefully designed, as all the other test-time approaches are. Specifically, 

1. Our NeuFace optimization, as elaborated in Sec. 3.2, is a TEST-TIME approach and does not involve training with train/validation/test splits. Our NeuFace step itself only exploits the input samples at test-time, but no additional data is used for the NeuFace itself.
2. We use the the pre-trained base model in NeuFace, e.g., DECA, for test-time fine-tuning on the input samples at test-time. Thus, the comparison with the pre-trained base model and our NeuFace (base model + NeuFace test-time optimization) is indeed in the fair setting; they have seen the exactly same data).
3. Our NeuFace is evaluated on each sample sequence independently. In Sec.3.4 and Fig. 4, for NeuFace optimization, we do not split the VOCASET into train/validation/test splits. We performed the test-time optimization for each sequence of VOCASET, starting from the pre-trained DECA checkpoint (pre-trained on VGGFace2 [C1], BUPT-Balancedface [C2], VoxCeleb2 [C3]), and detected 2D landmarks (we didn't use any GT information including mesh or 2D landmarks from VOCASET). Note that 2D landmark detections are not the ground-truth, as well. Thus, we claim that both DECA and ours do not have the chance to see any test dataset.
4. Likewise, our method is evaluated and compared for each test samples in each dataset (including VOCASET, MEAD, Voxceleb, and Celebv-HQ) independently without using their ground-truth mesh.

Our goal of experiments in Sec 3.4 and Fig. 4 was to compare the quality of pseudo 3D mesh annotations obtained from existing methods. In other words, the experiment in Sec 3.4 and Fig. 4 did not aim to compare the performance of each trained model on test datasets.

In the 3D face community, recent works [C4,C5] still naively use a pre-trained DECA or FLAME fitting (baseline) for annotator. In our experiments, we wanted to show how accurate and reliable NeuFace optimization is in generating pseudo 3D face annotations compared to existing methods.

To conclude, **NeuFace is not a learned model but a test-time optimization method; thus, it does not require training/validation/test sets.** Therefore, we confirm that our experiments were correct and fair, which we carefully designed.

[C1] Cao et al., VGGFace2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face & Gesture Recognition (FG) 2018.

[C2] Wang et al., Racial Faces in the Wild: Reducing Racial Bias by Information Maximization Adaptation Network. In ICCV 2019.

[C3] Chung et al., VoxCeleb2: Deep Speaker Recognition. In INTERSPEECH 2018.

[C4] Ng et al., Learning to listen: Modeling non-deterministic dyadic facial motion. In CVPR 2022.

[C5] Paraperas et al., Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos. In CVPR 2022.

Reviewer comment (Experiments):
For the same reason, the quantitative comparisons in Table 2 also seem unfair, since the optimizations in NEUFACE-*-datasets could see part of MEAD, VoxCeleb2 and CelebV-HQ data, whereas those involving DECA and EMOCA datasets did not. In Sec. A.3 and Table S1, it is shown that if DECA is refined on these datasets, the NME in MEAD reduces to 2.44, which is much lower than the 4.65 shown in Table 2.

Author response:
Our goal in Table 2 was to compare the quality of our pseudo 3D mesh annotations (NeuFace-D-dataset, NeuFace-E-dataset) with those obtained from existing methods (Base-dataset, DECA-dataset, EMOCA-dataset). 

**We emphasize that both DECA and ours do not have the chance to see any test dataset,** given the fact that 1) for Table 2, we do NOT train models using MEAD, VoxCeleb2, and CelebV-HQ at all, and 2) 2D landmark detections are not the ground-truth but estimated from the off-the-shelf module.

In Sec. A.3 and Table S1, we have compared the dataset quality obtained with our different TEST-TIME loss configurations. When we use only $\mathcal{L}_\text{2D}$ to refine the DECA estimations, the NME indeed improves than DECA’s initial prediction. However, as mentioned in [W2], NeuFace optimization is not a learned nor a fine-tuned model on MEAD, VoxCeleb2, and CelebV-HQ datasets. NeuFace optimization is a test-time optimization method for fitting 3D face for each video. Also, the 2D landmark detections itself is a pseudo ground-truth. Therefore, we conclude that our experiment for Table S1 was correct and fair as well.

Reviewer comment (Reproducibility):
Section 3.1 would benefit from an extended explanation by adding the dimension of each FLAME parameter. Also, the backbone network, e.g. DECA, not only estimates the 3DMM parameters, but also texture, lighting and a displacement map to model details outside the 3DMM linear model. While the approach discards the texture and lighting part, it is unclear what happens to the displacement map.

Author response:
Thanks for the thoughtful comment. We have added the detailed explanation in the revision (Sec. 3.1), highlighted in pink, as:

"We use FLAME, a renowned 3DMM, as a 3D face representation. 3D face mesh vertices $\mathbf{M}$ and facial landmarks $\mathbf{J}$ for $F$ frame videos can be acquired with the differentiable skinning: $\mathbf{M}, \mathbf{J}{=}\texttt{FLAME}(\boldsymbol{\mathbf{r}, \boldsymbol{\theta}, \boldsymbol{\beta}, \boldsymbol{\psi}})$, __where $\mathbf{r}\in{\mathbb{R}^{3}}$, $\boldsymbol{\theta}\in{\mathbb R}^{12}$, $\boldsymbol{\beta}\in{\mathbb R}^{100}$ and $\boldsymbol{\psi}\in{\mathbb R}^{50}$__ denote the head orientation, face poses, face shape and expression coefficients, respectively."

We think optimizing the mesoscopic detailed face geometry by re-parameterizing the displacement map is definitely an interesting future direction. 

As discussed in [W1], the face skin detail reconstruction is not our scope of this work, but “***reconstructing facial geometries, well complying with input facial gestures and motions.***” More importantly, we are motivated by the fact that there are lack of existing large-scale in-the-wild or multi-view video datasets that contain high-level face geometry, head motion, identity, and expressions that comply with input videos, in the community.

We believe our NeuFace dataset, which contains large-scale, diverse, natural, high-level human 3D face motion, would invigorate the 3D face community.

Reviewer comment (Reproducibility):
Section 3.2 does not explain how the ground-truth landmarks for equation 2 were obtained. Additionally, in the Multi-view consistency loss, there is no explanation of where the confidence values for each vertex come from.

Author response:
We used the off-the-shelf 2D landmark detection model, FAN [C6]. We also performed manual human verifications to reject the failure cases when constructing the NeuFace-dataset.

We have added this to the revision (Sec 3.2, second paragraph, page 4), highlighted in pink. Thanks for checking the details.

[C6] Face Recognition (FAN), https://github.com/1adrianb/face-alignment

We assign the confidence score per vertex by measuring the angle between the vertex normal and the camera ray. We set the vertices as invisible if the angle is larger than the threshold $\tau_a$, and the vertex has a deeper depth than $\tau_{z}$ (i.e., $z<\tau_z$). We empirically choose $\tau_a = 72^\circ$, $\tau_z=-0.08$. We have revised the paper accordingly (p5, Sec 3.2), highlighted in pink.

Reviewer comment (Experiments):
The experiments section must clearly explain what is the train/validation/test data used in each experiment and confirm that the results shown in the accuracy evaluation in Fig. 4 and Table 2 are correct and fair.

Author response:
We respectfully note that this concern misled by the reviewer's misunderstanding of the premise of our work. We hope the explanation below clearly addresses the points leading to the reviewer's misconcern.

First of all, the evaluation of our NeuFace test-time optimization is indeed fair, which we carefully designed, as all the other test-time approaches are. Specifically, 

1. Our NeuFace optimization, as elaborated in Sec. 3.2, is a TEST-TIME approach and does not involve training with train/validation/test splits. Our NeuFace step itself only exploits the input samples at test-time, but no additional data is used for the NeuFace itself.
2. We use the the pre-trained base model in NeuFace, e.g., DECA, for test-time fine-tuning on the input samples at test-time. Thus, the comparison with the pre-trained base model and our NeuFace (base model + NeuFace test-time optimization) is indeed in the fair setting; they have seen the exactly same data).
3. Our NeuFace is evaluated on each sample sequence independently. In Sec.3.4 and Fig. 4, for NeuFace optimization, we do not split the VOCASET into train/validation/test splits. We performed the test-time optimization for each sequence of VOCASET, starting from the pre-trained DECA checkpoint (pre-trained on VGGFace2 [C1], BUPT-Balancedface [C2], VoxCeleb2 [C3]), and detected 2D landmarks (we didn't use any GT information including mesh or 2D landmarks from VOCASET). Note that 2D landmark detections are not the ground-truth, as well. Thus, we claim that both DECA and ours do not have the chance to see any test dataset.
4. Likewise, our method is evaluated and compared for each test samples in each dataset (including VOCASET, MEAD, Voxceleb, and Celebv-HQ) independently without using their ground-truth mesh.

Our goal of experiments in Sec 3.4 and Fig. 4 was to compare the quality of pseudo 3D mesh annotations obtained from existing methods. In other words, the experiment in Sec 3.4 and Fig. 4 did not aim to compare the performance of each trained model on test datasets.

In the 3D face community, recent works [C4,C5] still naively use a pre-trained DECA or FLAME fitting (baseline) for annotator. In our experiments, we wanted to show how accurate and reliable NeuFace optimization is in generating pseudo 3D face annotations compared to existing methods.

To conclude, **NeuFace is not a learned model but a test-time optimization method; thus, it does not require training/validation/test sets.** Therefore, we confirm that our experiments were correct and fair, which we carefully designed.

[C1] Cao et al., VGGFace2: A dataset for recognising faces across pose and age. In International Conference on Automatic Face & Gesture Recognition (FG) 2018.

[C2] Wang et al., Racial Faces in the Wild: Reducing Racial Bias by Information Maximization Adaptation Network. In ICCV 2019.

[C3] Chung et al., VoxCeleb2: Deep Speaker Recognition. In INTERSPEECH 2018.

[C4] Ng et al., Learning to listen: Modeling non-deterministic dyadic facial motion. In CVPR 2022.

[C5] Paraperas et al., Neural Emotion Director: Speech-preserving semantic control of facial expressions in "in-the-wild" videos. In CVPR 2022.

Reviewer comment (Theory):
Often the base pre-trained model fails dramatically. In such cases, averaging the estimated mesh with others in the regularization terms would ruin the optimization, since the average operation is not robust. An alternative robust operation, such as median, may improve the results.

Author response:
Following the reviewer's suggestion, we have compared the mean, median, and weighted average (ours) in the multi-view loss. We have compared the results on two scenarios: 1) Standard cases (use original images) and 2) Extreme cases (use perturbated images). For extreme cases, we randomly apply perturbation with large black boxes to the facial areas for 2~3 views in the multi-view videos to mimic significant corruption scenarios. 

| Loss configuration |  Standard cases (CVD)↓ | Extreme cases (CVD)↓ |
| --- | --- | --- |
| Average | 0.106 | 0.124 |
| Median | 0.104 | 0.112 |
| **Weighted average (Ours)** | 0.103 | 0.113 |
- Results of standard cases: All three methods (mean, median, and weighted average) showed similar performance, with no marked difference in optimization results evaluated on the MEAD subset.
- Results of extreme cases: The median outperforms the mean in perturbed MEAD data.
- Our Method: Our methodology, which employs a weighted average grounded in the confidence scores from multiple view vertices, not only shows favorable performance over the simple average but also aligns closely with the median's results. This indicates that our approach maintains robustness in extreme cases, akin to the median operation highlighted by the reviewer.

The reviewer's suggestion about the median was insightful. Both our weighted average and the median work for extreme scenarios. One difference is that our visibility-based weighted average has a control parameter to adjust the robustness through hyperparameters (as in Q2-2.).