Paper: FLNERF: 3D FACIAL LANDMARKS ESTIMATION IN NEURAL RADIANCE FIELDS (ICLR-2024)

Reviewer comment (Evaluation):
A NERF model is normally constructed for each 3D object to render. This limitation seems to apply also to this work. This represents quite a drawback for the proposed solution in that a different NERF model should be constructed for each identity. This drastically reduces the generality of the approach and results in a substantially increased computational effort that appears to be not much compatible with a problem of landmarks detection.

Author response:
For Q1, we do need to train a seperate NeRF for each inference. However, instant-ngp already accelerated NeRF training from hours to seconds. Those 2D landmarks detections followed by triangulation or single-image 3D landmarks detections are also not immediate, taking from 30 seconds to 2 minutes. We believe as NeRF being more and more popular, more NeRF training acceleration techniques will be proposed. Also, there are many downstream tasks that do not require super fast face landmarks detection speed. For example, people may be able to produce his/her own face animations enabled by our FLNeRF + modified MoFaNeRF, as shown in our paper and video. Many mobile apps could already reconstruct NeRFs using mobile phone cameras in seconds. Imagine people could possibly make vivid face animations totally on their own, driving a cartoon face, a super man's face, or a celebrity's face by simply performing corresponding expressions themselves and feed the face NeRFs to our FLNeRF + modified MoFaNeRF pipeline, without experts drawing every frame.

Reviewer comment (Experiments):
Based on the above, the comparison with other methods that do not incur in such limitation is not completely fair in my opinion. Authors should at least clarify this point.

Author response:
For Q2, please refer to Sec. 2.1, Sec. 2.2 and Sec. 2.3 (Table 1-5) in our supplementary material. They provide detailed analysis of why performing 3D landmarks estimation on NeRFs is a better choice. Our estimation process can be effectively performed in a single forward pass through the model. As a result, the inference time for each expression is less than one second (tested on the GTX 1080), which is comparable to the magnitude of the 2D landmarks detector.

Reviewer comment (Evaluation):
The evaluation is performed with 5 identities from the test dataset from FaceScape. While this experiment provides some information, a sound comparison should include several other benchmark datasets in the literature and a widely used metric, such as the mean, median, and standard deviation reconstruction errors.

Author response:
For profoundness of comparisons, we also provided quantitative results in terms of adaptive average Wing loss and MSE. Please refer to Sec. 2.1 and Sec. 2.2 in the supplementary material, which show comparison in terms of more metrics in Table 1, Table 2, Table 3, Table 4 and Table 5 in the supplementary material. We conducted these comparisons using 20 different expressions for each identity, resulting in 100 diverse test cases. Acknowledging your insightful suggestion, we agree that testing on additional datasets would be beneficial. However, since NeRF and its variants are still young, and few people recognized its potential in 3D face landmarks estimation, FaceScape remains the only dataset to our knowledge providing multi-view face images with ground truth 3D face landmarks. Nonetheless, our tests on face NeRFs reconstructed by EG3D from single in-the-wild images demonstrate potential applicability to other datasets from another perspective.

Reviewer comment (Experiments):
There is doubt about whether the comparison is fair, since FLNeRF was trained with a train set from FaceScape, whereas competing approaches seem to have been trained with different datasets.

Author response:
For fairness of comparisons, we compared FLNeRF which estimates 3D face landmarks from face NeRFs, with triangulation and average single-image 3D landmarks estimation, as shown in Table 1 in main paper, and Table 1-5 in supplementary mateiral. For triangulation, we used state-of-the-art 2D face landmarks detection methods to detect 2D landmarks on single images, and then we performed triangulation to obtain 3D landmakrs. For average single-image 3D landmarks estimation, we performed 3D landmarks estimation on single images, and then take the average. As elaborated in Sec. 3.5 in main paper, the number of images to perform 2D landmarks detection followed by triangulation, the number of images to perform average 3D single-image 3D landmarks estimation, and the number of images to train a face NeRF used by FLNeRF, are all the same. Therefore, our comparison is fair in terms of the number of input images.

Reviewer comment (Experiments):
In 2D datasets, landmarks around the jaw do not have a fixed location, but rather represent the face occluding contour, so a reconstruction from their correspondences does not make much sense.

Author response:
As pointed out in Sec. 3.5 in our main paper, definitions of landmarks on cheeks are indeed different from FaceScape. This conforms to your comment that landmarks in those 2D datasets around jaw do not have a fixed location, but represent the face occluding contour. However, when performing comparisons, as shown in Table 1 in our main paper, and Sec. 2.1 and Sec. 2.2 (Table 1-5) in our supplementary material, we only show quantitative comparisons in those landmarks around mouth, eyes, and nose whose definitions are the same. Therefore, our comparison is fair in terms of landmarks definition.

Reviewer comment (Theory):
The authors should elaborate more on the complexity of estimating a detailed NeRF from an image compared to a set of facial landmarks, and discuss the advantages of estimating the landmarks from the NeRF rather than from the image.

Author response:
Quantitative and qualitative comparisons show that our FLNeRF which estimates 3D landmarks on NeRF significantly outperforms 2D landmarks estimation followed by triangulation, and average single-image 3D face landmarks estimation. Please refer to Sec. 2.2 and Sec. 2.3 in our supplementary material for reasons why our method produces better results. Hope this could answer your question regarding the advantage of estimating 3D landmarks on NeRF over estimating 3D landmarks on images.

Reviewer comment (Experiments):
There is a question about whether the comparison in Table 1 is fair.

Author response:
For fairness of comparisons, we compared FLNeRF which estimates 3D face landmarks from face NeRFs, with triangulation and average single-image 3D landmarks estimation, as shown in Table 1 in main paper, and Table 1-5 in supplementary mateiral. For triangulation, we used state-of-the-art 2D face landmarks detection methods to detect 2D landmarks on single images, and then we performed triangulation to obtain 3D landmakrs. For average single-image 3D landmarks estimation, we performed 3D landmarks estimation on single images, and then take the average. As elaborated in Sec. 3.5 in main paper, the number of images to perform 2D landmarks detection followed by triangulation, the number of images to perform average 3D single-image 3D landmarks estimation, and the number of images to train a face NeRF used by FLNeRF, are all the same. Therefore, our comparison is fair in terms of the number of input images.

Reviewer comment (Experiments):
There is a question about whether a test on a single image provides sufficient experimental support to conclude that FLNeRF can be generalized to estimate 3D face landmarks on 2D in-the-wild images.

Author response:
We are grateful for your suggestions on generalization. Accordingly, we will revise our terminology to 'generalization to in-the-wild NeRF' for greater accuracy. The reason for showing estimations results by our FLNeRF on face NeRFs reconstructed by EG3D from single images is to show that our FLNeRF could produce reasonable results even given face NeRFs of low quality and background. NeRFs are becoming more and more popular these days. Even some mobile apps could reconstruct NeRF from mobile phone cameras. Currently those mobile apps require multi-view photo taking. Given that our FLNeRF could achieve reasonable results on low-quality NeRFs reconstructed from single images, we believe our technology could benefit users who could use their mobile apps to reconstruct high-quality in-the-wild NeRFs from multi-view images. Users could perform fancy downstream tasks with our technology, such as face animation shown in our paper, enabled by our FLNeRF + modified MoFaNeRF.