Paper: An Efficient Multi-Task Transformer for 3D Face Alignment (ICLR-2024)

Reviewer comment (Presentation):
The title of the paper is not appropriate in my opinion. The title emphasizes a face alignment contribution, while the content focuses more on face landmarks detection.

Author response:
Our main task is 3D alignment, while landmark detection is only an auxiliary task. The major goal is to predict the 3DMM parameters to reconstruct 3D head shape. As shown in Figure 1, our model mainly consists of a 3DMM parameter regression branch and an auxiliary facial landmark detection branch. The 3D alignment relies on 3DMM parameters regression branch which can regress a set of 3DMM parameters and reconstruct the 3D head shape by a differentiable FLAME decoder. In the paper, Figure 1 shows the overall 3D alignment pipeline, Sec. 3.1 introduces the proposed 3D alignment framework, and Sec. 3.4 describes the overall losses for both landmark detection, as well as 3D shape reconstruction and reprojection. Besides, the Euler angle loss is proposed in Sec. 3.4.1 for improving the 3D alignment via 3D pose estimation. Thank you for your comment. We will further improve the presentation of Sec. 3.1 for the whole 3D alignment framework.

Reviewer comment (Experiments):
Parameters of a 3DMM are regressed in this work. However, it is not clear how much the choice of the 3DMM impacts on the results. Did the authors try with different 3DMMs?

Author response:
Thank you for your valuable suggestion. The 3DMM parameters used in DAD-3DNet and the proposed Trans3DHead are adapted to FLAME, which are extensions of the standard 3DMM parameters. The number of the extended 3DMM parameters used in the final Trans3DHead is 413, including 300 shape parameters, 100 expression parameters, 3 jaw parameters, 6 rotation parameters, 3 translation parameters, and 1 scale parameter to control FLAME, which allows to use the first few parameters of the 300 shape parameters, 100 expression parameters, and 3 jaw parameters to express the shape and expression. 

We have conducted experiments with two additional 3DMM parameter settings. The first one is with the same number of 3DMM parameters as in SynergyNet (40 shape parameters, 10 expression parameters without jaw parameter), while the second one is used in RingNet (100 shape parameters, 50 expression parameters without jaw parameter). Keeping the original setting of 6 rotation parameters, 3 translation parameters, and 1 scale parameter in FLAME, we use two parameter settings of 60 parameters and 160 parameters to train our Trans3DHead and DAD-3DNet on the DAD-3DHeads dataset, respectively. 

As shown in Table 8 in the appendix, regressing more parameters is helpful for better reconstructing 3D head shape for both DAD-3DNet and the proposed Trans3DHead. Besides, the proposed Trans3DHead is superior to DAD-3DNet in all three parameter settings. When only regressing 60 parameters, our model shows more significant advantages over DAD-3DNet, which also indicates that the cross-attention mechanism in Trans3DHead is effective to achieve the information communication among different elements of the 3DMM parameters.

Reviewer comment (Experiments):
In the ablation study it is not clear the impact of the 3DMM on the results.

Author response:
Thank you for your valuable suggestion. The 3DMM parameters used in DAD-3DNet and the proposed Trans3DHead are adapted to FLAME, which are extensions of the standard 3DMM parameters. The number of the extended 3DMM parameters used in the final Trans3DHead is 413, including 300 shape parameters, 100 expression parameters, 3 jaw parameters, 6 rotation parameters, 3 translation parameters, and 1 scale parameter to control FLAME, which allows to use the first few parameters of the 300 shape parameters, 100 expression parameters, and 3 jaw parameters to express the shape and expression. 

We have conducted experiments with two additional 3DMM parameter settings. The first one is with the same number of 3DMM parameters as in SynergyNet (40 shape parameters, 10 expression parameters without jaw parameter), while the second one is used in RingNet (100 shape parameters, 50 expression parameters without jaw parameter). Keeping the original setting of 6 rotation parameters, 3 translation parameters, and 1 scale parameter in FLAME, we use two parameter settings of 60 parameters and 160 parameters to train our Trans3DHead and DAD-3DNet on the DAD-3DHeads dataset, respectively. 

As shown in Table 8 in the appendix, regressing more parameters is helpful for better reconstructing 3D head shape for both DAD-3DNet and the proposed Trans3DHead. Besides, the proposed Trans3DHead is superior to DAD-3DNet in all three parameter settings. When only regressing 60 parameters, our model shows more significant advantages over DAD-3DNet, which also indicates that the cross-attention mechanism in Trans3DHead is effective to achieve the information communication among different elements of the 3DMM parameters.

Reviewer comment (Experiments):
For the qualitative comparisons, only DAD-3DNet is compared with. From table 5, DAD-3DNet is already worse than the proposed method. While SynergyNet, which performs better than the proposed method, is not compared with in the visual demos.

Author response:
Thank you for the comment. Please refer to the Common Questions and Responses 1 for the comparison to SynergyNet. 

In the 3D head alignment task, the DAD-3DNet is the most relevant work and the closed competitor to the proposed Trans3DHead, so the most important experimental comparisons of this work are shown in Table 1. Thank you for the encouraging statement that “DAD-3DNet is already worse than the proposed method”. 

Anyway, including SynergyNet for visual comparison is indeed a good idea. Thank you for your suggestion. Besides the new Figure 10, to further illustrate the difference between the SynergyNet and the proposed Trans3DHead, we have additionally visualized some examples of head pose estimation predicted by the SynergyNet and the proposed Trans3DHead on the AFLW2000-3D dataset, as shown in Figure 11 in the appendix. It can be observed that the SynergyNet performs slightly better in predicting pitch and roll, while the proposed model has more advantages in handling side faces.

Reviewer comment (Writing):
In section 3.2, the description about QA memory is not super clear. Does the proposed method use more features maps to trade for a lower resolution? What is the benefit in doing this?

Author response:
Thank you for your comment. Using low-resolution feature maps often leads to a decrease in accuracy. Based on an observation, we find that in Transformers the queries will act differently if the memory is of higher resolutions. Therefore, QAMem is proposed to simulate the working mechanism of high-resolution feature maps in the actual low-resolution feature maps we used to improve the accuracy. In practice, the QAMem layer can be simply implemented as a convolutional layer with 1 $\times$ 1 kernel, 1 stride, and N groups. Note that we do not use more feature maps to trade for a lower resolution. In our implementation of QAMem in Eq. (3), the feature map channels are the same $d$ channels, but N groups of 1 $\times$ 1 convolutions are applied to transform the $d$ channels per query. As a result, QAMem provides discriminative values for different queries with marginal additional computation, and enhances discriminative abilities of the model on using low-resolution feature maps. 

Note also that the implementation of applying high-resolution feature maps usually uses additional network for up-sampling, such as the commonly used FPN structure where the parameters, computation, and memory requirement are significantly larger.

Reviewer comment (Theory):
Due to large variation of head poses and face shape, it might not be a good idea to compute the average face and predict the residuals.

Author response:
The average face provides a general initial topology of the landmark detection with which the predicting task will be easier with small or not too large head poses. When encountering large head poses, the decoders with sufficient training are also able to output larger residual to correct the initial landmarks. Thank you for pointing out this. To address large variation of head poses and face shape better, a better way could be considering multiple average faces, which will be our future study.

Reviewer comment (Theory):
The paper proposes a Euler Angles loss, but from figure 4, it requires an estimation of the Euler angles from predicted 3DMM parameters. Therefore another module needs to be introduced to do the estimation. This module introduces additional errors and might not benefit the supervision for 3DMM parameters.

Author response:
The estimation of the Euler angles from the predicted 3DMM parameters is a deterministic matrix calculation, with no additional network or learnable parameters in introducing errors. Specifically, the 3DMM parameters used in DAD-3DNet and the proposed Trans3DHead are adapted to FLAME, which are extensions of the standard 3DMM parameters. The number of the extended 3DMM parameters regressed in the final Trans3DHead is 413, including 403 3DMM shape parameters, 6 rotation parameters, 3 translation parameters, and 1 scale parameter. Then, the Euler angles are estimated by a deterministic matrix calculation from the predicted 6 rotation parameters, because any orientation can be expressed as a composition of rotations. In our implementation, we use SciPy, a scientific computation Python library, through scipy.spatial.transform.Rotation.as_euler().

Reviewer comment (Evaluation):
From table 3, the QA module seems not to improve the accuracy compared with baseline.

Author response:
Thank you for your comment. The QAMem module is mainly proposed for the landmark detection subtask, so it has a relatively larger improvement on NME. For other metrics, although no improvements are observed, they are quite comparable. Note that the baseline is already a very good baseline of the proposed two-branch Transformer based method.

Reviewer comment (Evaluation):
From table 5, the proposed method is not quantitatively better than SynergyNet 3DV 2021.

Author response:
Thank you for the comment. Please refer to the Common Questions and Responses 1 for the comparison to SynergyNet.

Reviewer comment (Theory):
Can the authors give more explanation and proof about the QA module and its effectiveness? Does the module use more feature maps to trade for lower resolution?

Author response:
Thank you for your comment. Using low-resolution feature maps often leads to a decrease in accuracy. Based on an observation, we find that in Transformers the queries will act differently if the memory is of higher resolutions. Therefore, QAMem is proposed to simulate the working mechanism of high-resolution feature maps in the actual low-resolution feature maps we used to improve the accuracy. In practice, the QAMem layer can be simply implemented as a convolutional layer with 1 $\times$ 1 kernel, 1 stride, and N groups. Note that we do not use more feature maps to trade for a lower resolution. In our implementation of QAMem in Eq. (3), the feature map channels are the same $d$ channels, but N groups of 1 $\times$ 1 convolutions are applied to transform the $d$ channels per query. As a result, QAMem provides discriminative values for different queries with marginal additional computation, and enhances discriminative abilities of the model on using low-resolution feature maps. 

Note also that the implementation of applying high-resolution feature maps usually uses additional network for up-sampling, such as the commonly used FPN structure where the parameters, computation, and memory requirement are significantly larger.

Reviewer comment (Reproducibility):
In section 3.4.1, how are Euler angles estimated from 3DMM parameters? Is a neural network used here?

Author response:
The estimation of the Euler angles from the predicted 3DMM parameters is a deterministic matrix calculation, with no additional network or learnable parameters in introducing errors. Specifically, the 3DMM parameters used in DAD-3DNet and the proposed Trans3DHead are adapted to FLAME, which are extensions of the standard 3DMM parameters. The number of the extended 3DMM parameters regressed in the final Trans3DHead is 413, including 403 3DMM shape parameters, 6 rotation parameters, 3 translation parameters, and 1 scale parameter. Then, the Euler angles are estimated by a deterministic matrix calculation from the predicted 6 rotation parameters, because any orientation can be expressed as a composition of rotations. In our implementation, we use SciPy, a scientific computation Python library, through scipy.spatial.transform.Rotation.as_euler().

Reviewer comment (Novelty):
The contributions seem to be a combination of deep learning tricks like multi-task structure, QAMem module, MARR and Euler Angles Loss, the improvement of each module seems to be incremental and the combination of these incremental contributions do not become a significant contribution.

Author response:
Thank you for your review but we do not agree. This work is well motivated to solve respective problems instead of using deep learning tricks to improve the performance. First, this is the first work in regressing 3DMM parameters through Transformers, which can help information communication among 3DMM parameters via attention mechanism in joint estimation. With this architecture, the proposed multi-task framework for 3D head alignment simultaneously predicts 3D head shape and 2D landmarks. This not only facilitates applications by directly providing multiple available outputs, but also helps mutual improvements between the two branches. Our experimental findings show that with this design the proposed method is more robust with reducing 3DMM parameters (see A.1.6 and Table 8 in appendix), thanks to the unified Transformer architecture for two branched tasks.

Second, the proposed lightweight QAMem module is an interesting and valuable idea (supported by Reviewers QjYA and PGZe) for dense prediction tasks requiring high-resolution feature maps. It ingeniously uses separate memories with low-resolution feature maps and designs an efficient implementation to get rid of the high-resolution feature map requirement. The QAMem design enables improved accuracy with minimal additional computational burden. 

These are the two major innovations in this paper. We do not agree that they can be simply categorized into deep learning tricks or incremental combinations.

Reviewer comment (Experiments):
The ablation study in Table 3 don't support the effeictiveness of proposed modules as well. For example, the performance of Cham Dist for Trans3DHead is not as good as the baseline.

Author response:
Thank you for the comment. The measurements and comparisons in Table 3 are comprehensive, including evaluations of not only 3D face shape prediction (Chamfer distance), but also face landmark detection (NME), head pose estimation (pose error), and 3D head shape estimation (Z5). In DAD-3DHeads benchmark, Chamfer distance is used for measuring the 3D shape of human face, and the proposed method achieves quite comparable results with the baseline in terms of Chamfer distance (2.796 vs. 2.791). Except this the proposed method outperforms the baseline in all other metrics. Especially, the proposed model performs better than the baseline with Z5 (0.9585 vs. 0.9578). As Z5 is used for measuring 3D head shape, it is more important than Chamfer distance for 3D face shape only.