Paper: Comprehensive Comparison between Vision Transformers and Convolutional Neural Networks for Face Recognition Tasks (ICLR-2024)

Reviewer comment (Novelty):
This paper seems more like an experimental evaluation report, primarily focusing on the organization of test numbers.

Author response:
We extend our gratitude for your thoughtful evaluation of our paper on the comparative analysis of Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) for face recognition tasks. Your insights are invaluable in refining our study.

We appreciate your recognition of the thorough experimentation conducted across diverse network architectures and datasets, providing insights into the performance of these models in face recognition applications. However, we understand your concerns regarding the presentation and contribution aspects of our work.

We acknowledge the need to move beyond a mere presentation of experimental results and strive to offer deeper insights and novel perspectives derived from our findings. In our revised manuscript, we will extrapolate and discuss new insights garnered from the results, exploring avenues for fresh perspectives in training methodologies, evaluation techniques, and potential optimizations specific to face recognition tasks. Our goal is to not solely present numerical outcomes but to derive insights that could impact the design and implementation of network structures in face recognition applications.

We understand the importance of expanding the narrative beyond a pure experimental evaluation and will focus on extrapolating new insights to elucidate deeper implications and potential avenues for further exploration within the field of face recognition.

Your feedback is immensely valuable, and we are committed to addressing the outlined weaknesses and enhancing the manuscript to provide richer insights and broader perspectives derived from our experimental evaluations.

We're actively implementing your suggestions to refine our manuscript comprehensively. Our revised version will be uploaded before the discussion period ends.

Reviewer comment (Evaluation):
It would be beneficial for the authors to extrapolate some new insights from these figures, potentially providing fresh perspectives on training or evaluation in face recognition.

Author response:
We sincerely appreciate the time and effort you dedicated to evaluating our paper and providing constructive feedback. Your insights have played a crucial role in refining our work.

In response to your valuable feedback, we've made concerted efforts to extrapolate new insights from our experimental results. Specifically, in Section 3 and more prominently in Section 3.4, we've delved deeper into the implications of our findings, providing a more nuanced discussion of the performance ViTs in comparison to CNNs for face recognition tasks. While we might not have uncovered some new insights, we believe our comparative analysis contributes to the ongoing discourse and helps illuminate the landscape of ViTs and CNNs in face recognition, potentially guiding future research directions.

Throughout the manuscript, we've implemented minor improvements to enhance the overall clarity and coherence of our presentation. We aim to strike a balance between the detailed organization of experimental results and the derivation of meaningful insights that extend beyond a mere numerical account.

Your feedback has been instrumental in shaping the direction of our revisions, and we remain dedicated to providing a manuscript that not only presents thorough experimental evaluations but also offers valuable perspectives for the design and optimization of network structures in face recognition applications.

Reviewer comment (Experiments):
The benchmark compared the performance of 6 vanilla networks trained for face recognition tasks, whereas there are many dedicated face recognition methods and pipelines including face detection and alignment. The face recognition field probably cares more about the end-to-end performance of the whole face recognition pipeline.

Author response:
We acknowledge your point regarding the scope of our work within the larger context of face recognition methodologies. While our study primarily focuses on the individual performance of network architectures, we agree that a comprehensive evaluation might include the entire face recognition pipeline, encompassing aspects like face detection and alignment. We aimed to delve specifically into the comparison of ViTs and CNNs in isolation, highlighting their standalone effectiveness in face recognition tasks.

Reviewer comment (Evaluation):
The Face Recognition Technology Evaluation (FRTE) is likely the most thorough evaluation for the industry, testing the performance of binary programs provided by different FR vendors on a blind set with no limitations on the training dataset or other factors.

Author response:
Regarding your suggestion about incorporating the FRTE evaluation, we sincerely appreciate the consideration. We carefully reviewed the suggested FRTE methodology, recognizing its significance in assessing the end-to-end performance of face recognition systems. However, due to the time constraints within the discussion period and the considerable preparation required to conform to the FRTE format, it was challenging to integrate this evaluation into our current study. We aimed to provide a focused comparison between ViTs and CNNs in this iteration, focusing on their individual performances in face recognition tasks across various datasets.

Reviewer comment (Novelty):
Important references are missing, including:
- DeepFace: closing the gap to human-level performance in face verification, CVPR 2014
- Deep learning face representation by joint identification-verification, NIPS 2014
- Comparing vision transformer and convolutional neural networks for image classification: a literature review, 2023.

Author response:
Regarding the missing references, we appreciate your suggestions and will duly consider incorporating these important references to enrich the discussion and further contextualize our findings within the broader landscape of face recognition literature.

Reviewer comment (Experiments):
The performance appears saturated on some face recognition datasets, and the results of several cases may affect the benchmark. The reviewer questions whether there is any more challenging test set for face recognition.

Author response:
Concerning the saturation of performance on certain datasets, we understand your concern. While we acknowledge the observation of performance saturation on the VGG-Face 2 dataset, it's crucial to note that across the other selected datasets, we did not encounter similar saturation phenomena. We selected these datasets intentionally to encompass a range of challenges, including people diversity, varying distances from the camera, and occlusions caused by masks and glasses. However, we acknowledge the possibility of additional datasets that might present further challenges, and we will explore and consider including more demanding test sets for a more comprehensive evaluation.

Reviewer comment (Reproducibility):
No uncertainty (e.g., standard deviations across random seeds) are presented for the different results. This may be ok since the models are evaluated across many different datasets, but in that case the seed should be fixed and that should be stated.

Author response:
Regarding the absence of standard deviations across random seeds, we recognize the importance of robustness in presenting results. In response, we will ensure to clarify our methodology, including fixed seed settings, particularly across diverse datasets, to provide a more robust evaluation framework.

Reviewer comment (Experiments):
Missing a comment on how the hyperparameters common for all architectures were selected (e.g., following another paper's set up, just as standard hyperparameters, etc). It is important that they were not selected to optimize a specific architecture, and it would be good to convince the reader about this.

Author response:
Concerning the hyperparameters, we apologize for the oversight in not explicitly detailing their selection process. We will provide a clear rationale for their choice, ensuring they were not optimized for specific architectures, thus assuring readers of the neutrality of our approach.

Reviewer comment (Writing):
I would avoid referring to my own paper as having 'paramount significance' (strong wording, verging on over-selling). The number of times the word 'remarkable/y' (6) is a bit exaggerated as well.

Author response:
We acknowledge the critique regarding the tone and phrasing within the manuscript and will revise the language to maintain a balanced and accurate representation of our work without overstating its significance.

Reviewer comment (Evaluation):
No explanation is offered for why the ResNet surpassed the ViT in Fig. 7b.

Author response:
Additionally, in response to the question about the figure where VGG (we understand the reviewer meant VGG and not ResNet, as this is the network that outperformed ViT in this scenario) outperforms ViT (Fig. 7b), we've incorporated an explanation in the last paragraph of section 3.4, addressing the reasons behind this observed occurrence.

Reviewer comment (Presentation):
It could be nice with a table summarizing the 5 datasets and tasks.

Author response:
Additionally, we will consider the inclusion of a summarized table for datasets and tasks and provide more accessible information in figure captions for improved clarity.

Reviewer comment (Writing):
Page 5, "is bounded between 0 (the worst measure of separability) and 1 (a perfect measure of separability), with 0.5 indicating that a network has no class separation capability whatsoever." >> if 0.5 already has 0 separability, what happens between 0.5 and 0? Maybe rephrase

Author response:
We acknowledge the reviewer's observation regarding the potential confusion in the phrasing of the sentence about the Area Under the Curve (AUC). In our revised manuscript, we have rephrased this statement for better clarity. Now, the sentence explicitly articulates that an AUC of 0 denotes the worst measure of ~separability~ performance, an AUC of 1 signifies the best measure of ~separability~ performance, and an AUC of 0.5 indicates that a network lacks any class separation capability.

Reviewer comment (Experiments):
3.3: it is not clear to me if you use the checkpoint from the best epoch or from the 25th (last) epoch for the test results in Table 2? If you just use the last epoch (which I suspect since you say that each model has been trained for 25 epochs), it would be more informative to report the validation accuracy at this epoch (for the model you actually use.)

Author response:
We want to clarify that the reported results in both Table 1 and Table 2 stem from the checkpoint of the best epoch. Although we ensured that all models completed 25 epochs during training, the values presented in these tables reflect the accuracies obtained using the weights from the best epoch. As explicitly mentioned in the manuscript, the accuracy values in Table 1 correspond to the highest achieved on both the training and validation sets during training. This distinction ensures that the reported accuracies represent the models' best performance on the given datasets.

Reviewer comment (Experiments):
This work completely disregards other popular works in the face recognition literature. The authors compare general-purpose CNNs and ViT_B32 when efficient face recognition-specific approaches are already available such as MobileFaceNet (Chen et al. 2018), ShuffleFaceNet (Martínez-Diaz et al., 2019), VarGFaceNet (Yan et al., 2019), GhostFaceNets (Alansari et al., 2022), EdgeFace (Geroge et al., 2023), among others.

Author response:
We sincerely appreciate the insightful feedback provided on our manuscript evaluating Vision Transformers (ViTs) against Convolutional Neural Networks (CNNs) for face recognition tasks. Your detailed assessment has been immensely helpful in refining our study.

We acknowledge the rich landscape of face recognition literature, encompassing specialized approaches like MobileFaceNet, ShuffleFaceNet, VarGFaceNet, among others, and recent studies on transformers for face recognition. While our paper might not extensively delve into these specific methodologies, our primary intent was not to challenge or replicate the state-of-the-art in face recognition but rather to present a comparative analysis between ViTs and CNNs within this domain.

In response to your feedback, we will include explicit mentions and discussions of previous studies utilizing both transformers and CNNs for face recognition. We aim to highlight their significance within the context of our analysis, thereby elucidating the specific contributions of our work.

We are committed to enhancing the clarity of our paper by explicitly delineating the objectives and scope of our comparative study. Additionally, we will include a broader acknowledgment of pertinent literature, thus enriching the context and relevance of our findings within the broader landscape of face recognition research.

Reviewer comment (Novelty):
The work does not mention previous studies on transformers for face recognition (Zhong et al., 2021) and part-based face recognition with vision transformers (Sun et al., 2022), for example.

Author response:
Thank you for your thorough evaluation of our manuscript. We've diligently worked on incorporating your insightful suggestions into our revised version. In response to your keen observations regarding the omission of pertinent literature, we have made explicit mentions and discussions of key works you highlighted, specifically within the introduction section of the manuscript.

The updated version now explicitly references studies utilizing both transformers and CNNs for face recognition, including MobileFaceNet, ShuffleFaceNet, VarGFaceNet, among others. These additions serve to contextualize and emphasize the significance of these methodologies within the scope of our comparative analysis between Vision Transformers (ViTs) and Convolutional Neural Networks (CNNs) for face recognition tasks.

Your feedback has been key in refining the depth and breadth of our analysis, contributing significantly to the contextual richness of our study. We deeply appreciate your efforts in guiding us toward a more comprehensive and informative manuscript.

Reviewer comment (Experiments):
The datasets described are not divided into scenarios and do not include relevant challenging datasets (e.g. using TinyFace (Zheng et al., 2018) and SurvFace (Zheng et al., 2018) to complement low resolution comparisons with SCface).

Author response:
Regarding the choice of datasets, our intention was to provide a diverse set covering a range of challenges pertinent to real-world scenarios, such as people diversity, varying distances from the camera, and occlusions caused by masks and glasses. While we acknowledge the importance of additional challenging datasets like TinyFace and SurvFace, our selected datasets were chosen deliberately to address these challenges, even if not explicitly listed in your suggestions.