Paper: Enhanced Face Recognition using Intra-class Incoherence Constraint (ICLR-2024)

Reviewer comment (Theory):
The loss L_{dissim} should be described more detailedly, especially the relationship between the motivation describe in introduction and minimizing the cosine similary of teacher and student feature.

Author response:
For Weakness 1, we specify the computation as follows:

$$L_{dissim}= \frac{<emb,ref\\_emb>}{||emb||_2\cdot||ref\\_emb||_2} $$

Here, $<a,b>$ represents the inner product of vectors a and b, and $||\cdot||2$ denotes the L2 norm. The reason for this design is that as the cosine similarity of the vectors decreases, $L_{dissim}$ becomes smaller. This term is to make the two embeddings as dissimilar as possible, which intends to introduce innovation. In reality, the exact method for achieving the optimal solution is unknown. We aim to utilize the aforementioned dissimilarity measure between $emb$ and $ref\\_emb$ to promote learning better features. However, we acknowledge that there may be alternative approaches to describe this relationship more effectively.

Reviewer comment (Experiments):
The student network and teacher network are in the same sturcture. Will it be better if using the trained student network to be the teacher network since the goal of the method is a student network with higher performance? More analysis should be given.

Author response:
For Weakness 2, thank you for sharing your novel idea and we conduct some experiments following your idea（denoted as "ArcFace+IIC+IIC"）. The table shows the performance of different methods on various datasets.

|  | LFW | CFP-FP | AgeDB-30| CALFW | CPLFW | VGGFace2 |Average
| :----:| :----: | :----: | :----: | :----: | :----: | :----: | :----: 
| ArcFace+IIC | 99.43 | 96.03 | 94.12 | 93.05 | 89.45 | 93.36 | 94.24
| ArcFace+IIC+IIC | 99.45 | 96.29 | 93.95 | 92.97 | 89.47 | 92.78 | 94.14

Based on the results, it seems that the proposed method does not provide significant improvements. One possible reason for this could be that the optimal direction may be the unknown global optimum but our optimization is for an existing known feature. The proposed IIC method can improve the existing feature directions so that they become closer to local optima only. The use of this method multiple times may cause the feature direction to zigzag around the local optimal direction. As a result, the improvement could be limited.

Reviewer comment (Experiments):
Sicne this method mainly focuses on feature distillation and representation learning, more comparision with the SOTA distillation methods should be given.

Author response:
For Weakness 3, we appreciate your perspective on analyzing face recognition from the view of feature distillation and representation learning. However, most of the settings in state-of-the-art (SOTA) methods for distillation are different from ours. While our teacher model and student model share the same structure, this setting differs from the typical approach employed in the SOTA methods. Most SOTA methods aim to approximate the performance of the original model by reducing the model parameters, which often leads to a loss in accuracy.

In contrast, our initial intention in designing the loss function was not solely focused on achieving distillation (i.e., reducing model parameters to closely match the performance of the original model). Instead, our goal is to learn new information and enhance model accuracy by leveraging the dissimilarity of features. Furthermore, we have compared our method with SOTA FR approaches and achieved better performance.

Reviewer comment (Theory):
One major weakness of the paper is that most of the concepts and associated discussion seem to be based on the 2-D spaces in Figs. 1 and 2 whereas the feature vectors are in a much higher dimensional space. One problem with over-relying on the figures is that authors speak as if there is one perpendicular component to the inferior model in Fig. 1. For a given model in n-dimensional space, there is a (n-1)-dimensional space orthogonal to it. How do we choose the innovation feature from this space?

Author response:
For Weakness 1, we use the 2D space to describe the feature space solely for the purpose of convenience in analysis and visualization. Analyzing and synthesizing features in a high-dimensional space is a non-trivial task. It is precisely because we conduct thorough experiments and analysis in 2-D space that we come up with the idea of using innovation to improve model accuracy. Finally, we propose an adaptive learning method (i.e., IIC) in a high-dimensional space.

Reviewer comment (Theory):
Another major weakness is that there are no theoretical proofs or justifications for the suggested improvements. If a superior model can be further improved by combining it with something orthogonal to an inferior model, doesn't that imply that the "superior" model may not have been trained sufficiently?

Author response:
For Weakness 2, we are trying to explore the theoretical proof for the suggested improvements. However, the fact that a superior model can be improved does not necessarily imply insufficient training but could also be due to the corresponding algorithm not effectively capturing more informative features. The idea of combining multiple inferior models is interesting, but how to properly integrate these features is a challenging task. This is because multiple models have different feature vectors, while the optimal vector is unknown. Nevertheless, we have made some attempts, and the experimental results are shown in the table below.

|  | LFW | CFP-FP | AgeDB-30| CALFW | CPLFW | VGGFace2 |Average
| :----:| :----: | :----: | :----: | :----: | :----: | :----: | :----: 
| 1 inferior model | 99.15 | 95.69 | 94.37 | 93.05 | 88.47 | 92.66 | 93.90
| 2 inferior models| 99.10 | 95.61 | 94.05 | 93.03 | 88.45 | 92.44 | 93.78

It seems that the method did not achieve significant improvements. One possible reason for this could be that the optimal direction is unknown and may represent a global optimum. However, our optimization is based on an existing known feature. Compared to the global optimal direction, the existing feature direction may be near the local optimal. The proposed IIC method can improve the existing feature directions to be closer to local optima. The use of multiple models may cause the feature direction to oscillate back and forth in the local optimal direction. As a result, the improvement is limited.

Reviewer comment (Experiments):
Related to the above, why should we stop with combining the innovation from just one inferior model? Why not consider multiple inferior models and extract and use features orthogonal to these multiple models?

Author response:
For Weakness 2, we are trying to explore the theoretical proof for the suggested improvements. However, the fact that a superior model can be improved does not necessarily imply insufficient training but could also be due to the corresponding algorithm not effectively capturing more informative features. The idea of combining multiple inferior models is interesting, but how to properly integrate these features is a challenging task. This is because multiple models have different feature vectors, while the optimal vector is unknown. Nevertheless, we have made some attempts, and the experimental results are shown in the table below.

|  | LFW | CFP-FP | AgeDB-30| CALFW | CPLFW | VGGFace2 |Average
| :----:| :----: | :----: | :----: | :----: | :----: | :----: | :----: 
| 1 inferior model | 99.15 | 95.69 | 94.37 | 93.05 | 88.47 | 92.66 | 93.90
| 2 inferior models| 99.10 | 95.61 | 94.05 | 93.03 | 88.45 | 92.44 | 93.78

It seems that the method did not achieve significant improvements. One possible reason for this could be that the optimal direction is unknown and may represent a global optimum. However, our optimization is based on an existing known feature. Compared to the global optimal direction, the existing feature direction may be near the local optimal. The proposed IIC method can improve the existing feature directions to be closer to local optima. The use of multiple models may cause the feature direction to oscillate back and forth in the local optimal direction. As a result, the improvement is limited.

Reviewer comment (Presentation):
Please clarify in the paper how the feature vector diagrams in Figs. 1 and 2 generalize to higher dimensions?

Author response:
For Q.1, the feature vectors in Fig.1 and Fig.2 are used to illustrate the differences between the innovation and other feature vectors, as well as to demonstrate the possibility of synthesizing better features. As you mentioned, the high-dimensional space is more complex and difficult for visualization. Figs.1 and 2 are included to assist in showing the potential relationships among the features, which subsequently motivates the development of a new adaptive learning method based on IIC in this work to learn better features in high-dimensional space.

Reviewer comment (Theory):
On Page 2, it is stated that "...innovation is always independent of features from CosFace". Independence and orthogonality are different concepts. Do you mean orthogonality or independence here? If later, please provide a justification for why orthogonality implies independence.

Author response:
For Q.2, our intention here is to state that innovation contains useful information for improving face recognition accuracy that was not yet captured by the inferior model. Innovation being orthogonal to the feature vector from the inferior model is consistent with this statement. We are going to use the term orthogonality, instead of independence, in the revised manuscript for clarity.

Reviewer comment (Theory):
In Section 3.1, it is stated that "In the first case, as shown in Fig. 2(a), a and b are on different sides of d." In 2-D, it is clear what we mean by different sides. How are different sides defined in higher dimensional spaces?

Author response:
For Q.4, we acknowledge that a and b being on different sides of d in fact implies the more general case where a and b are on different sides of the projection of d on the hyperplane specified by a and b. Under this scenario, the arguments presented in the manuscript remain valid. In other words, changing the modulus of the innovation may still be necessary to synthesize feature vectors close to the projection of the optimal feature vector d. We are going to point this out in the revised manuscript.

Reviewer comment (Theory):
In Eq. (3), should there be a summation over i on the RHS? If not, how come there is no dependence on i on the LHS?

Author response:
For Q.5,  Eq. 3 and Eq. 4 are both total loss functions but they represent two different formulations. On the right hand side of Eq. 3, $L_{{dissim}_i}$ represents the dissimilarity between the outputs of the $i$-th layer of the teacher and student networks, where $i$ takes a value within [1, N]. In Eq. 4, we utilize all the available N+1 dissimilarities (N hidden layers + final output layer). As such, Eq. 3 can be considered as a special case of Eq. 4. We compare the performance under different loss functions  in Table 8 in the appendix of our paper. In the experiments, we chose to include the dissimilarity between the features in the final layer only.