Paper: Cross-domain Few-shot Classification via Invariant-content Feature Reconstruction (ICLR-2024)

Reviewer comment (Novelty):
The single attention head and scaled dot production attention mechanism proposed in this paper are not novel concepts and are so simple that the technical contribution is insufficient to acknowledge the quality.

Author response:
Per your question, we guess that there might be some understandings here. 

The core idea of our paper is __reconstructing fine-grained invariant-content features for few-shot classification__ instead of proposing a single attention head. __In fact, these basic and general modules, such as attention head and FiLM, have been widely used to realize different ideas in previous works, such as Tri-M and CrossTransformer etc.__.

As mentioned in our paper, based on the assumptions: __(i) the two fundamental elements of an image are _content_ and _style_, (ii) content and style are independent, i.e. style changes are content-preserving__, a feasible way of reconstructing invariant-content features is __firstly locating the positions of invariant content features__ and __then retrieving these features to reconstruct invariant-content features__.

Based on such idea, we choose to use a single attention head since __(i) the process of calculating similarity matrix is equivalent to measuring the similarities between each query and all keys to locate the most similar features in pixel-level__ and __(ii) the process of multiplying the similarity matrix to values is equavilent to retrieving features that are invariant to style modifications in the way of weighted sum__.

Reviewer comment (Novelty):
The presented Theorem is just borrowed from the existing works.

Author response:
Per your question, the main goal of the theorem is to __provide a direct and intuitive explanation of our concern that whether applying complex attention module (compared with linear transformation head) results in extremely large distance (e.g., infinity) between two original samples that share similar content features__ based on the existing theoretical results.

According to the theorem proved in Vuckovic et al., 2021, we find that the distance between the features of two samples obtained from the attention module is upper boundedby the distance between the two original samples with a uniform Lipshitz constant. Thus, given a well pre-trained backbone (e.g., the pre-trained backbone proposed in URL or SUR), the distances extracted from the attention module will not be extremely large if the features themselves are close.

Reviewer comment (Writing):
It is not easy to know exactly what “high-level features” and “invariant-content features” denote. It is not clearly explained in the manuscript, and from the caption in Figure 2, one can only guess that the output features of the backbone are high-level features, and the output features of the attention head are invariant-content features.

Author response:
Per your concern, for the "high-level features", a common sense in representation learning is that the output of a backbone is usually treated as high-level features that can depict the general semantic information of the input.

For the "invariant-content features", we have described them in both introduction section and section 3.1. The description of "invariant-content features" is based on two assumptions proposed in ReLIC. To be brief, the "invariant-content features" are those discriminative features which depict the fundamental information of each image and are robust to style changes.

Reviewer comment (Experiments):
Comparison with SoTA methods is insufficient. The baseline (URL, published in 2020) of the toy example in Figure 1 is outdated, and the experiment also needs to be compared with TSA[1] and TriM[2], which are currently recording higher performance than URL on the leaderboard.

Author response:
Per your question, in our method, we mainly compare IFR with those methods which train the entire pipeline from scratch or fine-tune a head on top of the pre-trained backbone. Thus, we did not take TSA into consideration. Thank you for your comments, we will further apply our IFR method to TSA to evaluate the performance TSA+IFR. As for TriM, we have compared IFR and TriM in experiment section of our paper(Table 1).

Reviewer comment (Evaluation):
Motivation is poor. The motivation of this paper is that prior works cannot capture representative key features, and the only evidence to support this is a comparison of the figure's activation map. This comparison alone is not sufficient to point out the shortcomings of prior works, and since it is a qualitative comparison, it is not accurate. As a result, the significant performance gains in Tables 1 and 2 are not credible, and even if it is true, the performance improvement cannot be sufficiently explained in the paper, so the contribution is greatly limited.

Author response:
Per your concern regarding our motivation derived from illustration figure, a fact that we have to agree is that many works in deep learning areas arise from intuitions that are not well backed up by empirical evidence. In our paper, we have provided as much evidence as possible to illustrate our motivations and effectiveness of our proposed method.

Although Fig.1 seems insufficient to point out the shortcomings of prior works from your perspective, __it demonstrates that such phenomenon indeed exists__. Thus, __we utilize these figures to intuitively illustrate our main idea to readers__. Moreover, __as we mentioned in section 3.4, from the angle of model, since the previous work URL merely fine-tunes a linear transformation head on top of the fronzen pre-trained model, it is obvious that the function space in URL is less complexity__. We think both Fig.1 and the less complexity of function space of URL can demonsrate our motivation to some extent.

As for the empirical results, we would like to note that __(i) our proposed method achieves better performance on unseen domains with obvious gaps in both Table 1 and 2; (ii) IFR also achieves better generalization performance on meta-dataset with other pre-trained models, and the results are reported in Table 12, Fig.5 and Fig.8__. According to these results, we do not think that the performance gains are not credible.

Reviewer comment (Presentation):
There are redundant contents that are not very important. For example, Figure (4) is simple enough that it does not need to be shown as a figure, and there is no need to provide Theorem 2 since it is not proposed in this paper.

Author response:
__[About incosistent citation.]__ Thank you for your comments, we will modifiy this in our future versions.

__[About redundant content.]__ From our perspective, we do not completely agree that Theorem 2 is redundant content. The main goal of the theorem is to __provide a direct and intuitive explanation of our concern that whether applying complex attention module (compared with linear transformation head) results in extremely large distance (e.g., infinity) between two original samples that share similar content features__ based on the existing theoretical results.

Reviewer comment (Writing):
In the second paragraph on page 2, it is said that the proposed method considers both informativeness and discriminativeness at the same time. However, it is not easy to tell the difference between the two concepts through the current introduction. How exactly are they different, and how does the proposed method use these two concepts at the same time?

Author response:
Per your question, an intuition about fine-grained featrues that are extracted in pixel-level is they contain more information. Besides, an implicit assumptions, which is widely adopted in learning representation for classification, is the class-specific representations are general among samples within the same class while discriminative enough between different classes. Since the invariant-content features are extracted in pixel-level and aims at learning robust features that depict the fundamental information of images, they are informative and discriminative.

Reviewer comment (Theory):
Page 7 says that query heads have Lipschitz continuous property and the author’s comment follows as: “we can reliably leverage IFR to find good representations…”. But isn't this explanation too insincere and uninformative? Although it is known that Lipschitz continuity improves robustness against perturbation, more detailed analysis needs to be provided.

Author response:
Per your question, the main goal of the theorem is to __provide a direct and intuitive explanation of our concern that whether applying complex attention module (compared with linear transformation head) results in extremely large distance (e.g., infinity) between two original samples that share similar content features__ based on the existing theoretical results.

According to the theorem proved in Vuckovic et al., 2021, we find that the distance between the features of two samples obtained from the attention module is upper boundedby the distance between the two original samples with a uniform Lipshitz constant. Thus, given a well pre-trained backbone (e.g., the pre-trained backbone proposed in URL or SUR), the distances extracted from the attention module will not be extremely large if the features themselves are close. 

To be specific, a key step of IFR is measure the similarities among samples in pixel-level. Thus, by taking adavantage of the single attention head, we can guarantee that the original similar pixels, which depict the invariant-content features, are still close to each other so that we can locate the positions of the invariant-content features.

Reviewer comment (Novelty):
The novelty of the proposed methodology is limited. The proposed IFR method only performs cross-attention between the original images and their augmented images. The transformer-based cross-attention has been widely used.

Author response:
Per your concern regarding the comparison between single attention head and cross-attention module, we guess that there might some misunderstandings here.

The key motivation in our paper is __reconstructing fine-grained invariant-content features for few-shot classification tasks__ instead of just imply applying attention module.

As mentioned in our paper, based on the assumptions: __(i) the two fundamental elements of an image are _content_ and _style_, (ii) content and style are independent, i.e. style changes are content-preserving__, a feasible way of reconstructing invariant-content features is __firstly locating the positions of invariant content features__ and __then retrieving these features to reconstruct invariant-content features__.

Based on such idea, we choose to use a single attention head since __(i) the process of calculating similarity matrix is equivalent to measuring the similarities between each query and all keys to locate the most similar features in pixel-level__ and __(ii) the process of multiplying the similarity matrix to values is equavilent to retrieving features that are invariant to style modifications in the way of weighted sum__.

Reviewer comment (Writing):
Limited Explanation of Methodology: While the paper provides a high-level overview of the IFR approach, it could benefit from a more detailed explanation of the methodology, including the attention module and how it specifically contributes to invariant-content feature extraction.

Author response:
Thank you for your effort in reviewing our work. We appreciate your valuable suggestions and comments. Now, we post our responses for your concerns in the following.
> __Weakness 1 & Q1.__ Limited Explanation of Methodology: While the paper provides a high-level overview of the IFR approach, it could benefit from a more detailed explanation of the methodology, including the attention module and how it specifically contributes to invariant-content feature extraction. / Can you provide a more detailed explanation of the attention module used in IFR and how it specifically contributes to the extraction of invariant-content features?

__[About attention module.]__ Per your question about attention module, we have explained it in our experimental setting part. To be specific, the attention module adopted in our paper is composed of 3 linear layers with the same dimension $512 \times 512$. Following URL, in order to take advantage of the features extracted from the frozen pre-trained backbone, all linear layers in the attention module are initialized as identity matrix at the beginning of each learning episode.

<font color=blue>__[About function of attention module.]__</font> Regarding to problem of how the attention module specifically contributes to the extraction of invariant-content features, __we have made a detailed explanation in Section 3__. __The main contribution is that the attention module fits the procedure of invariant-content features quite well__.

First of all, an important assumption of our method is: __(i) the two fundamental elements of an image are _content_ and _style_, (ii) content and style are independent, i.e. style changes are content-preserving__. According to these assumptions, content features that depict the fundamental information of an image are invariant to style modifications.

Generally, the reconstruction of invariant-content features mainly include two steps: __locating the positions of invariant content features__ and __retrieving these features__. In this paper, the two steps are realized with a single attention module.

Specifically, we first locate the positions of invariant-content features by measuring the similarities between original and augmented image representations in pixel level. As illustrated in Fig. 3(a), given a set of queries embedding and keys embeddings, $\boldsymbol{Q}\in\mathbb{R}^{w\times h\times d}$ (transformed from original images) and $\boldsymbol{K}\in\mathbb{R}^{w\times h\times d}$ (transformed from augmented images), a similarity matrix $\boldsymbol{M}_{\rm sim}\in\mathbb{R}^{wh\times wh}$ is measured to describe the similarity between arbitrary two pixels respectively from queries and keys embedding. As the aforementioned assumptions, content features are invulnerable to the changes of style. Thus, two pixels that depict the same content information ought to be highly similar to each other and the similarity score will be relatively larger than those depict trivial style information. Thus, the positions of pixels that depict the same invariant-content features in augmented representations are determined for each pixel of original representations.
Then, with the similarity matrix, the invariant-content features are reconstructed by retrieving pixels with corresponding similarity scores (as shown in Fig. 3(a)). The reconstruction process is equivalent to a recombination of pixel vectors where the invariant-content features assigned with larger similarity scores are highlighted (as shown in Fig. 3(b).

Reviewer comment (Experiments):
Lack of Comparative Analysis: The paper presents experimental results demonstrating the effectiveness of IFR, but it lacks a thorough comparative analysis with existing methods, discussing in detail why IFR outperforms them. In addition, since this proposed work resembles [1][2], why not compare with these two methods in the Experiments Section?

Author response:
> __Weakness 2.__ Lack of Comparative Analysis: The paper presents experimental results demonstrating the effectiveness of IFR, but it lacks a thorough comparative analysis with existing methods, discussing in detail why IFR outperforms them. In addition, since this proposed work resembles [1][2], why not compare with these two methods in the Experiments Section?


__[About comparative analysis.]__ Thank you for your constructive suggestion. As we mentioned in our paper, __the main discrepancy between IFR and previous URL is that IFR further considers capturing more discriminative and informative fine-grained invariant-content features that are robust to style modifications and depict the fundamental information of the image__. Meanwhile, __compared with IFR which merely fine-tunes a simple linear transformation head on top of a pre-trained backbone, IFR, which trains a more complicated module, owns much more complex function space to fit the representations__. Thus, IFR obtains better results.

In our paper, both quantative results in tables and visualization results of learned representations demonstrate our statement. Besides, we also have conducted several ablation studies to explore the effectiveness of each part of our proposed modules.

__[About comparison to the mentioned references.]__ Per your concern regarding the comparison between IFR and the given two references, in fact, we have provided discussions in Appendix A (Paragraph 2, Page 15). 

We agree that the insights behind our proposed IFR are similar to those in [1, 2]. All of these methods aims to __drop the unimportant style informtion like colors and preserve the robust and invariant features with good discrimination ability__. __However, there are some differences in the ways to reach the goal__. For example, MemREIN proposed in [1] reduces the cross-domain descrepancies(colors etc.) by performing instance normalization and then uses a memorized restitution to preserve the discrimination ability of the refined features, and the adversarial task augmentation proposed in [2] takes the idea to learn robust features by generating the inductive bias-adaptive 'challenging' tasks. But in our IFR, we use a simpler way. We weaken the style information and highlight the invariant-content features by extracting the invariant information between the original data and their augmented counterparts with a single-head attention module. For the method proposed in [3], we think it is quite different from [1, 2] and our proposed IFR. Different from [1, 2] and our IFR that tries to extract more robust and useful features from the original features, the method proposed in [3] introduces additional hyperparameters to learn feature transformation. The proposed feature wise transformation resembles the FiLM proposed in [4].

Per your question regarding comparing two methods in experiment section, since we __mainly focus on cross-domain few-shot classification tasks with vary-way vary-shot settings__ proposed in meta-dataset[5], we did not take these two methods taken into consideration.

Reviewer comment (Experiments):
Potential for Overfitting: Given that the approach focuses on fine-grained features, there might be a risk of overfitting, especially when dealing with extremely limited data in few-shot scenarios. The paper does not address this potential issue or provide strategies to mitigate it.

Author response:
> __Weakness 3 & Q3.__ Potential for Overfitting: Given that the approach focuses on fine-grained features, there might be a risk of overfitting, especially when dealing with extremely limited data in few-shot scenarios. The paper does not address this potential issue or provide strategies to mitigate it.

Thank you for pointing out this, we also notice that there exists overfitting phenomenon and such phenomenon further negatively affect the performance of our method. __The main reason that we did not apply any technique to mitigate such phenomenon is that few work, as far as we know, has applied strategies for mitigating overfitting during meta-test phase__. A feasible way for such case is to consider modifying the adaptation loss to supress data overfitting. We will think about this problem in future version of our paper.