Paper: Less is More: On the Feature Redundancy of Pretrained Models When Transferring to Few-shot Tasks (ICLR-2024)

Reviewer comment (Novelty):
The proposed method is over-simple and more like a trick. And thus there are no ablation study. I suggest the method can be more systematic.

Author response:
Thanks for your helpful advice! We follow your suggestions and conduct several ablation studies, including the use of augmentations for augmenting training set and estimating FIs, the type of data augmentations, the number of augmentations per image, the number of ways, and the number of shots. All these experiments have been added as Appendix F in the revised paper. As seen from the results, our method can give robust performance improvement in different settings. We hope these added contents will make our method more systematic, in order to let readers know how to follow the specific design details.

Reviewer comment (Experiments):
Fine-tuning is typically a more preferred method and achieve better performance than linear probing, so it has a weak value to solve linear probling alone. How the proposed method perform for fine-tuning?

Author response:
We follow your suggestions and conduct experiment with full finetuning on ImageNet-pretrained ResNet-50. Please see the details in Appendix G of the revised paper. We report the results below.

1-shot: CUB: 76.48 + 1.6 Traffic Signs: 55.33 + 2.9 Aircraft: 43.62 + 0.5 Crop Disease: 79.21 + 0.8 EuroSAT: 72.43 - 0.9 Fungi: 50.93 + 1.6 ISIC: 34.57 + 0.8 Omniglot: 81.11 + 1.3 Quick Draw: 54.59 + 0.2 VGG Flower: 83.13 + 1.0 ChestX: 22.57 + 0.0 Avg: 59.45 + 0.9

5-shot: CUB: 91.89 + 1.1 Traffic Signs: 81.11 + 3.4 Aircraft: 59.83 + 4.7 Crop Disease: 92.20 + 1.1 EuroSAT: 87.69 + 0.3 Fungi: 74.63 + 2.9 ISIC: 53.23 + 3.7 Omniglot: 95.15 + 2.1 Quick Draw: 73.57 + 2.2 VGG Flower: 95.83 + 0.8 ChestX: 24.76 + 0.2 Avg: 75.44 + 2.0

As can be seen, our method can improve fine-tuning performance under few-shot transfer scenarios, and sometimes by a large margin. It is very interesting to analyze how features and their importance change during finetuning, and we leave this to future work.

Reviewer comment (Experiments):
As shown in Figure 2, the proposed phenomenon seems to disappear when way is large. In fact, I think the large way is more realistic and ractically useful.

Author response:
Please take another closer look at Figure 2, and we can see that the feature redundancy phenomenon weakens but not completely disappear when way is large, different from the trend of the number of shots; for example, from the figure we observe that, 255 dimensions, only 10% of the full dimensions, are enough for obtaining the original 100-way performance, thus 90% of the feature dimensions can be seen as redundant. In addition, we see that there is always a large gap between the performance obtained by Oracle feature importance adjustment (red line) and the original performance (blue line) regardless of the number of ways, leaving room for feature adjustment.

Reviewer comment (Experiments):
How the proposed method perform under many-way classification? The authors only provide the results on 5-way setting, how about the results on 20-way, 50-way setting?

Author response:
Following the previous response, we conduct experiments on higher ways as part of ablation studies in Table 7 in the revised version of the paper. Please take a look. We can see that the performance improvement still exists for high ways like 100, though not that significant for some datasets, which verifies our statement in the previous response.

Reviewer comment (Experiments):
While this paper studies the few-shot problem, it mainly focuses on the nearest-centroid classifier (NCC) and employs Euclidean distance as the distance metric. However, it is worth noting that most practical few-shot methods use **cosine distance** rather than Euclidean distance. Therefore, to enhance precision and applicability, it is advisable for the paper to explicitly limit its claims to Euclidean distance-based scenarios. It would be beneficial for the authors to acknowledge this limitation and potentially explore how the theoretical and empirical analysis may adapt to cosine distance-based methods. This could provide a more comprehensive understanding of the practical implications of the work.

Author response:
**1. While this paper studies the few-shot problem, it mainly focuses on the nearest-centroid classifier (NCC) and employs Euclidean distance as the distance metric. However, it is worth noting that most practical few-shot methods use cosine distance rather than Euclidean distance.**

We apologize for the misunderstanding caused by negligence in writing. We normalized all pretrained features to have unit $l_2$ norm throughout the paper but forgot to mention this point. Thus in fact, **we are using cosine distance, not Euclidean distance** in all our experiments except for the theoretical part. Also, the toy example in the theoretical part shows that using Euclidean distance can lead to feature redundancy as well. We will further show in our next point of response that using other metrics like Kendall distance also leads to the feature redundancy phenomenon. Thus the feature redundancy problem, as an intrinsic problem of pretrained features, generally exists regardless of the transfer-time classifier used. In addition, if interested, the reviewer can view Appendix G which shows that even using full finetune as the adaptation method, the feature redundancy problem still exists (as part of the response to Reviewer Xfoy). We have revised the paper to explicitly mention the use of cosine distance in Appendix C.

Reviewer comment (Evaluation):
I understand that different features have different importance [1]. The central argument of this paper is that utilizing a reduced set of crucial features outperforms the use of all available features. However, I noticed another paper [2] providing empirical evidence to the contrary, suggesting that "tail" features are more important than the core features in the few-shot problem. I wonder if the contradiction between these perspectives comes from the use of distinct distance metrics, as [2] employs Kendall distance. If so, this reminds us again that this work is primarily applicable within the context of Euclidean distance.

Author response:
**2. I understand that different features have different importance [1]. I noticed another paper [2] providing empirical evidence to the contrary, suggesting that "tail" features are more important than the core features in the few-shot problem.**

 We would like to clarify that, the two papers are essentially **suggesting the same idea**. The confusion comes from the different definitions of feature importance: in [2], the authors define core features as the ones having the largest average magnitude, while in [1], the authors define core features using Eq. (1) shown in our paper. If we dive into paper [1] and look at Figure 6 in their paper, we will immediately realize that **core features defined by [1]** are mostly those features with a small average magnitude, which are **non-core features defined by [2]**. Thus two papers both suggest that **pretrained neural networks put wrong emphasis on feature dimensions with large average magnitude**, and our paper goes further, showing that most dimensions (non-core by definition in [1] but core by definition in [2]) carry disturbing information for few-shot transfer and should be removed entirely.

**3. I wonder if the contradiction between these perspectives comes from the use of distinct distance metrics, as [2] employs Kendall distance. If so, this reminds us again that this work is primarily applicable within the context of Euclidean distance.**

To further alleviate your concerns, we experiment with Kendall distance to see if we can observe the same phenomenon. Please visit https://app.gemoo.com/share/image-annotation/583219007069962240?codeId=DGVkWVjJJrR6O&origin=imageurlgenerator&card=583219004561768448 to see **exactly the same feature redundancy phenomenon with Kendall distance**. We also show below how our proposed method can improve few-shot transfer performance when using Kendall distance in the downstream task:

1-shot: CUB: 86.39 + 1.3 Traffic Signs: 62.26 + 1.6 Aircraft: 65.97 + 0.6 Crop Disease: 78.90 + 1.3 EuroSAT: 64.71 + 1.2 Fungi: 54.21 + 1.7 ISIC: 30.76 + 0.4 Omniglot: 86.73 + 1.1 Quick Draw: 64.93 + 1.4 VGG Flower: 93.75 + 0.8 ChestX: 21.19 + 0.1

5-shot: CUB: 96.74 + 0.1  Traffic Signs: 81.30 + 1.5  Aircraft: 81.28 + 0.3 Crop Disease: 92.91 + 0.4 EuroSAT: 82.58 + 0.6 Fungi: 74.99 + 1.6 ISIC: 42.30 + 0.7 Omniglot: 95.88 + 0.5 Quick Draw: 84.32 + 0.5 VGG Flower: 99.16 + 0.1 ChestX: 22.94 + 0.1

In the experiment above we use the visual encoder of CLIP as the pretrained model.

Reviewer comment (Experiments):
The proposed solution relies on estimating feature importance, which itself can be challenging in a few-shot setting as mentioned in the paper. A compromise method is applying data augmentation to the training data. However, this is a little bit unfair for few-shot learning since it could be regarded as an implicit means of augmenting the number of shots through the incorporation of human knowledge.

Author response:
**4. Applying data augmentation to the training data is a little bit unfair for few-shot learning since it could be regarded as an implicit means of augmenting the number of shots through the incorporation of human knowledge.**

We understand your concerns that the advantage of the method may come from the data augmentation itself, not from the proposed method. To alleviate your concerns, we conduct two additional experiments: 1. do not use data augmentation (noAug) and 2. use data augmentations to augment the training set and/or estimate feature importance. We show the average 5-way 5-shot performance and the improvement brought by the method over all 11 datasets in the Table below. For details and 1-shot experiments please see Table 3 and Table 4 in the revised version of our paper. Aug-est refers to only using augmentations for estimating feature importance, while Aug-est+sample refers to using augmentations for augmenting training set as well.

|Model|noAug|Aug-est|Aug-est+sample|
|-|-|-|-|
|DINOv2|77.48 + 0.9|77.84 + 1.0|76.07 + 1.2|
|EsViT|71.00 + 1.8|71.15 + 1.9|69.57 + 1.9|
|ResNet-50|72.00 + 2.3|71.51 + 2.2|70.02 + 2.5|
|CLIP|76.22 + 2.3|75.83 + 3.0|74.05 + 3.1|
|IBOT|74.14 + 1.4|74.37 + 1.5|72.66 + 1.6|
|Swin|68.05 + 1.3|71.82 + 1.4|70.72 + 1.4|

We can see from the result that (1) our method without data augmentations can still obtain significant performance improvement; (2) simply putting data augmentations into the downstream training set cannot bring improvement in most circumstances, while our method can take advantage of data augmentations. (3) our method can work when augmenting the downstream training set with data augmentation.

Thus we can see that the benefit of our method is universal regardless of the use of data augmentations, and more importantly, simply adding more data augmentations does not make an improvement for few-shot transfer (as opposed to many-shot learning), and our method takes a step forward to successfully make good use of data augmentations. We have noticed that in few-shot learning literature, data augmentations are rarely used for support data at the adaptation time (except for some generation-based methods that design specifically-desined augmentations that only work for in-domain setting), so this is in line with our findings, and it is quite interesting to see why data augmentations do not work for few-shot transfer settings in the future.

Reviewer comment (Writing):
Theorem 5.1: The formula of error rate in Eq. (3) is not intuitive enough. I suggest to use the equivalent form $\Pr[|(z_1,z_2)-(p_{(a,1)},p_{(a,2)})| > |(z_1,z_2)-(p_{(b,1)},p_{(b,2)})|]$ instead.

Author response:
**5. Page 6: z depends on label y, and thus you should use different subscripts to distinguish them. Moreover, you should remind readers that you use the same $\sigma_1$ and $\sigma_2$ for different y, since they are different in Eq. (1). Theorem 5.1: The formula of error rate in Eq. (3) is not intuitive enough. I suggest to use the equivalent form instead.**

Thanks for pointing out these typos and giving constructive suggestions. We have revised the paper according to your advice.

Reviewer comment (Theory):
Theorem 5.2: The simultaneous presence of both "with probability at least 0.9 (a certain number)" and "big O" renders this theorem meaningless. For example, choosing $999999999\sqrt{\log n/n}$ as $O(\sqrt{\log n/n})$ and choosing $0.0000001\sqrt{\log n/n}$ as $O(\sqrt{\log n/n})$ should have different probabilities. I believe it is possible to derive a closed-form expression for $O(\sqrt{\log n/n})$.

Author response:
**6. Theorem 5.2: The simultaneous presence of both "with probability at least 0.9 (a certain number)" and "big O" renders this theorem meaningless.**

We would like to clarify that the utilization of big O and "with probability at least 0.9" is to simplify the presentation of our theoretical results. This is because the goal of Theorem 5.2 is to show adding new dimensions is statistically useful for improving the performance, the additional term $O(\sqrt{\log(n)/n})$ comes from the uncertainty characterization of the random draw of training set, which can be vanishing when increasing the training sample size. 

Additionally, the constant in the big-O notation can be certainly obtained by conducting the proof in a more precise way. For instance, **in our proof, the constant in the Hoeffding's inequality can be exactly obtained to be $1/\sqrt{2}$, and the covering number of the Hypothesis can be also upper bounded by some detailed configurations of the hypothesis set. We have modified our proof for Theorem 5.2 in the appendix of the revised version,  which shows that the constant in the Big-O notation can be roughly bounded by 9.** Thus our bound is definitely meaningful. We further emphasize that such a constant is proved by combining many loose inequalities and can be certainly improved with more detailed calculations, but this is never the scope of this paper.