Paper: Few and Fewer: Learning Better from Few Examples Using Fewer Base Classes (ICLR-2024)

Reviewer comment (Theory):
The authors state that pre-training on the base classes “may even have a deleterious effect if the domain gap is too large” – and hence investigate if one can reduce the domain gap by fine-tuning on a subset; while this is ‘merely’ used as motivation, there is doubt that in cases where the domain gap is so large that it significantly harms transfer learning, fine-tuning on a subset of these classes would help (i.e. the presented approach). Comments or insights regarding this would be helpful.

Author response:
We would like to thank the reviewer for raising this point. Indeed, there exist obvious conditions where the domain gap can be reduced by selecting a subset of base classes, for example if the target task is precisely this subset of classes.
In conditions where the target distribution is far from the base distribution, such as Omniglot compared to ImageNet, the intuition behind the approach is not as evident.
Nevertheless, our experiments show that, even in the case of this large domain gap, the proposed methodology can significantly improve performance over the baseline, providing at least empirical evidence to support the idea that even in those cases fine-tuning on a subset of classes would help. We enriched the Methodology part of the main paper to reflect these points.

Reviewer comment (Experiments):
The presented results are somewhat constrained to the authors’ own setup – i.e. one baseline architecture that is used and then improved – and there is no indication of what state-of-the-art methods using the same backbone currently achieve on the chosen datasets. While it is not expected for the authors to beat any state-of-the-art method, it would be interesting for the reader to know how far the plain baseline and the improved one are from top-performing methods. Some insight into whether this sub-class finetuning setup might help in other methods as well (e.g. during meta-finetuning) would also be helpful to further support the paper’s findings.

Author response:
We have run new experiments using logistic regression instead of the NCM classifier on top of our existing backbones. This approach is supposedly close to SOTA according to a recent paper A Closer Look at Few-shot Classification Again by Luo et al. Our results (Table 2) show a very similar trend to those we obtained using the NCM classifier. As such, we see them as strong evidence that the proposed methodology could benefit other more sophisticated approaches, as it is orthogonal to most other works in the domain.

Reviewer comment (Experiments):
The authors choose the top 50 classes (AA) in the informed settings to create the subsets, mainly to keep a similar size of subsets to the uninformed settings. It is unclear how important this selection is to achieve good results (robustness), and some ablation would be helpful.

Author response:
We added an experiment varying the value of M (the size of the class subset) to see its influence on performance. Results are presented in the Appendix in Figure 2. It can be seen that any value of M between roughly 20 and 80 can lead consistently to improved results on most datasets, showing robustness in the selection of the optimum M.
We agree that this value is likely dependent on the base dataset. We observed that clusters of about 50 classes obtained in the UI setting were typically quite coherent. It is not necessarily expected that such clusters would be obtainable on any base dataset.
We also showcased the value of M obtained by using a cumulative total (activation) mass reaching a proportion of 90%, to illustrate that this method would indeed lead to a good selection in our setting.

Reviewer comment (Evaluation):
The importance of subset composition might be highly dependent on the dataset (base classes) composition. This raises questions about the generalizability of the results.

Author response:
We agree that this value is likely dependent on the base dataset. We observed that clusters of about 50 classes obtained in the UI setting were typically quite coherent. It is not necessarily expected that such clusters would be obtainable on any base dataset.

Reviewer comment (Experiments):
There might be more generalizable, dynamic, and better-justifiable ways to select classes than a fixed “top x” constant, such as selecting a certain percentage of the total mass/classes that cumulatively achieve a threshold in the softmax, or setting a threshold on the minimal contribution within the softmax.

Author response:
We also showcased the value of M obtained by using a cumulative total (activation) mass reaching a proportion of 90%, to illustrate that this method would indeed lead to a good selection in our setting.

Reviewer comment (Evaluation):
Judging by the results presented in Fig 1, the random subsets seem to be surprisingly better when using AA, FIM and RH. Insights or suspicions as to why this might be the case would be valuable.

Author response:
We currently do not have an unequivocal explanation. We are investigating this and we think it might be related to the fact that choosing between feature extractors specialized on meaningful subsets (X) can actually lead to worse outcomes than some feature extractors generalist subsets (R).

Reviewer comment (Evaluation):
Fig 8 (Appendix) indicates that the Semantic features generally provide better upper bounds as shown by the oracle. Clarification is needed as to whether this is due to the more powerful underlying CLIP training, or because semantics (text) generally provide richer features for such selection, assuming only one type could be chosen.

Author response:
This is a very interesting phenomenon. We believe the two hypotheses you propose to be relevant. We think one way to answer this question would be to compare the visual and semantic features of CLIP (which allegedly have features of equal “representational power” ).

Reviewer comment (Novelty):
The paper, while presenting a method for selecting base classes and achieving better performance with fine-tuning in few-shot scenarios, lacks a strong sense of novelty. It would be beneficial if the authors could establish the optimality of their method, specifically by showing that it can identify the best 50 classes from a pool of 1000 ImageNet classes. Alternatively, if reaching the optimum is unfeasible, the paper should strive to provide approximate solutions that approach the upper bound.

Author response:
We respectfully disagree with the reviewer about the novelty of our approach. To our knowledge, we are the first to clearly hypothesize, and then empirically demonstrate, that fine-tuning to a subset of the base classes can actually result in improved overall accuracy. This finding opens a new line of research, as most papers do not question the base dataset and use it as a whole, indivisible entity.

We sincerely would like to provide results with optimality, but we believe that this is a very challenging question, probably beyond our reach. From a combinatorial perspective, there are 712 chooses 50 possible subsets to create, from which we know no theoretical tool that can predict the performance. 

Heuristics, such as greedy selection of the best classes for a target domain, could be added to the paper, but this would inevitably look arbitrary and lacking depth. We think that this question should be considered as future work, keeping our main contribution a demonstration of the ability of improving performance by selecting a subset of base classes.

The approach we introduced and discussed in the paper significantly improves over the baseline, showing that we can improve on the performance in a very competitive field. We think that this should be weighted positively by the reviewer.

Reviewer comment (Evaluation):
While the paper does show improvements over a baseline without fine-tuning, it would be more insightful if the upper bound, representing the optimal 50 classes for each task, could be explored and discussed, if practical. This would provide a clearer perspective on the significance of the achieved results.

Author response:
We respectfully disagree with the reviewer about the novelty of our approach. To our knowledge, we are the first to clearly hypothesize, and then empirically demonstrate, that fine-tuning to a subset of the base classes can actually result in improved overall accuracy. This finding opens a new line of research, as most papers do not question the base dataset and use it as a whole, indivisible entity.

We sincerely would like to provide results with optimality, but we believe that this is a very challenging question, probably beyond our reach. From a combinatorial perspective, there are 712 chooses 50 possible subsets to create, from which we know no theoretical tool that can predict the performance. 

Heuristics, such as greedy selection of the best classes for a target domain, could be added to the paper, but this would inevitably look arbitrary and lacking depth. We think that this question should be considered as future work, keeping our main contribution a demonstration of the ability of improving performance by selecting a subset of base classes.

The approach we introduced and discussed in the paper significantly improves over the baseline, showing that we can improve on the performance in a very competitive field. We think that this should be weighted positively by the reviewer.

Reviewer comment (Theory):
What is the optimal solution? And how to approach it?

Author response:
We sincerely would like to provide results with optimality, but we believe that this is a very challenging question, probably beyond our reach. From a combinatorial perspective, there are 712 chooses 50 possible subsets to create, from which we know no theoretical tool that can predict the performance. 

Heuristics, such as greedy selection of the best classes for a target domain, could be added to the paper, but this would inevitably look arbitrary and lacking depth. We think that this question should be considered as future work, keeping our main contribution a demonstration of the ability of improving performance by selecting a subset of base classes.

Reviewer comment (Experiments):
The simplicity of the NCM classifier limits what conclusions can be drawn about the quality of representations. Testing on more complex classifiers would strengthen the results.

Author response:
We agree that NCM is limited as a representative of existing strategies. We have added experiments using logistic regression. Results (in Table 2), where we see very similar conclusions to the case of NCM. We would like to thank the reviewer for pointing out this concern as it helped broaden our claims.

Reviewer comment (Experiments):
Clustering classes for the static library has no guarantee of generating coherent subgroups. Better ways to determine class groupings could be developed rather than simple hierarchical clustering.

Author response:
We agree with the reviewer that clustering might not be the best approach here. However, since the scope of the paper is mainly to show the existence of strategies that can improve performance, we did not want to dive too much into the optimization of these strategies. (see Appendix for more detail).

Reviewer comment (Evaluation):
The class centroids used for NCM classification could be skewed by the fine-tuning. Could improvements just come from this distortion rather than better representations?

Author response:
It is a very interesting question. To investigate whether the improvement is due to mere distortion as opposed to a better representation, we conducted a set of experiments where the backbone was frozen and only an additional linear layer was trained on the class subset. This linear layer can thus distort the feature space without fundamentally changing the representation. Our results (Table 6) show that our method is more effective than a mere distortion of the original representation.

Reviewer comment (Experiments):
Can the authors provide a rigorous statistical or empirical rationale for selecting $M=50$ as the fixed number of base classes in the Average Activations (AA) and Unbalanced Optimal Transport (UOT) selection strategies? How does this choice influence the balance between the breadth of class representation and the manageability of the subset size, particularly in relation to the diversity of the domain examples $D$ and the overall size of the base class set $\mathcal{C}$? Additionally, what impact might this have on the representational capacity of the fine-tuned feature extractor, especially when considering domains with a significantly higher or lower intrinsic class cardinality?

Author response:
That is indeed an important question that we spent time investigating during the rebuttal. We ran experiments to observe the effect of varying M in the top-M class selection. (as shown in Figure 2 of the Appendix). We see that the accuracy change is not particularly sensitive to M given M is sufficiently large. As a rule of thumb, any M above 20 classes and below 80 classes is a good fit for every sampling and datasets, justifying the choice of 50. These numbers should be considered highly dependent on the fact we are using ImageNet, on which there exists coherent subsets of classes with dozens of elements each, as displayed in the Appendix. Considering a different base dataset, we expect that the choice of M can be challenging, or require input from an expert.

Reviewer comment (Experiments):
How can the method be robustified to account for potential discrepancies between the support and query distributions, which are common in real-world scenarios?

Author response:
We agree that this is an interesting and important question within few-shot learning, as addressed in recent works such as Bennequin et al., “Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift”. However, for the purposes of this investigation, we have restricted our scope to the simpler but common setting where the support set and query set can be assumed to be drawn from the same distribution. We have added a few lines in Section 4 acknowledging this limitation and citing a relevant work.

That said, we believe there is potential to investigate a transductive variant of our algorithm that is informed by the query set as well as the support set or domain. Techniques such as transported prototypes (Bennequin et al.) could be used to seek an alignment of the distributions. Alternatively, in the UI setting, one could incorporate the distance between the two distributions as part of a heuristic for selecting a feature transform from the static library.

Reviewer comment (Novelty):
In general, authors investigate a collection of heuristics to select subsets for training. While some of these heuristics are effective in certain domains, it is difficult to draw a concise conclusion (e.g., Figure 1). It is more like empirical analysis than a novel approach.

Author response:
We respectfully disagree with the reviewer. We would like to point out that the collection of heuristics that is mentioned in the comment is not our main contribution in this paper. As stated in the paper, the main contribution is to show that there exist subsets of base classes that, when fine-tuned on, can lead to improved performance in few-shot classification of images, and that such subsets can be retrieved even when very limited knowledge of the target task is available. As far as we know, this is indeed a novel approach, and it offers new opportunities for improving few-shot classification. It is true that the paper is mostly empirical, but we still believe it can constitute a significant contribution to the field due to the originality of the described approach.

Reviewer comment (Evaluation):
In Table 1, the performance improvements seem to dependent on the similarities between the classes of target datasets and those of base classes (e.g., Aircraft and Traffic Signs). This may limit the application of the proposed heuristics.

Author response:
In fact, while class similarity is one important factor, our experiments on a wide variety of domains indicate that it is not the *only* factor determining the success of the method. For example, while the Fungi and VGG Flower domains have several similar base classes in ImageNet, and the Omniglot domain has none, we observe larger improvements for Omniglot using our method in the DI and TI settings (see Table 1 and pie charts in appendix). This shows that the simple Average Activation strategy is capable of identifying a subset of useful classes even when the distributions of the base classes and target classes do not overlap at all.

Reviewer comment (Reproducibility):
Some necessary contents, including Alg. 1 and details of heuristics, are put in the appendix. This makes the audience unable to see a rigid, complete version of their method from a main paper, raising concern about page limit circumvention.

Author response:
We made some minor changes to free enough space so that Alg. 1 is now part of the main paper.

Reviewer comment (Reproducibility):
In Sec. 3.2 about informed settings, a method is directly put there without any analysis, reasons, and details. I would like the authors to provide the motivation and some details behind these methods: Why is AA designed in this form? What is the implementation of $g$, fine-tuning on a pre-trained, fixed $h$; using a non-learning method on $g$; or co-train $h$ and $g$ on the pre-trained dataset? Is the same $g$ used in the latter testing phase? Does the scale of $g$ (as logits) matter in selecting $\mathcal{X}$?

Author response:
We have revised the description of the method and moved the algorithm into the main paper. See revised paragraph below:

> To choose a class subset, we need a method by which to identify the base classes which are most useful for a given this set of examples $\mathcal{X}$. Fortunately, the base model already comprises a classifier that assigns a score to each base class. We therefore propose to simply compute the average class likelihoods predicted by the base model on the novel set~$\mathcal{X}$, and then select the $M$ highest-scoring classes. This straightforward selection strategy will henceforth be referred to as Average Activations (AA), and is outlined in Algorithm 1. While it is by no means guaranteed to select the class subset that will yield the optimal representation for the final task after fine-tuning, it is a reasonable proxy for that purpose.

We hope this makes clear that the $g$ used for AA subset selection is precisely that which was learned with the base feature transform $h$, without additional fine-tuning. Fine-tuning is only applied *after* class subset selection, to adapt the base feature transform to the selected subset.

Reviewer comment (Novelty):
In Sec. 3.3 about uninformed settings, the authors introduce the method just by listing and citing a number of related works. So, I would like to know what the authors contribute to this part, or whether they are just summarizing or aggregating existing methods.

Author response:
Section 3.3 describes our procedure for building a static library of feature extractors in the uninformed setting, for which we adopt Ward’s linkage algorithm and CLIP text embeddings. Indeed, the contribution here lies not in any novel algorithm for clustering or text embedding, but rather in the idea of applying these standard tools to build such a library of feature extractors by fine-tuning on class subsets. We designed the algorithm to be as simple and straightforward as possible since our purpose was not to over-optimize the feature extractors for the Meta-Dataset domains, but rather to demonstrate that the approach can lead to consistent improvements in accuracy even when not tailored to a specific target dataset. We clarified the text.

Reviewer comment (Experiments):
All the tasks are image classifications. I wonder if other tasks, other than image classification, can be applied with the proposed pipeline. I would like to see some simple yet representative results on any other tasks. If the proposed method can be applied not only in image classification but also in many other tasks, the contribution of this paper can be even higher.

Author response:
We would like to thank the reviewer for the interesting suggestion. We designed a new experiment where the DI feature extractors are used to perform semantic segmentation afterwards, using only one image for each considered class. We used Cityscape and adapted the backbones to these tasks by adding a small multilayer subnetworks trained on the few available shots, a methodology similar to Mining latent classes for few-shot segmentation by Lihe Yang et al.  As can be seen in Table 5 in the Appendix, we obtained results consistent with classification with significant increase in performance in the DI setting.

Reviewer comment (Writing):
(Minor) In Alg. 1, the set of examples is defined as $\mathcal{X}$, but the pseudo code uses $X$. And, line 2 is actually not required since operation 3 "selecting top $M$" already requires a top-$M$ partition.

Author response:
We also fixed the typo you found in the algorithm.