Paper: Is Pre-training Truly Better Than Meta-Learning? (ICLR-2024)

Reviewer comment (Novelty):
The paper is solid from the empirical point of view, presenting a large variety of results. However, it is somehow limited in terms of novelty as it does not introduce any new techniques or analyses.

Author response:
We respectfully differ. For instance, we have introduced a novel method of comparing two algorithms via the effect size, which we believe is more robust to sample size than p-values and confidence intervals (as described in Section 3.2). Furthermore, our paper also introduces a Task2Vec-based diversity coefficient that measures the task diversity of a given few-shot benchmark. We further present a detailed effect size-based statistical analysis of performance that gives substantial support to our claim that meta-learning outperforms pretraining if the data diversity is large enough.

Reviewer comment (Experiments):
A significant limitation of the paper lies in its reliance on two methodologies, MAML and PT with fine-tuning restricted to the head, which may not represent the current best practices in the field. Firstly, while MAML is undoubtedly foundational in meta-learning, its relevance has waned over time. Contemporary advancements have introduced more efficient derivatives, such as MAML++ (Antoniou et al., 2018). Incorporating comparisons with these modern variants could have enriched the paper's insights.

Author response:
We see this paper as a direct response to existing work that claims that meta-learning methods are worse than pre-training approaches, using equivalent methods to ours ( https://arxiv.org/abs/2003.11539 ). We propose the use of better experimental methods to allow a fair comparison between the techniques and show a reversal of what are now folklore conclusions. We note that the use of MAML as opposed to other meta-learning methods furthers our claim, since MAML is the simplest meta-learning approach. If MAML can outperform pre-training, better meta-learners should be able to do so too. Please find more details about this in the appendix, section A.6, paragraph 2.

Reviewer comment (Experiments):
The paper's approach to fine-tuning is notably narrow, focusing only on the head's parameter adjustments. Contrarily, cutting-edge methods today, like BiT (Kolesnikov et al., 20202), fine-tune the entirety of both body and head parameters, while others, like FiT (Shysheya et al., 2022), selectively adjust a subset of body parameters. A juxtaposition against these state-of-the-art techniques would have been insightful. These oversights are critical as the paper's primary conclusions might shift when evaluated against more contemporary, optimized methods.

Author response:
Regarding the use of alternative methods for comparison: We emphasize the impossibility of achieving a fair comparison once we decide to go down that route. Can such a comparison actually ever be done, given that it's impossible to truly compare the best algorithms in practice? Since pre-training is more widely studied, it is not fair to compare the best pre-training and the best meta-learning algorithm. Owing to the bias in favour of pre-training in the literature, can we rule our confirmation bias if we use advanced algorithms for pre-training? Similarly, how do we argue our choice of MAML & pre-training wasn't cherry picked? 

We argue that MAML and head fine-tuning are the most similar to each other, without this type of bias, as well as the most fundamental approaches in both fields. This allows us to conduct the most "fair" comparison that we could. Our study aims to unravel the true difference between meta-leanring vs pre-training through a fair comparison with the validated hypothesis that data type plays a crucial factor as we demonstrated.

Reviewer comment (Presentation):
The presentation of data exclusively in tabular format, though beneficial for transparency, hinders a quick understanding of the trends. I recommend the authors to enhance data representation by incorporating visual aids, such as scatter plots. This would facilitate a more intuitive grasp of the data patterns. The tables could be conveniently relocated to the appendix to maintain thoroughness without overwhelming the main content.

Author response:
Thanks for the suggestions. We have moved the detailed results to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We have additionally added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Reviewer comment (Presentation):
There are some formatting issues, e.g. (i) Table 6 should be within the body of the paper, (ii) in the text "Meta-Data set" should be replaced with "Meta-Dataset".

Author response:
Thanks for pointing this out. We have fixed these issues in the revision.

Reviewer comment (Novelty):
No novel methods are introduced in this paper, which itself is ok if the results are intriguing.

Author response:
We respectfully differ. For instance, we have introduced a novel method of comparing two algorithms via the effect size, which we believe is more robust to sample size than p-values and confidence intervals (as described in Section 3.2). Furthermore, our paper also introduces a Task2Vec-based diversity coefficient that measures the task diversity of a given few-shot benchmark. We further present a detailed effect size-based statistical analysis of performance that gives substantial support to our claim that meta-learning outperforms pretraining if the data diversity is large enough.

Reviewer comment (Presentation):
Presentation of the results is very poor. All results are just listed in tables, and there are no attempts to present the results in a comprehensible/impressive manner. Since I could not find any meaningful insight from the tables, I recommend adding more comprehensive figures which should be contributions of the paper.

Author response:
Thanks for the suggestions. We have moved the detailed results to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper.  We’ve added box plots and violin plots (Figure 1) to address this issue. We have additionally added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Reviewer comment (Experiments):
The results of Cohen's d can be caused from (1) meta-training dataset, and/or (2) meta-test dataset, in addition to learning algorithms, but I could not find which factor causes the results of Cohen's d. In other words, which diversity of meta-training datasets or meta-testing datasets makes the difference between the two methods?

Author response:
We use the meta-training dataset to compute the Task2Vec diversity for each few-shot benchmark, as we believe that the data diversity of the meta-training should be the influencing factor that allows the meta-learner to meta-learn better. Since each few-shot task for meta-training and meta-testing is sampled from the same few-shot benchmark, the diversity of the test and training splits should be more or less the same.

Reviewer comment (Evaluation):
A discussion of the drawbacks of effect size -- it is useful to understand where this may be inadequate (except the fact that one must choose a threshold level). Relatedly, the paper states that standard effect sizes are 0.2, 0.5, and 0.8, but it is unclear how the choice of 1% relates to these.

Author response:
We acknowledge the reviewer’s point that it is important to understand the limitations of using any statistical test. One of the major limitations of using effect sizes is that they can be misleading if considered in isolation of other metrics such as meta-test accuracies. We present these values in the appendix, section A.2. We further note that, as with any other statistical test, we need to define arbitrary thresholds for the interpretation of effect sizes. While the common values are 0.2,0.5,0.8, we have used the 1% threshold as a decision rule based on it being common to accept papers due to 1% differences in machine learning. We have provided a detailed description of our choice of the 1% threshold in the appendix, section A.6, paragraph 3. Note that the choice of 1% doesn’t affect our primary results, in which we report average of raw effect size values. 

	We highlight and answer the aforementioned issues in detail in Section 3.2 of our paper.

Reviewer comment (Presentation):
Presentation of experimental setup: given this paper is primarily related to empirical benchmarking, a summary of the experimental setup in the main paper would help contextualise the investigation. This includes summary information about the datasets, abbreviations used (which reflect the results tables), training setup etc.

Author response:
We have noted the hyperparameters and experimental setup in Section 4 in the revision (re-arranged from the appendix). While the information about datasets and abbreviations are still mentioned in the appendix, we have moved the detailed results section to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We have also added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Reviewer comment (Evaluation):
Clarity in results exposition: when reporting the overall effect size, it is unclear how these are obtained -- my guess is an average over the cases where that hypothesis was determined correct. This is not mentioned (as far as I saw) -- it would help to do so.

Author response:
Thanks for catching that. It is indeed the average over effect sizes that resulted in the same decision. We have added a brief explanation to the caption of table 1.

Reviewer comment (Presentation):
Clarity in results table: The tables could be better presented, for example by having the captions clearly specify that MAML5 refers to 5 adaptation steps, and having the same formatting for the heading and the main body (like all uppercase). Also, the detail on seeds in Table 3 is hard to understand -- what are seed1 and seed1 vs seed2 comparisons? The caption mentions 5 layer CNNs but the table refers to Resnet12 and Resnet50? Overall, the results are interesting but the presentation is very hard to follow.

Author response:
Thank you for pointing these errors out. We have fixed all aforementioned errors in the revision. We have additionally moved the detailed results to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We have also added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Reviewer comment (Novelty):
Minor: possibly missing citation in Section 5 when referring to feature reuse vs rapid learning

Author response:
We have added the citation in, thanks for the catch!

Reviewer comment (Theory):
What is the main principle behind the new test? Specifically, if it rejects a hypothesis, what can we tell about its confidence in doing so? For example, using t-test, we are implicitly assuming the performance differences follows a student-t distribution and the t-statistic basically tells us if we can reject the null hypothesis at a certain confidence level. Do we have a similar principle under the new test?

Author response:
Section 3.2 addresses why effect size is a good metric (compared to confidence intervals) to compare MAML and PT. In summary, effect size provides a way of quantifying the difference between the two algorithms - if the absolute value of the effect size is greater than the calculated standardized 1% threshold, we reject the null hypothesis. The effect size also quantifies the magnitude of the difference - the standard values are 0.2, 0.5, and 0.8 for a small, medium, and large effect size difference.

Reviewer comment (Evaluation):
For the task space with high diversity, I tend to think that the pre-training would not be effective with small architecture for obvious reason. Thus, the statistic test should only be applied to the reported performance on the best model architecture for pre-training (but across different test domains) — right now, it seems the performance report on ResNet50 would favor pre-training; and the conclusion that meta-learning is better is most likely caused by the included performance on lower-complexity ResNet structure.

Author response:
Our goal in this paper is to provide a fair comparison between pre-training and meta-learning, and to provide empirical evidence to the fact that meta-learning outperforms pre-training if the data diversity is large enough. We define a fair comparison as using the same model, same optimizer and training to convergence. We hence provide comparisons with different models and with different data diversities to conclude that high diversity and small architectures should lead to better meta-learning performance. With bigger architectures, the difference between meta-learning and pre-training is less pronounced due to the possibility of pre-trained models to meta-overfit.

Reviewer comment (Evaluation):
Can the authors elaborate more on why p-value & confidence interval would become zero on larger batch size?

Author response:
As the batch size increases, our sample size gets larger, and we tend to see that the confidence interval becomes zero (which can sometimes misleadingly suggest that the differences between two groups is significant - see https://www.nature.com/articles/s41598-021-00199-5). As addressed in Section 3.2, effect size is less affected by sample size which alleviates this drawback.

Reviewer comment (Theory):
Please elaborate on the main principle behind a new test (i.e., its underlying assumption regarding the distribution of performance difference between two algorithms and what can we say about the confidence in rejecting the null hypothesis).

Author response:
Section 3.2 addresses why effect size is a good metric (compared to confidence intervals) to compare MAML and PT. In summary, effect size provides a way of quantifying the difference between the two algorithms - if the absolute value of the effect size is greater than the calculated standardized 1% threshold, we reject the null hypothesis. The effect size also quantifies the magnitude of the difference - the standard values are 0.2, 0.5, and 0.8 for a small, medium, and large effect size difference.

Reviewer comment (Evaluation):
For high diversity setting, I think the statistic test should be based only on the best model architecture. Please consider revising this part of the evaluation protocol.

Author response:
Our goal in this paper is to provide a fair comparison between pre-training and meta-learning, and to provide empirical evidence to the fact that meta-learning outperforms pre-training if the data diversity is large enough. We define a fair comparison as using the same model, same optimizer and training to convergence. We hence provide comparisons with different models and with different data diversities to conclude that high diversity and small architectures should lead to better meta-learning performance. With bigger architectures, the difference between meta-learning and pre-training is less pronounced due to the possibility of pre-trained models to meta-overfit.