Paper: Evaluating the Evaluators: Are Current Few-Shot Learning Benchmarks Fit for Purpose? (ICLR-2024)

Reviewer comment (Experiments):
If class balance is an assumption that can be made (based on the composition of a given task), then maybe estimators could be adapted to have splits respect that constraint. Cross-validation could have leave-one-shot out (the support set would be balanced, k-1 shot and the valid set would have one example of each class), sampling for bootstrapping could also be aware of classes. If it works, this might be a practical contribution.

Author response:
**A3:** We agree it is likely that some of the performance degradation in the balanced setting comes from class imbalance introduced by the splitting procedures. Section 4.3 (Figure 6 right, in particular) provides some supporting evidence for this.

Reviewer comment (Evaluation):
Could you also report training accuracy, for the different estimators (incl. oracle)? Or is it almost always 100% and there's no signal there?

Author response:
**A1:** This is a good suggestion. We plan to add support set accuracy as an evaluator in the final copy of the paper.

Reviewer comment (Evaluation):
Is it the case that all the estimators actually underestimate the test accuracy (up to statistical noise)?

Author response:
**A2:** From Figure 1 we can see that methods are more likely to underestimate the accuracy than to overestimate it.

Reviewer comment (Experiments):
If we can assume that tasks of interest have balanced training sets, can class-aware splits provide a better estimation? (see point 2 in "Weaknesses" above)

Author response:
**A3:** We agree it is likely that some of the performance degradation in the balanced setting comes from class imbalance introduced by the splitting procedures. Section 4.3 (Figure 6 right, in particular) provides some supporting evidence for this.

Reviewer comment (Experiments):
If class balance cannot be assumed in general (as I'd think would be the case in real world applications), maybe it would be worth doing experiments on episodes with un-balanced support sets, either from other benchmarks, or altering the composition of episodes from usually-balanced ones.

Author response:
**A4:** We agree that class imbalance is a common feature of real-world problems, and experiments in this area would be interesting. We investigated the balanced setting because that is what FSL literature typically focuses on.

Reviewer comment (Evaluation):
The part related to the first question is unclear: all three evaluations (hold-out, cross-validation, and Bootstrapping) are widely used for few-shot evaluations. The only difference is using the support set only. But if the support set is treated as the whole set, there are no obvious differences.

Author response:
**A1:** They are not. The standard way that FSL methods are evaluated in the literature is via Aggregated Evaluation, where performance measurements are aggregated over a large number of meta-test episodes. In contrast, the evaluation protocol we investigate is task-level validation. Section 3 of our paper described this distinction in detail.

Reviewer comment (Evaluation):
Disagreement with the hypothesis in Figure 1 — since the oracle set is larger than the estimate set and there is a lot of variance for each estimate set, it is not surprising that the performance on the oracle set is more stable.

Author response:
**A2:** Large query sets are not available in real-world applications. The purpose of our paper, as discussed in the introduction, is to demonstrate that using only the support set for model training and validation results in performance estimates that are unstable to the point of being unusable. Figure 1 shows this.

Reviewer comment (Theory):
(Section 3, AE) The reviewer questions the statement “We observe that in a true FSL setting, this is an unrealistic assumption.” — Why is this not realistic? In practice, few-shot aims to tune a model with a few samples so it can handle a lot of samples, implying a much larger query set.

Author response:
**A3:** If a larger labelled query set was available then these points could be combined with the support set to form a dataset that is too big to be considered “few-shot”.

Reviewer comment (Writing):
(Section 4.1) Clarification is needed on what the evaluators are. They should be clearly defined. The paper uses many similar terminologies, such as “estimators” and “oracle accuracy,” which do not sound general enough for an audience outside of few-shot learning.

Author response:
**A4:** They are described in Section 3 under the Task-Level Evaluation heading.

Reviewer comment (Evaluation):
I am uncertain about how useful in real application would the conclusion from this paper be. After all the experimental investigation and analysis, the suggestion given by the authors is that every few-shot setting should design specialized evaluation procedure. This would be laborious and complicated for future works which makes this suggestion infeasible. Also, there is no example given by the authors about how to design the evaluation procedure based on certain specific tasks.

Author response:
**A1:** In many real applications it is important to know the performance of a model before deploying it. Our work demonstrates that existing evaluation procedures are not suitable for the standard FSL setting, so new methods must be developed. Whether this is laborious or not does not change the situation about the need for such evaluation methods. Moreover, we suggest developing evaluation methods based on Bayesian inference, as this paradigm is already widely used in the FSL community for developing new learning algorithms for various settings.

Reviewer comment (Writing):
I assume "CV" is short for "cross-validation" and "CV LOO" means "cross-validation leave-one-out", right? However, there is no explanation in the paper about what they mean, which causes confusion.

Author response:
**A2:** Yes, CV and CV-LOO are short for cross-validation and cross-validation leave-one-out, respectively. We have updated the paper to make this clear.

Reviewer comment (Evaluation):
From the results in Tab. 1, it seems that all estimators generally have under-estimated accuracy compared with the oracle method, so all of them should be pessimistic. However, in the conclusion section, it seems that only some of the estimators like LOO-CV and Bootstrapping are pessimistic estimators. I am wondering whether there exist an inconsistency or I have misunderstood the results in Tab. 1 and the conclusion on "Performance Estimation".

Author response:
**A3:** All evaluators are pessimistic in expectation (i.e., the estimators are biased), but LOO-CV and Bootstrapping have lower variance as well. This means the risk of overestimating performance on an individual episode is lower than it is for, e.g., hold-out. Figure 1 provides a good visualisation of this.