From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: This work presents an algorithm that chooses a tradeoff between training examples that promote generalization or memorization as a function of the active learning label budget (with implicit dependence on the loss, model family, and data distribution). The paper provides theoretical guarantees for a setting where the input domain is a mixture of two disjoint regions (easy vs hard examples) and the hypothesis being learned is a mixture of two models, each one trained on a separate region.

As pointed out by reviewers, the formulation of the optimal strategy rests upon a relatively stylized setting, which assumes a partition into pre-defined "easier" and "harder" regions, which stay static throughout the course of training. Furthermore, while the empirical results are encouraging, reviewers mentioned they did not reflect practical (real-world) applications (e.g. multi-round active learning). Addressing at least one of these limitations would be required for acceptance.

justification_for_why_not_higher_score: The analyzed learning setting is stylized and likely does not hold in practice, but is amenable to theoretical analysis and provides a starting point to build intuition.  If the empirical results would have shown the value of the proposed method convincingly in a more practical setting, then that would justify a higher score 
justification_for_why_not_lower_score: N/A

From: ICLR.cc/2023/Conference/Paper398/Authors
title: Response to myFd
comment: We agree that adding more practical applications can significantly enhance the method's impact, and this is left for future work (which we intend to seriously pursue). 

In its first step, which is executed in each active learning round, MiSAL decides (taking into account the prescribed budget) which active learning strategy it should use given the current budget. Ideally, it would then choose the best strategy that is available from the state of the art, and thus achieve the promised benefits.

However, in order to provide a fair comparison in this work, we evaluated all methods using datasets on which the original papers reported they perform well on. Both in the low and high-budget cases, this entails CIFAR-10/100, and subsets of ImageNet, as these datasets are the most widely used in the active learning literature. 

To the best of our knowledge, the low-budget strategies we used were not tested on datasets with prohibitive labeling costs. In our future work, we will pursue other datasets, and suitable low-budget methods, from areas such as medical imaging, to increase the impact of our work and seek practical applicability.

In the current revision, we added a short discussion of these points, to better portray the limitations of our method. These changes can be found in the first paragraph of Section 4.1 and the first paragraph of Section 4.3 in the revised manuscript.

From: ICLR.cc/2023/Conference/Paper398/Authors
title: Response to MDLU
comment: point by point response to the mentioned weaknesses:

---------------------

1) Can you please elaborate more on what you mean by saying that the theoretical analysis is built upon "weird assumptions"? As neural networks cannot be analyzed directly in a tractable way, we use a simplified model, with assumptions that were designed to new insights with regard to active learning. The hope is that while neural networks do not satisfy all the assumptions, they are sufficiently related such that insights from the model can be translated to insights for active learning in neural networks. This specific model and assumptions already appear in the literature (see [1]), and are shown in this paper and in other papers to have interesting insights regarding active learning.

    Specific points:

    a) In (2), we define the error of the simplified model, which is derived from the model definition.


    b) Learning each region independently allows us to easily model the concept of active learning: in this model, active learning translates to the question of which region should be over-sampled. This in turn allows us to perform the entire analysis. While, as you mentioned, this is not the case for neural networks, this assumption is not so unrealistic: this is a good abstraction of how nearest-neighbor-based models learn, which are often used as an inspiration for the analysis of neural networks.


    c) Our assumption is that models which are trained on more data points, on average, perform better than similar models trained on fewer data points from the same training distribution. In the original manuscript, this point may have been confusing, since we did not emphasize that the error in question is averaged over all training sets of a fixed size, which is crucial. Thank you for raising this concern. We clarified this point in the current submitted revision, in the revised Section 2.1.
        

    d) Both in the empirical section and in the theoretical section, the models are trained on the entire labeled data set $\mathbb{L}$ alone. The unlabeled dataset $\mathbb{U}$ is only used to ask for further annotations in the active step.   

-----------------------

2) The assumptions were indeed spread throughout the theoretical section in order to improve its readability -- we worried that clumping all the assumptions together at the beginning may overburden the reader, and confuse him regarding the point that the theory is trying to establish. However, after reading the reviews we now agree that your suggestion is better, and we restructured the theoretical section as you had suggested (see section 2.1 in particular).

-----------------------

3) The claim that "active learning mostly benefits from multiple rounds of interaction" is a rather strong claim. While many works indeed employ several rounds of active interactions, there are some works, mainly in the low-budget regime, which mostly benefit from a single iteration [1,2]. Hence, the analysis of a single round is very relevant. More importantly, note that the analysis of a single round can be readily extended to multiple rounds: one should simply employ our method several times, with varying sizes of budgets, consistently updating the size of the labeled set to reflect the fact that the learner has received labeled data in the previous rounds.

-----------------------

4) There is a key difference between model selection and our work: in model selection, the learning model itself is chosen, to best address the problem at hand. In our settings, the model is fixed, and the AL strategy is chosen, which only affects the labeled data that the fixed model has access to. As such, while the model selection is interesting and important, we did not feel that an elaborate discussion is directly relevant to our work, in particular given the limited space available to our manuscript. 

---------------------

[1] Hacohen et al, Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets, ICML 2022

[2] Mahmood et al, Low budget active learning via wasserstein distance: An integer programming approach, ICLR 2022

From: ICLR.cc/2023/Conference/Paper398/Authors
title: Response to EZMG
comment: point by point response to the mentioned weaknesses:

----------------------------
1) In (2), we analyze the error of a mixture model, trained separately on 2 disjoint regions of the distribution. However, in order to simplify the theoretical analysis, these errors can not be arbitrary functions, but rather should come from the same family (for example, both these errors could be some exponentials, but not necessarily the same function). To formalize this connection, we introduce the assumption in (1).
    
    It indeed suggests that the error in each region - average over all training sets of a fixed size - depends only on the number of training points in the region. However, when considering a mixture of 2 or more such regions, the error depends also on the distribution of the data between the regions, which allows an analysis of AL approaches in both meaningful and tractable ways.

    We agree that the steps to guide the reader from (1) to (2) are somewhat confusing in the original manuscript. In the current revision, we restructured the assumptions section of the theoretical analysis, hopefully making this transition clearer and easier to follow. These changes can be found in Section 2.1.

----------------------------

2) Somewhat unintuitively, the budget size indeed plays a major role in deciding which AL policy is good or bad. There have been several works (refer to [1,2]) showing that different AL strategies are good for different budgets, showing how some strategies are expected to fail on low budgets and succeed on high budgets and vice-versa.

    Still, the question of what other factors may affect the success of an active learning method is a valid one. In particular, as our suggested algorithm indeed depends on pre-trained features, we agree that the method should be tested with different feature generation techniques, to better understand their role in the decision process of the algorithm. As such, we conducted experiments running our suggested method on CIFAR-10 but using several different generation techniques, as you suggested. The results of these experiments were added to App C.2.1 in the newly uploaded revision, with a clear reference in Section 4.

    In these experiments, we find that in all tested feature generation techniques, our suggested algorithm performs similarly; in particular, the same AL strategy is picked in all given budgets regardless of the features used. These results suggest that the feature space itself plays a rather minor role in the outcome of the algorithm within the domain investigated in our empirical study and that it is indeed mostly impacted by the size of the budget.

----------------------------

[1] Hacohen et al, Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets, ICML 2022

[2] Chan et al, On the marginal benefit of active learning: Does self-supervision eat its cake?, ICASSP 2021

From: ICLR.cc/2023/Conference/Paper398/Authors
title: Response to JaE6
comment: point by point response to the mentioned weaknesses:

1) The analysis done in the paper does not directly (or indirectly) assume that the hardness of a specific instance is fixed during training: we agree that such an assumption is indeed unrealistic. Instead, the assumption used in the paper is that some examples are learned faster than others. In other words, if we train a model on a given dataset, there will be some examples that will be classified correctly before other examples in a consistent and reliable manner, across different conditions.

Although this assumption may seem equally unrealistic, it is actually well supported by the literature: there is a large body of work on the subject of 'simplicity bias', suggesting that "simpler" examples are consistently learned before more complex examples. Please refer to [1,2,3,4,5] below.

This refined assumption was already mentioned in section 2.1 of the previous manuscript. However, as several reviewers have had questions about the assumptions, we revised section 2.1 in the current revision. Hopefully, our revision clarifies the exact assumptions that are being used, and their plausibility.  

--------------------------------

2) The suggested analysis directly motivates our budget-aware algorithm, which outperforms the previous state-of-the-art. We would argue that this by itself could be counted as "significant new insights for better understanding budget-aware AL".

--------------------------------

3) There was an abuse of notation in algorithm 1, which has been fixed in the current revision. Thank you for pointing it out to us.
$m=B-|L|$ denotes the active set size, meaning the number of examples to be added in each active step. It should appear only in Alg.2, where AL is employed. In Alg.1, we simply remove a small fraction of the data from $L$, which does not necessarily depend on $m$. This was also written in the original manuscript, in the last paragraph of section 3.1. We changed the notations in Alg.1 to include $\epsilon$ instead of $m$, in order to avoid this confusion.

Note that the algorithms do not imply that $|L|>B/2$, as this indeed may not be satisfied in many real-world AL applications.

--------------------------------

4) State-of-the-art AL methods are usually good only for a range of budgets (typically, either high-budget or low-budget). MiSAL simply suggests a way to evaluate which strategy is suited for the current budget. As such, it cannot surpass the particular method that achieves state-of-the-art performance in any specific budget. Rather, and unlike other methods, it reaches state-of-the-art performance in **every budget**, which constitutes the main contribution of this paper.
    
--------------------------------

[1] Gunasekar et al, Implicit Bias of Gradient Descent on Linear Convolutional Networks, NIPS 2018

[2] Soudry et al, The implicit bias of gradient descent on separable data, JMLR 2018

[3] Guillermo et al, Deep learning generalizes because the parameter-function map is biased towards simple functions, ICLR 2019


[4] Wei et al, The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks, NIPS 2020

[5] Yuan et al, Towards Understanding the Spectral Bias of Deep Learning, IJCAI 2021

From: ICLR.cc/2023/Conference/Paper398/Reviewer_JaE6
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: In this paper, the budget-aware active learning (AL) strategy is studied. The work is based on the discovery of existing studies, such that when the budget is small, querying easy instances would be beneficial, otherwise querying hard instances would be beneficial instead. The paper studies in theory about the best query strategy when the proportions of easy and hard instances vary under the budget limited setting. An AL algorithm is also proposed, whose performance is tested with benchmark datasets.
strength_and_weaknesses: Strength:

1. In my view, the paper is well-motivated. budget-aware AL is a very interesting research topic receiving few studies until recently. 

Weaknesses:

1. I think the analysis in the paper relies on unrealistic assumption: the paper assumes that whether an instance is hard or easy is fixed during training. However, in practice, hardness and easiness are related to the model, thus is a changing property during learning. 

2. Even though the analysis is delicate, it does not reveal significant new insights for better understanding budget-aware AL.

3. In Line 5 of algorithm 1, there should be $m$ instances removed from the labeled data, which implies that $|L| > m$, in which $|L|$ is the number of the labeled data. Meanwhile, according to algorithm 2, $m = B- |L|$. The above two facts indicate that $|L| > B/2$, which means that the labeled data should be larger than half of the budget. This assumption may not be satisfied in many real-world AL applications. 

4. Even though it is desirable to see that many main-stream AL approaches are included as baselines, the experimental results do not show significant improvement over them.
clarity,_quality,_novelty_and_reproducibility: I think significant effort should be done to improve the clarity of the paper, in special the theoretical part. The notations are quite cluttered, making the results relatively hard to read and follow. For example, there are two quantities represent the query strategy, $\hat q$ and $\hat r$, which would lead to confusion in following the results (In special, I think the definition of $\hat r$ is not clearly stated in the paper) Furthermore,  I wonder how the equation $\alpha < \frac{p}{1-p}$ below Equation (2) is obtained, when the function $E$ is not given. The derivation is missing in the paper.
summary_of_the_review: To summarize, even though I think the paper studies a very interesting problem, the technical quality is somehow limited due to the issues stated above.

--- after rebuttal ---

The author has put efforts on fixing notational issues, so my technical questions seem to be fixed. This makes me feel that my score can be increased from 3 to 5. On the other hand, even though I think the paper studies an interesting problem (budget-aware active learning), I think that the assumption issue has not been fully fixed. This is the reason I keep my score below the acceptance line.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper398/Reviewer_EZMG
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: It is generally well understood that different active learning strategies perform better than others at different stages of a model’s maturity. This paper identifies the effect of budget on active learning strategy, proposes a model to predict which active learning strategy should be used for what budget ranges, and then develops a mixed AL strategy based on this proposal. Numerical results demonstrate the performance of the proposed strategy to be on a Pareto front over all budget ranges.

strength_and_weaknesses: Strengths:
- The empirical results are nice and match up to the theoretical analysis that diversity versus uncertainty have different roles for active learning. 

Weaknesses:
- The assumption for eq (1) seems very strong. From my reading, it suggests that the only thing that matters for generalization error is training set size. If that is the case, why even perform active learning?   It seems to me that the authors want to study (2), which makes more sense as a model to work with, but the steps to guide a reader from (1) to (2) are unclear.

- Intuitively, it is not clear why the budget alone determines whether an AL policy is good or not. What role does the self-supervised pre-train have in changing/controlling the budget? Especially since the method depends on pre-trained features, it would be nice to test with different feature generation techniques, including SSL and the classical AL setup of using features from the previous iteration.

clarity,_quality,_novelty_and_reproducibility: The paper provides an intuitive strategy that is clear. Implementation details are provided in the appendix.
summary_of_the_review: The paper adds to the increasingly popular active learning literature that suggests simple methods are often more desirable than sophisticated. It presents an intuitive model for combining other strategies and then presents a practical means of implementing it. The results also support the motivation.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper398/Reviewer_MDLU
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the problem of selecting appropriate deep active learning algorithms with respect to a given budget (low or high). The authors start by theoretically analyzing mixture models and then propose algorithms that can determine the budget regime and allocate samples to different sampling strategies. The authors run experiments to show the efficacy of their proposed method.
strength_and_weaknesses: Strength: 
1. The authors study an interesting problem that is important to active learning---how to adjust active learning strategies with respect to the given budget. 
2. The authors have run many empirical simulations targeting the proposed problem, and the empirical results show the efficacy of their proposed algorithm.

Weaknesses: 
1. I'm not sure if I fully understand the theoretical analysis, but they seem to be built upon wired assumptions. For example, I'm not sure why we should target to optimize Eq. (2), and how it is derived. Why would I learn independently in each region from respective training data points instead of combining all training data points? You already assume in Eq. (1) that the excess error decreases wrt the number of samples. Also, it seems that, in the experimental section,  neural networks are trained on all labeled data points, which doesn't align with the studied theory.
2. Multiple assumptions are made across sections 2.1, 2.2, and 2.3. I strongly suggest the authors highlight all assumptions made in order to conduct theoretical analysis (e.g., write Assumption 1, Assumption 2, etc.), and clearly state which theoretical analyses are developed under which assumptions. As for now, it seems that all assumptions are hidden in the text (e.g., starts with "we assume ...", and "further assume ... ").
3. Both theoretical analyses and empirical studies seem to focus on the active learning setting with a single interaction. However, we know that active learning mostly benefits from multiple rounds of interactions. For instance, binary search is viewed as a powerful active learning strategy to learn a 1-d threshold function. However, you cannot run binary search with a single round of interaction. 
4. The proposed algorithm seems to be closely related to the "model selection" problem studied in passive/online learning---how to automatically switch between algorithms in learning. However, I didn't see any discussion on the model selection problem.
clarity,_quality,_novelty_and_reproducibility: Based on the weaknesses pointed out above, I believe both clarity (e.g., regarding assumptions made in order to derive theoretical results) and quality (e.g., how Eq. (2) is derived, why only study active learning with a single interaction) can be improved. I believe the novelty is limited except for studying a novel problem of selecting active learning algorithms wrt the given budget.
summary_of_the_review: This paper studies an interesting problem in active learning. However, based on the weaknesses pointed out above, I am inclined to reject this paper and suggest the authors make a major revision.

====after rebuttal====

I would like to thank the authors for their rebuttal. However, I will keep my current scores and believe this paper can be made stronger (e.g., by weakening the assumptions considered in this paper). Also, the proposed approach is indeed closely related to model selection in the online learning setting where one can also choose the learning algorithms besides simply choosing the model classes (e.g., see [1]).

[1]  Pacchiano et al. "Model Selection in Contextual Stochastic Bandit Problems"
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper398/Reviewer_myFd
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: The paper introduces a novel approach to active learning (AL) that automatically detects and adapts-to the best AL strategy for the budget at hand (i.e., for the final size of the training set). This approach is based on a mixture model that has a strong theoretical foundation and promosing empirical results.
strength_and_weaknesses: The paper introduces a novel, compelling approach to active learning. Its main contribution is to remove the need for an a-priori guess of the best querying strategy for active learning. The proposed approach has clear theoretical justification and performs well in the chosen empirical evaluation  

The main weakness of the paper comes from it's lack of compelling real-world application, which impacts its impact/significance. Even though the results in Table 2 are promising, it is hard to justify the practical application of MiSAL for either low or high budgets. On the three chosen datasets, the performance at low/medium budget is quite abysmal, thus clearly lacking practical applicability; it would be a game changer if the paper also used a real-world dataset on which low-budgets are mandatory (due to, say, prohibitive labeling costs) but high-performance is achievable. Similarly, at high budgets, the performance of MiSAL is far behind the best results on these three datasets; as such, a real world domain where, at high-budget, MiSAL offers a performance competitive to fully supervised learning would make the authors' argument quite compelling 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and organized. The approach appears to be novel, the theoretical justification is strong, and the empirical evaluation emphasizes the approach's potential. 
summary_of_the_review: Due to the novelty and potential future impact of the proposed approach, I am inclined to accept this paper. However, the addition of two real-world datasets (for low and high budgets, respectively) would make the paper significantly more compelling.

-- After discussion phase --

After carefully reading the authors’ answers to all reviewers, I would like to downgrade my recommendation to a 6: I still think that the ideas are good, but the execution is mediocre.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
details_of_ethics_concerns: n/a
recommendation: 6: marginally above the acceptance threshold