Paper: Time- and Label-efficient Active Learning by Diversity and Uncertainty of Probabilities (ICLR-2024)

Reviewer comment (Novelty):
The paper seems to overlook several crucial references relevant to the discussed topic. Fundamental concepts such as the theoretical importance of the near decision boundary have been comprehensively explored and articulated in previous works, notably in references [1] and [2]. These pivotal papers, along with others, offer profound insights that would augment the paper’s foundational grounding. Moreover, the proposition of a linear-time scalable algorithm, a core element presented in the paper, has previously been introduced and elaborated upon in reference [3]. In addition to that, an influential work cited as [4] such as PowerBALD/or PowerEntropy adopting a sampling similar to Eq (6) but with better originality has also proposed a scaling approach that intriguingly maintains the algorithm’s linear-time complexity.

Author response:
Thanks for pointing us to these interesting works, we will include them in our discussion. Note that, e.g., PowerBALD is compared in [4] to BADGE, yielding sometimes worse quality than BADGE. As our experiments show that FALCUN is superior to BADGE, worse methods than BADGE were excluded from our competitors.

Reviewer comment (Experiments):
The active learning results presented in the paper for LeNet on datasets like EMNIST or RepeatedMNIST seem somewhat unconvincing. Typically, in prevailing literature and experiments, around 300-500 images are required (not > 1000 images) to achieve accuracy comparable to supervised learning methods on these datasets. The reported results in the paper appear to deviate from these established benchmarks. It may be beneficial to revisit and scrutinize the experimental setup, methodology, and the specific implementation of LeNet in the active learning context to ensure that the presented results are robust, reliable, and in alignment with existing standards and expectations in the field. This could enhance the credibility and persuasiveness of the results and the overall contributions of the paper.

Author response:
Note that such accuracy values depend heavily on the exact settings. While choosing the best out of all models for each data set individually can lead to higher accuracy, this does not allow to conclude one method’s superiority over another. 
We rather opted for a fair and broad comparison, where the models and datasets are fixed, and we evaluate all AL methods for these fixed settings. This does not bias the ranking among methods’ qualities, but excludes effects that come from choosing overfitted models. 
This ensures a robust quality in the real world, as users usually do not know beforehand the very best model and might opt for the one they have already implemented or know best. 
Furthermore, there are different versions of those data sets with different splits. We state which versions we use in Section 4. E.g. for RepeatedMNIST we use a much larger redundancy ratio and do not train until a training accuracy of 99% is reached for gray scale images.

Reviewer comment (Experiments):
The performance of BADGE as presented in the paper raises some questions. It seems that with appropriate hyperparameter settings, BADGE should be capable of delivering much improved results. This discrepancy suggests that there might be room for optimizing the configuration of BADGE in the experiments, ensuring that it operates under the most suitable conditions for a fair and rigorous comparison. To uphold the integrity and reliability of the comparative analysis, it might be beneficial to revisit and fine-tune the hyperparameters used with BADGE, ensuring that its performance is accurately represented and evaluated against its full potential.

Author response:
We use similar settings as proposed in the papers of the competitors such as BADGE or AlfaMix. Please note, that hyperparameter tuning in the Active Learning setting is not comparable to a fully supervised setting. Especially in the AL setting, there is less a priori knowledge about the data than in the fully supervised setting. Thus, robustness is important and active learning strategies that should be usable in real world need to perform well over a range of models, data sets, and settings - which is, what we achieved with FALCUN. Note, that we also did not optimize any hyperparameters for FALCUN.

Reviewer comment (Theory):
What criteria determine the **most informative region**? Is a point considered most informative solely based on its proximity to the decision boundary?

Author response:
It is determined by the proximity to the decision boundary as well as the diversity to already chosen instances in the query batch. This leads to the selection of uncertain instances while it minimizes information overlap. For an elaborated discussion, see Section 2.2 in the paper.

Reviewer comment (Theory):
**Could the authors please provide a more explicit explanation regarding the source of diversity in the proposed formula?** The margin considers two class boundaries. It might not be sufficiently generalizable. It would be beneficial to have a more explicit elucidation on how diversity is incorporated and manifests itself within the algorithmic design. Understanding the origins and implementation of diversity within the formula can enhance the comprehension of its functionality and overall impact on the method's performance. An in-depth clarification would contribute to a more robust and insightful evaluation of the proposed approach’s effectiveness and novelty.

Author response:
With respect to diversity, the margin uncertainty is only used as an initialization for the diversity score as it represents the distance to the decision boundary. However, over the remaining course, the diversity aspect is incorporated based on the  L1 distance between the probability vector of two instances. Intuitively, this captures the difference between the most prevalent concepts. 
We want to refer to Equations (3) and (5), where the inclusion of diversity is stated. We will include a longer discussion on why a high distance in probability space corresponds to diverse instances.

Reviewer comment (Experiments):
It seems crucial to **re-evaluate the experiment setups in the study to enhance the reliability and comprehensiveness of the findings**. Including a broader selection of benchmarks in the evaluation process would be instrumental in demonstrating the robustness and versatility of the proposed method across varied scenarios. **A more diversified array of benchmarks** will not only contribute to a deeper, better understanding of the method's performance but also bolster the study's overall credibility and impact. Therefore, revisiting and expanding the experiment setups with additional benchmarks is a highly recommended step to enrich the empirical validation of the study.

Author response:
We again disagree that the experimental evaluation is too narrow, but we will include CIFAR10 and Resnet50 to expand the evaluation.

Reviewer comment (Experiments):
Section 4.4 discusses the influence of $\gamma$ parameter, indicating that **different benchmarks may require distinct $\gamma$ parameter values for optimal performance**. This aspect raises a practical concern: how can users effectively determine or choose an appropriate $\gamma$ value a priori for various benchmarks? The ability to discern and select suitable parameters is crucial for the method's practical applicability and usability in real-world scenarios. Clarification or guidance on this matter would significantly enhance the method’s practical value.

Author response:
As explained in Sec. 4.4., the value of gamma is not fine-tuned. In the ablation study in Table 3, we show that the performance of FALCUN is fairly similar and stable for diverse gammas in a reasonable range. Gamma does not have to be tuned but should just be used with the proposed default value of 10, as suggested in the paper (Sec. 4.4).

Reviewer comment (Theory):
The key methodological motivation for the algorithm seems to be to derive a function that balances the uncertainty of the existing model and the diversity between samples selected in the current batch. However, there is no theoretical justification for why the proposed algorithm is a good strategy.

Author response:
"W1) However, there is no theoretical justification for why the proposed algorithm is a good strategy. "
- While we did not label the theoretical justifications as such explicitly, they are contained in the paper. We will incorporate them in a separate paragraph or subsection for the camera-ready version to elaborate on them in more detail as follows:
  - (1) Calculating diversity in the probability space is faster than in the feature space and it gives high similarity between objects that might be diverse in feature space, but do belong to the same class with high probability. Such instances are, thus, not of high interest for annotation and can be avoided more easily and elegantly.
  - (2) Using margin as initialization for diversity is advantageous because margin uncertainty has its maximal values along all decision boundaries between all classes. Thus, the set of chosen instances out of those with maximal margin uncertainty can be diverse. In contrast, e.g., entropy uncertainty follows a uniform distribution and, thus, has a global optimum. Samples chosen according to this distribution can not be diverse, as all of them will be close to this global optimum. Thus, chosen instances will be very similar or even redundant as shown in Fig. 1 in the paper. (see also explanation in response to Question 1)
  - (3) Our novel combination of an adaptive diversity score and the fixed uncertainty score within one active learning round ensures diversity with each included object and slowly transitions from the most uncertain to less uncertain regions. 
  - (4) Sampling from a probability rather than a deterministic top-k selection robustifies results and contributes to robust performance on a wide range of diverse settings and datasets (see also the Ablation in Section 4.4.)

Reviewer comment (Experiments):
The experiments primarily use relatively simple image datasets and small models, making it unclear whether these results would meaningfully transfer to more realistic datasets (e.g., ImageNet).

Author response:
"W2) Moreover, the experiments primarily use relatively simple image datasets and small models. 
It is not clear whether these results would meaningfully transfer to more realistic datasets (e.g., ImageNet). "
- We do not agree with your statement: we do use complex models, e.g. Resnet18. We included large datasets, e.g. our experiments on tabular data include data sets with ~800,000 instances, and complex image datasets, e.g. SVHN and especially the medical datasets.  However, we extend our experiments by CIFAR10 and Resnet50 as a further benchmark dataset in this domain. We furthermore want to note that we aim at a large variety of settings and robustness over all types of experiments, including both, very large as well as not-so-large data sets.

Reviewer comment (Evaluation):
The difference between the methods and benchmarks appears small in every dataset considered. Without rigorous theoretical justification or validation on hard benchmark problems, the argument for the proposed method in practice is unconvincing.

Author response:
"W3) Currently, the difference between the methods and benchmarks appear small in every dataset considered."
- FALCUN is notably better than state-of-the-art methods, as shown in our large variety of experiments and especially in the dueling matrix. Here, we did only count statistically relevant wins resp. losses, thus, while the differences might appear small, they are indisputably significant.
While this is a great achievement in itself, FALCUN additionally is much faster than the strongest competitors. 
For instance, compared to the very strong competitor BADGE, we are faster by one to two orders of magnitude (see Table 2) while having similar or better label efficiency.

Reviewer comment (Novelty):
The problem of dealing with two competing objectives, diversity and uncertainty, has been studied in recent works, but the paper misses a lot of this related literature (e.g., [1], [2]).

Author response:
"W4) Lacking rigorous theoretical justification or validation on hard benchmark problems, the argument for the proposed method in practice is unconvincing. The problem of dealing with two competing objectives, diversity and uncertainty, has been studied in recent works, and the paper misses a lot of this related literature (e.g., [1], [2]). This also leads to a missed opportunity for discussion and validation of how these two objectives trade off overall. What is the major contributor overall to selection? Does the uncertainty score dominate or the diversity score?"

Thank you for hinting at these works; we will add them to our related work section. We want to refer to Section 4.3 and Figure 6 in the paper, where we discuss this trade-off and elaborate in more detail in the following: 
  - Across AL rounds: In early iterations, the uncertainty scores might be less expressive due to very limited labels. Here, uncertainty scores among instances might be more similar. Therefore, the diversity aspect is considered more. 
  - Within one AL round: Initially, the most uncertain and diverse regions are considered. As more instances from the most uncertain regions are selected, the remaining uncertainty scores get smaller (while the diversity score always plays a crucial role, to avoid information overlap). Over the course of one AL round, the selection scheme slowly transitions from the most uncertain to the less uncertain (probably less interesting) regions. The figure displaying the relevance score depicts this behavior over multiple queries within one AL round.

Reviewer comment (Evaluation):
The lack of discussion and validation on how diversity and uncertainty trade off overall leads to a missed opportunity. It is unclear what is the major contributor to selection—does the uncertainty score dominate, or the diversity score?

Author response:
"W4) Lacking rigorous theoretical justification or validation on hard benchmark problems, the argument for the proposed method in practice is unconvincing. The problem of dealing with two competing objectives, diversity and uncertainty, has been studied in recent works, and the paper misses a lot of this related literature (e.g., [1], [2]). This also leads to a missed opportunity for discussion and validation of how these two objectives trade off overall. What is the major contributor overall to selection? Does the uncertainty score dominate or the diversity score?"

Thank you for hinting at these works; we will add them to our related work section. We want to refer to Section 4.3 and Figure 6 in the paper, where we discuss this trade-off and elaborate in more detail in the following: 
  - Across AL rounds: In early iterations, the uncertainty scores might be less expressive due to very limited labels. Here, uncertainty scores among instances might be more similar. Therefore, the diversity aspect is considered more. 
  - Within one AL round: Initially, the most uncertain and diverse regions are considered. As more instances from the most uncertain regions are selected, the remaining uncertainty scores get smaller (while the diversity score always plays a crucial role, to avoid information overlap). Over the course of one AL round, the selection scheme slowly transitions from the most uncertain to the less uncertain (probably less interesting) regions. The figure displaying the relevance score depicts this behavior over multiple queries within one AL round.

Reviewer comment (Evaluation):
Compared to identifying the most informative samples, the time cost of data evaluation is not the primary concern since the online data evaluation is not required.

Author response:
Unfortunately, even after discussion, we are not sure we understand your concern correctly, more precisely, what you mean with “(online) data evaluation”. In case “data evaluation” should mean the annotation by experts: Fast acquisition methods are important for settings where experts label the data in an online setting. E.g., as elaborated in the paper, when the expert is only employed for a certain time frame. Then, it is important that experts do not need to wait long for the acquisition function to query the labels for the next batch of samples. In contrast to other works that aim to reduce the retraining times (e.g. by fast model updates [1]), FALCUN reduces the time costs of step 1) of the three major components of AL: 1) Acquisition: Computation of the set which should be sent to the oracle 2) Labeling by experts 3) Retraining of the model.

Reviewer comment (Novelty):
In uncertainty component, the margin uncertainty lacks novelty.

Author response:
Using a well-performing uncertainty score as a basis is an established way to create novel hybrid approaches (e.g. [2,3]). The novelty comes not from using margin for the uncertainty but from the novel unique combination and dynamic adaption of the combination of both aspects, uncertainty, and diversity. Furthermore, while margin uncertainty per se is not novel, its application in the context of diversity is novel.

Reviewer comment (Evaluation):
As for diversity component, the rationale behind calculating the diversity score using the distance of predicted probabilities is still unclear. Why do the authors choose to measure the distance of predicted probabilities instead of using feature embeddings?

Author response:
Features have usually a much higher dimensionaity than the final classification layer (eg the MLP used for the tabular data, Resnet18, Resnet50). Thus, operating on the (lower-dimensional) probability space is faster than operating on the feature embeddings. 
- Furthermore, In the probability space, diverse concepts where the model is confident (and hence less relevant) are very similar. Calculating diversity in the features space does not capture this information directly and requires an additional step and careful design to exclude diverse objects for which the model already has a high certainty. In contrast, when operating on the probability space, we can directly focus on diverse and uncertain objects simultaneously.

Reviewer comment (Experiments):
The authors conducted experiments on small-scale datasets. We recommend verifying the efficacy of the proposed method on more challenging and large-scale benchmarks, such as CIFAR-10, CIFAR-100, and ImageNet.

Author response:
While we disagree with this statement, as explained below, we will incorporate CIFAR10 in the camera-ready version. Note that our experiments cover a broad range of diverse datasets (see Table 4 in the Appendix). For instance, OpenML-155 has a much larger data pool (~800,000 instances) than CIFAR10 and CIFAR100. SVHN has the same input dimensionality and pool size as CIFAR10 and CIFAR100. Including the medical datasets is very important because, in this domain, experts are very limited and costly, which is one of the most important active learning use cases.

Reviewer comment (Experiments):
The chosen baselines are reasonable but given the focus on computational efficiency, comparing to BatchBALD would have strengthened the empirical claims.

Author response:
As Bayesian approaches usually need several forward passes (e.g., Monte Carlo drop-out), they are tendentially computationally intensive especially when the unlabeled pool is very large. As one of the main aspects of our method is its speed, we focused on approaches that only need one forward pass for better comparability. BatchBALD is, especially for large batches, slow  [0] and was thus not included in the experimental evaluation. In case of acceptance, we will add it to our experiments in the camera-ready version.

Reviewer comment (Experiments):
For the colored image experiments, using pre-trained weights gives FALCUN an advantage over baselines that train from scratch. Comparisons should be fair.

Author response:
We agree that comparisons should be fair, and want to clarify that for the datasets where pre-training was used, all methods use pre-trained weights. We will make this more clear in the final version.

Reviewer comment (Experiments):
Some dataset choices like MNIST and FashionMNIST are dated. More modern complex datasets would better highlight benefits.

Author response:
Thank you very much for this suggestion. We agree that modern complex datasets are important: colored image datasets like SVHN, BloodMNIST, and DermaMNIST are examples of those and they are included in our experimental evaluation. Table 4 shows the broad range of tested datasets regarding size, input features, and number of classes. 
We additionally show FALCUN’s high performance on other, older benchmark datasets like MNIST to show the broad range of its use cases. 
Furthermore, we already started experiments on, e.g., CIFAR10 with the following results:

- Intermediate Results (Ordered by Final Accuracy)
- Falcun: Final Accuracy: 76.00; 13 minutes (=cumulated times of all acquisitions)
- BadgeSampling: Final Accuracy: 75.82; 363 minutes 
- CdalCS: Final Accuracy: 75.52; 10 minutes 
- AlphaMixSampling: Final Accuracy: 75.63; 28 minutes 
- RandomSampling: Final Accuracy: 74.00; 0.00 minutes

Reviewer comment (Experiments):
The empirical methodology uses a limited set of architectures. Testing on bigger models like ResNets would be important to substantiate scalability claims.

Author response:
In our experiments, we use ResNet 18 for all colored datasets.
For the experiments on CIFAR10 that we will include, we use ResNet50.

Reviewer comment (Experiments):
More rigorous hyperparameter tuning for baselines could lead to better optimized versions for fairer comparison with the proposed approach.

Author response:
As hyperparameter tuning usually requires a labeled set and these labels are not given in the AL setting, we followed the commonly used approach in the field. E.g., for tabular data we use setting as given in BADGE [1]

Reviewer comment (Theory):
The margin-based uncertainty measure is intuitive, but are there any theoretical justifications for using it over other alternatives like entropy or Bayesian uncertainty?

Author response:
Yes. Margin uncertainty has proven to be very effective, see, e.g. in “Is Margin All You Need? [...]” [2]. Furthermore, it allows for a diverse selection as it has no global minimum: E.g., [3], Fig. 5 visualizes some uncertainty functions. Least confidence and entropy have a global minimum at the one point in probability space where all probabilities are equal. Margin uncertainty, in contrast, does not have this global minimum, thus, there exist diverse samples within the set of minima. Margin uncertainty is, thus, one of the most suitable choices for combining diversity and uncertainty and especially for initializing the diversity score.

Reviewer comment (Theory):
For the diversity initialization and update, were other potential approaches considered? Is there a principled basis for the specific design choices made?

Author response:
Of course, there are several options to initialize and update the relevance score, out of which we selected the most meaningful one. Furthermore, we tested our method without initialization and with the variations shown in the ablation in Table 3.

Reviewer comment (Experiments):
How sensitive is FALCUN to the choice of the γ parameter for sampling from the relevance distribution? Is tuning gamma required for different datasets?

Author response:
In our experiments, we did not tune $\gamma$. Our ablation in Table 3 shows that the results are robust w.r.t. $\gamma$ values over a broad range. Thus, we suggest $\gamma$ = 10 as a default value, which we also used for all data sets.

Reviewer comment (Experiments):
What impact does the neural network architecture have on FALCUN's performance, if any? Does it generalize across model families like CNNs, MLPs etc?

Author response:
Similar to other AL approaches, FALCUN benefits from models that are suitable for the regarded dataset. As we show in our experiments, FALCUN’s results are on average better than those of any other tested method over a broad range of datasets and architectures, where we tested MLP, LeNet, ResNet18.
In Figure 14 in the appendix, we also show the results of using different architectures for the same dataset.

Reviewer comment (Experiments):
BatchBALD is a highly relevant Bayesian batch active learning method – would be good to compare against it. What advantages can FALCUN provide over Bayesian approaches?

Author response:
As explained above, Bayesian AL methods usually use Monte Carlo dropout and subsequently calculate statistics over multiple forward passes. However, when the unlabeled pool is large, this is computationally expensive. Furthermore, BatchBALD is slow for large batch sizes. [0]

Reviewer comment (Experiments):
The datasets seem heavily focused on MNIST variants – were results consistent on more complex, modern datasets? How was performance with higher input dimensionality?

Author response:
While the data sets are named *MNIST, all but repeatedMNIST are not variants of MNIST, but complex, diverse and modern Benchmark datasets. They cover different aspects and are very diverse:
- EMNIST has 6 times as many classes as MNIST. 
- The medical datasets contain challenging colored images (i.e., they have much higher input dimensionality, (d= 28x28x3=2352), thus they are much more complex than MNIST.
- They are also especially suited for the use case that we tackle (active learning with fast acquisition times) since experts in this field are very limited. 
- SVHN has input dimensionality 32x32x3 (same as CIFAR10). 
- Table 4 in the Appendix gives an overview of the key differences between datasets.

Reviewer comment (Experiments):
What was the hyperparameter tuning strategy for baselines? Would better optimized baselines affect relative comparisons?

Author response:
As explained above, we did not tune hyperparameters for a realistic active learning scenario following the common AL setting. We use similar or equal settings as in other Papers and did not optimize the setting for FALCUN.

Reviewer comment (Experiments):
How was robustness to things like random initialization and train-test splits evaluated?

Author response:
We evaluated all experiments on five seeds and report standard deviation to show whether the results are significant. Apart from that, we follow the official train test splits used in other works. We can emphasize this in the camera-ready version.