Paper: Is margin all you need? An extensive empirical study of deep active learning on tabular data (ICLR-2024)

Reviewer comment (Experiments):
Actually, deep neural networks can not achieve good performance on tabular data. This may influence the fairness of the experiments.

Author response:
As we note in the main text, the SCARF paper compares deep NNs to gradient boosted decision trees and shows that on the contrary, deep NNs are competitive (especially when pre-trained). Could you please post experimental results or references that contradict this?

Reviewer comment (Experiments):
Deep neural networks can not achieve good performance on tabular data. Will this influence the fairness of the experiments?

Author response:
As we note in the main text, the SCARF paper compares deep NNs to gradient boosted decision trees and shows that on the contrary, deep NNs are competitive (especially when pre-trained). Could you please post experimental results or references that contradict this?

Reviewer comment (Experiments):
The conclusion section states that margin sampling outperformed other strategies in deep neural network models. However, it is important to note that the experiment solely relied on a specific model architecture (SCARF) as the backbone. Consequently, it is recommended to diversify the model architectures used in the experiments to ensure the generalizability of the findings.

Author response:
Thank you for the review.

To clarify, the model architecture is a feedforward ReLU network. SCARF is an architecture agnostic pre-training algorithm. We show results with and without SCARF pre-training. Since our domain is tabular data, within the realm of deep models, feedforward networks are the standard choice (though niche alternatives exist) -- which is to say, we don't think it'd make much sense to try CNNs or Transformers here.

Reviewer comment (Experiments):
It is advisable to incorporate an ablation study and detailed analysis to assess the potential influence of pre-training backbones on the performance of various active learning strategies, including margin sampling. This additional analysis would provide valuable insights into the influence of pre-training on the efficacy of different AL methods.

Author response:
Our paper precisely ablates the effect of SCARF pre-training in every experiment. SCARF is the most appropriate pre-training scheme to consider for this paper as it is agnostic to modality and was shown to work well precisely for the type of real-world tabular data we use here.

Reviewer comment (Experiments):
To enhance the persuasiveness of the conclusions, it is crucial to conduct more experiments encompassing a broader range of active learning settings. The current results are derived from a limited set of active learning settings, and expanding this scope would enhance the overall validity and generalizability of the study's findings.

Author response:
*RE: "To enhance the persuasiveness of the conclusions, it is crucial to conduct more experiments encompassing a broader range of active learning settings."*. We believe the analysis is pretty thorough; could you suggest specifically which settings are missing?

Reviewer comment (Theory):
For the results to be truly groundbreaking and instructive, the paper should strive for innovation and provide additional theoretical analysis and in-depth discussions. While the current implication is that margin sampling excels as a general active learning strategy, it would be significantly enhanced by offering a deeper understanding of why margin sampling outperforms other methods. Moreover, practical suggestions should be included to guide future research in selecting the most appropriate active learning techniques for specific scenarios.

Author response:
*RE: "...the paper should strive for innovation and provide additional theoretical analysis and in-depth discussions."* We note in paragraph 3 of the Related Works that AL methods like Margin can be difficult to analyze theoretically and that there is little theoretical understanding of the method. We attempt to provide a deeper understanding in the "Deeper Analysis" section -- we say, "While it can be challenging to pinpoint why one method outperforms another, we attempt to provide some insight by comparing which examples each method chooses to acquire."
We think the guidance this paper provides is clear -- practitioners doing active learning on tabular data should opt for Margin sampling.

Reviewer comment (Evaluation):
BADGE outperforms all methods when existing statistically remarkable difference between them, while this paper do not analyze the reason, which makes the conclusion lacking of convince.

Author response:
*RE: BADGE*. Indeed, BADGE performs well in our experiments. However, looking at both the win and relative improvement plots in Figure 1, the difference with Margin is not too significant in our opinion. We would like to echo what Andreas Kirsch (a public reviewer) noted about BADGE. While Margin is linear as it involves doing a single forward pass for unlabeled points and sorting them by a single scalar value (the margin), BADGE requires a forward pass to infer labels and a backward pass to compute loss gradients for each unlabeled sample. It then needs to cluster these gradients using k-means, which typically takes $O(nkdi)$, where $n$ is the number of $d$-dimensional vectors ($d$ is the dimensionality of the penultimate layer embedding), $k$ is the number of samples to acquire (i.e. number of clusters), and $i$ is the number of iterations needed to converge. $i$ can vary greatly in practice, and can make the effective runtime of k-means either linear or super-polynomial.  
Given the little performance difference between Margin and BADGE but the significant additional computational overhead of the latter, our recommendation to practitioners remains Margin. We will update the paper with discussion on this point.

Reviewer comment (Experiments):
What is the criteria for choosing comparison methods? Margin, Entropy and Least Confidence are all based on uncertainty, which are somewhat repeatedly. As far as I know, there are other kinds of deep active learning methods. Adding them in it will improve the convince of this work.

Author response:
Thank you for the review.

We benchmarked 16(!) diverse AL methods that are popular or purported as state-of-art. We believe these baselines provide great coverage.

*RE: "As far as I know, there are other kinds of deep active learning methods"*. Could you please specify precisely which ones and why you think it is worthwhile to include them?

Reviewer comment (Experiments):
The paper assesses various AL algorithms within the framework of tabular datasets. It's important to note that the outcomes may not be universally applicable, as the study does not investigate their potential implications in other contexts or domains, such as image data and others, potentially limiting its broader relevance.

Author response:
Thank you for the detailed review.

The focus of this work is a large-scale analysis specifically on real-world tabular data. We think this is much needed as a lot of the other active learning papers have focused solely on images or text. We have good evidence that Margin works well in these other domains. For example, in paragraph 2 of the Related Works, we note that "Hu et al. (2021) showed that margin attained the best average performance of the baselines tested on two image and three text classification tasks across a variety of neural network architectures and labeling budgets."

Reviewer comment (Evaluation):
While the paper underscores the effectiveness of margin sampling, it falls short in offering comprehensive practical guidelines and actionable recommendations for practitioners seeking to make informed choices regarding AL algorithms in real-world applications.

Author response:
Thank you for the detailed review.

The actionable recommendation the paper is making for practitioners is precisely that they should use Margin sampling: "Margin has no hyper-parameters and is consistently strong across all settings explored. We cannot recommend a single better method for practitioners faced with data labeling constraints, especially in the case of tabular data, or baseline to be benchmarked against by researchers in the field."

Reviewer comment (Experiments):
How might the paper be enhanced to investigate the transferability of the studied AL algorithms to diverse domains, beyond tabular datasets, to provide a more comprehensive understanding of their applicability across different contexts? Also, it may be very helpful if the paper can provide more comprehensive and practical guidance to assist practitioners in effectively selecting and implementing AL algorithms in real-world applications.

Author response:
Thank you for the detailed review.

The focus of this work is a large-scale analysis specifically on real-world tabular data. We think this is much needed as a lot of the other active learning papers have focused solely on images or text. We have good evidence that Margin works well in these other domains. For example, in paragraph 2 of the Related Works, we note that "Hu et al. (2021) showed that margin attained the best average performance of the baselines tested on two image and three text classification tasks across a variety of neural network architectures and labeling budgets."

The actionable recommendation the paper is making for practitioners is precisely that they should use Margin sampling: "Margin has no hyper-parameters and is consistently strong across all settings explored. We cannot recommend a single better method for practitioners faced with data labeling constraints, especially in the case of tabular data, or baseline to be benchmarked against by researchers in the field."