Revisiting Uncertainty-based Query Strategies for Active Learning with Transformers (2022)

Abstract: Active learning is the iterative construction of a classification model through targeted labeling, enabling significant labeling cost savings. As most research on active learning has been carried out before transformer-based language models (“transformers”) became popular, despite its practical importance, comparably few papers have investigated how transformers can be combined with active learning to date. This can be attributed to the fact that using state-of-the-art query strategies for transformers induces a prohibitive runtime overhead, which effectively nullifies, or even outweighs the desired cost savings. For this reason, we revisit uncertainty-based query strategies, which had been largely outperformed before, but are particularly suited in the context of fine-tuning transformers. In an extensive evaluation, we connect transformers to experiments from previous research, assessing their performance on five widely used text classification benchmarks. For active learning with transformers, several other uncertainty-based approaches outperform the well-known prediction entropy query strategy, thereby challenging its status as most popular uncertainty baseline in active learning for text classification.

## introduction collecting labeled data for machine learning can be costly and time-consuming. a key technique to minimize labeling costs has been active learning, where an oracle (e.g., a human expert) is queried to label problem instances selected that are deemed to be most informative to the learning algorithm's next iteration according to a query strategy. active learning is characterized by the realworld machine learning scenario in which large amounts of training data are unavailable, which may explain why comparably little research has investigated deep learning in this context. the recent widely successful transformer-based language models can circumvent the limitations imposed by small training datasets @xcite . pre-trained on large amounts of unlabeled text, they can be fine-tuned to a given task using far less training data than when trained from scratch. however, their high number of model parameters renders them computationally highly expensive, for query strategies that are targeted at neural networks or text classification @xcite , resulting in prohibitive turnaround times between labeling steps. in this paper, we systematically investigate uncertainty-based query strategies as a computationally inexpensive alternative. despite their relative disadvantages in traditional active learning, when paired with transformers, they are highly effective as well as efficient. our extensive experiments assess a multitude of combinations including state-of-the-art transformer models bert @xcite and distilroberta @xcite , five well-known sentence classification benchmarks, and five query strategies. 1 2 related work uncertainty-based query strategies used to be the most common choice in active learning, using uncertainty scores obtained from the learning algorithm @xcite , estimates obtained via ensembles @xcite , or prediction entropy @xcite . more recentlypredating transformers-neural network-based active learning predominantly employed query strategies that select problem instances according to (1) the magnitude of their backpropagation-induced gradients @xcite , where instances causing a high-magnitude gradient inform the model better, and (2) representativitybased criteria (e.g., coresets @xcite ), which select instances from a vector space to geometrically represent the full dataset. for today's deep neural networks, ensembles are too computationally expensive, and prediction entropy has been observed to be overconfident @xcite . the exception are flat architectures, where, among others, @xcite showed fasttext @xcite to be effective, well-calibrated, and computationally efficient. prior to transformers, query strategies relying on expected gradient length @xcite achieved the best results on many active learning benchmarks for text classification @xcite . gradients depend on the current model, which means, when used for a query strategy, they scale with the vast number of a transformer's parameters, and moreover, they need to be computed per-instance instead of batch-wise, thereby becoming computationally expensive. the cost of ensembles, the adverse scaling of network parameters in gradient-based strategies, and a history of deeming neural networks to be overconfident effectively rule out the most predominantly used query strategies. this might explain why transformers, despite the success of fine-tuning them for text classification @xcite @xcite , have only very recently been considered at all in combination with active learning @xcite @xcite . all of the related works mitigate the computationally complex query strategies by subsampling the unlabeled data before querying @xcite @xcite , by performing fewer queries with larger sample sizes @xcite , or by tailoring to less expensive settings, namely binary classification @xcite . subsampling, however, introduces additional randomness which can aggravate comparability across experiments, and large sample sizes increase the amount of labeled data, which is contrary to minimizing the labeling effort. due to this computationally challenging setting, the uncertainty-based prediction entropy query strategy @xcite is therefore a frequently used baseline and a lowest common denominator in recent work on active learning for text classification @xcite @xcite @xcite @xcite . apart from being employed as base-lines, uncertainty-based query strategies have not been systematically analyzed in conjunction with transformers, and moreover, comparisons to the previous benchmarks by @xcite have been omitted by the aforementioned related work. our work not only closes this gap, but also reevaluates the relative strength of uncertainty-based approaches, including two recently largely neglected strategies, thereby challenging the status of prediction entropy as the most popular baseline. ## transformer-based active learning the goal of active learning is to minimize the labeling costs of training data acquisition while maximizing a model's performance (increase) with each newly labeled problem instance. in contrast to regular supervised text classification ("passive learning"), it operates iteratively, where in each iteration (1) a so-called query strategy selects new instances for labeling according to an estimation of their informativeness, (2) an oracle (e.g., a human expert) provides the respective label, and (3) a learning algorithm either uses the newly labeled instance for its next learning step, or a model is retrained from scratch using all previously labeled instances. this work considers pool-based active learning @xcite , where the query strategies have access to all unlabeled data. notation-wise, we denote instances by x 1 , x 2 , . . . , x n , the number of classes by c, the respective label for instance x i by y i (where ∀i : y @xmath0, . . . , c}), and p (y i |x i ) is a probability-like predicted class distribution. ## evaluation we evaluate five query strategies in combination with bert, distilroberta and two baselines. ## conclusions an investigation of the effectiveness of uncertaintybased query strategies in combination with bert and distilroberta for active learning on several sentence classification datasets shows that uncertainty-based strategies still perform well. we evaluate five query strategies on an established benchmark, for which we achieve results close to state-of-the-art text classification on four out of five datasets, using only a small fraction of the training data. contrary to current literature, prediction entropy, the supposedly strongest uncertainty-based baseline, is outperformed by several uncertaintybased strategies on this benchmark-in particularly by the breaking ties strategy. this invalidates the common practice of solely relying on prediction entropy as baseline, and shows that uncertainty-based strategies demand renewed attention especially in the context of transformer-based active learning.