Towards Computationally Feasible Deep Active Learning (2022)

Abstract: Active learning (AL) is a prominent technique for reducing the annotation effort required for training machine learning models. Deep learning offers a solution for several essential obstacles to deploying AL in practice but introduces many others. One of such problems is the excessive computational resources required to train an acquisition model and estimate its uncertainty on instances in the unlabeled pool. We propose two techniques that tackle this issue for text classification and tagging tasks, offering a substantial reduction of AL iteration duration and the computational overhead introduced by deep acquisition models in AL. We also demonstrate that our algorithm that leverages pseudo-labeling and distilled models overcomes one of the essential obstacles revealed previously in the literature. Namely, it was shown that due to differences between an acquisition model used to select instances during AL and a successor model trained on the labeled data, the benefits of AL can diminish. We show that our algorithm, despite using a smaller and faster acquisition model, is capable of training a more expressive successor model with higher performance.

## introduction active learning (al) @xcite is an approach for reducing the amount of dataset annotation required for achieving the desired level of machine learning model performance. this is especially important in domains where obtaining labeled instances is expensive or wide crowdsourcing is unavailable. for example, annotation of clinical and biomedical texts usually requires the help of physicians or biomedical researchers. the time of such highly qualified experts is extremely valuable and should be spent wisely. straightforward annotation of datasets can be very redundant, wasting the time of annotators on unimportant instances. al alleviates this problem by asking human experts to label only the most informative instances selected according to the information acquired from a machine learning model. the algorithm for selection of such instances is called a query strategy, and a model used to estimate the informativeness of yet unlabeled instances is called an acquisition model. al starts from a small seeding set of labeled instances, which are used to train an initial acquisition model. a query strategy ranks unlabeled instances in a large pool according to a criterion that measures their informativeness based on the acquisition model output. one of the most widely adopted criteria is the uncertainty of the acquisition model on instances in question @xcite . eventually, top selected instances are presented to annotators, and this active annotation process iteratively continues. after labels are collected, we would like to train a model for a final application. in the same vein as @xcite , we call it a successor model. al can help reduce the amount of annotation required to achieve a reasonable quality of the successor text processing model by multiple times @xcite . recently, deep learning has given us a tool for solving one of the essential problems of al. when we start annotating, we have to build an acquisition model almost without insights from the data that could help us to do feature engineering or to introduce inductive bias. deep learning does not require feature engineering and transfer learning with deep pre-trained models like elmo @xcite , bert @xcite , and followers such as electra @xcite provide near stateof-the-art performance on a variety of tasks without any modifications to their architectures. however, deep learning introduces another problem related to computational performance. since al annotation typically is an interactive process, we have to train acquisition models and perform inference on a huge unlabeled pool of instances very quickly. this imposes constraints on the acquisition model size and entails another issue. ideally, the architectures of acquisition and successor models should be the same. @xcite demonstrate that when the acquisition model is different from the successor model, the performance of the latter one can degrade compared to the performance of the model trained on the same amount of annotation obtained without al. the performance drop in the case of acquisition-successor mismatch (asm) raises the question of whether al is a practical technique at all since the usage of different models on the annotated dataset is a common practice. the problem is complicated by a contradiction between the fact that the acquisition model is required to be as lightweight as possible to mitigate computational overhead and the successor model should be as expressive as possible because we apparently care about the quality of our final application. in this work, we propose a simple algorithm based on pseudo-labeling and demonstrate that it is able to alleviate the asm problem. moreover, we show that it is possible to substitute a resourceintensive acquisition model with a smaller one (e.g., take distilbert instead of bert) but train a more powerful successor model of an arbitrary type (e.g., electra) without loss of quality. this helps to accelerate the execution of al iterations and reduce computational overhead. we also find that the most time-consuming part of an al iteration with uncertainty-based query strategies can be the inference on the unlabeled pool of instances, while a set of the most certain instances usually does not change substantially from iteration to iteration. therefore, the straightforward approach to instance acquisition wastes much time on instances shown to be unimportant in previous iterations. we leverage this finding and propose an algorithm that subsamples instances in the unlabeled pool depending on their uncertainty scores obtained on previous al iterations. this helps to speed up the al iterations further, especially when the unlabeled pool is large. a series of experiments on text classification and tagging benchmarks widely used in recent works on al demonstrate the efficiency of the proposed algorithms. the contributions of the paper are the following: ## related work deep learning, to a large extent, has freed data scientists from doing feature engineering, which has been one of the essential obstacles to annotation with al. this advantage has sparked a series of works on deep active learning (dal) in natural language processing (nlp). @xcite conduct one of the first investigations on dal in sequence tagging tasks. they propose an efficient way of quantifying the uncertainty of sentences, namely maximal normalized log probability (mnlp), by averaging log probabilities of their tokens. they also address the problem of excessive duration of a neural network training step during an al iteration by interleaving online learning with training from scratch. in our work, we take mnlp as a query strategy for experiments on sequence tagging tasks since it has demonstrated a good trade-off between quality and computational performance. we consider that online learning can potentially be used as a complement to our algorithms. since the most time-consuming part of an al iteration can be model inference instead of training, in this work, we also pay attention to the acceleration of the inference step. several recent publications investigate deep pretrained models based on the transformer architecture @xcite , elmo @xcite , and ulmfit @xcite in al on nlp tasks @xcite @xcite . we continue this line of works by relying on pre-trained transformers since this architecture has been shown promising for al in nlp due to its good qualitative and computational performance. a few works have experimented with bayesian query strategies for al. @xcite , siddhant and lipton (2018), @xcite leverage monte carlo dropout @xcite for quantifying uncertainty of models. @xcite also apply the bayes by backprop algorithm @xcite for performing variational inference of a bayesian neural network. this approach demonstrates the best improvements upon the baseline but introduces large computational overhead both for training and uncertainty estimation of a model, as well as the memory overhead for storing parameters of a bayesian neural network. the query strategies based on monte carlo dropout do not affect the model training procedure and do not change the memory footprint. however, they also suffer from slow uncertainty estimation due to the necessity of making multiple stochastic predictions, while their empirical evaluations with transformers in recent works @xcite do not demonstrate big advantages. therefore, we do not use bayesian query strategies in our experiments and adhere to the classical uncertainty-based query strategies. recently proposed alternatives to uncertaintybased query strategies leverage reinforcement learning and imitation learning @xcite @xcite . this series of works aims at constructing trainable policy-based query strategies. however, this requires an excessive amount of computation while the transferability of learned policies across domains and tasks is underresearched. finally, @xcite question the usefulness of al techniques in general. they demonstrate that due to the asm problem, al can be even detrimental to the performance of the successor. this finding is also revealed for classical machine learning models by @xcite , @xcite , @xcite and supported by experiments with transformers in @xcite . our work directly addresses the question raised by @xcite and suggests a simple solution to the asm problem. moreover, we combine it with the method proposed by @xcite , who suggest using distilled models for instance acquisition and their teacher models as successors. ## background this section describes models and al query strategies used in this work. ## proposed methods this section outlines two proposed algorithms that help to reduce the computational cost of al. ## conclusion we investigated several obstacles to deploying al in practice and proposed two algorithms that help to overcome them. in particular, we considered the acquisition-successor mismatch problem revealed by @xcite , as well as the problem related to the excessive duration of al iterations with uncertainty-based query strategies and deep learning models. we demonstrate that the proposed plasm algorithm helps to deal with both of these issues: it removes the constraint on the type of the successor model trained on the data labeled with al and allows the use of lightweight acquisition models that have good training and inference performance, as well as a small memory footprint. the unlabeled pool subsampling algorithm helps to substantially decrease the inference time during al without a loss in the quality of successor models. together the plasm and ups algorithms help reduce the duration of an al iteration by more than 60%. we consider that the conducted empirical investigations and the proposed methods will help to increase the practicality of using deep al in interactive annotation tools. we note that applying plasm requires some conditions to be met. particularly, when a pseudolabeling model is of considerably lower performance than a successor model, and filtering is not strict enough, training a successor directly on labeled instances acquired during al with a different acquisition model may result in higher performance. consequently, despite the pseudo-labeling model may be less expressive compared to the successor model, it should not be too "weak". in practice, we suggest comparing results obtained by the models trained with pseudo-labeling and without on a hold-out set and selecting the best model. there are still many issues that hinder the application of al techniques. we consider that one of the most important obstacles is the necessity of hyperparameter optimization of deep learning models that can take a prohibitively long time to keep the annotation process interactive. we are looking forward to addressing this problem in future work.