On the Fragility of Active Learners for Text Classification (2024)

Abstract: Active learning (AL) techniques optimally utilize a labeling budget by iteratively selecting instances that are most valuable for learning. However, they lack “prerequisite checks”, i.e., there are no prescribed criteria to pick an AL algorithm best suited for a dataset. A practitioner must pick a technique they trust would beat random sampling, based on prior reported results, and hope that it is resilient to the many variables in their environment: dataset, labeling budget and prediction pipelines. The important questions then are: how often on average, do we expect any AL technique to reliably beat the computationally cheap and easy-to-implement strategy of random sampling? Does it at least make sense to use AL in an “Always ON” mode in a prediction pipeline, so that while it might not always help, it never under-performs random sampling? How much of a role does the prediction pipeline play in AL’s success?We examine these questions in detail for the task of text classification using pre-trained representations, which are ubiquitous today.Our primary contribution here is a rigorous evaluation of AL techniques, old and new, across setups that vary wrt datasets, text representations and classifiers. This unlocks multiple insights around warm-up times, i.e., number of labels before gains from AL are seen, viability of an “Always ON” mode and the relative significance of different factors.Additionally, we release a framework for rigorous benchmarking of AL techniques for text classification.

## introduction within a supervised learning setup, active learning (al) techniques @xcite use a query strategy (qs) to identify an unlabeled set of instances which is optimal in the following sense: if labelled and added to the training data, they lead to the greatest improvement in model accuracy, relative to any other same-sized set. in cases where labelling is expensive, the value proposition of al is that it is cost-efficient compared to random sampling, and a model reaches greater accuracy with a smaller number of labelled instances. in practice, an al technique is selected based on the strength of prior reported results, i.e., there are no "prerequisite checks": tests that one might perform on an unlabeled dataset, that help to select a technique suited to a problem foot_0 . this trust extends to related decisions such as batch and seed sizes, as well as the hyperparameters (if any) of the al technique since there is no way to empirically pick them: to compare with random sampling, or among techniques, labels are required. but if one had labels, they wouldn't need al @xcite ! in this sense, the al setup is unforgiving as one needs to make the optimal choice in one shot @xcite . this leads us to ask multiple questions about the broader area. how valid is the implicit but consequential assumption of transferability? a related question is whether the focus on qses alone is warranted -how much do the other components of a prediction pipeline affect outcomes? and finally, does it make sense to use al at least in an "always on" mode in a data labeling workflow; this is akin to asking if al might perform worse than random sampling. we need to quantify both the frequency and magnitude of gains from al, to be able to evaluate the cost of such pipelines. this is because even simple al techniques require a model to be evaluated over the unlabeled data pool, which can be expensive depending on the model complexity, size of the data pool and the latency allowed per al iteration. to be clear, we don't question if al results are reproducible within the original setups they were reported in 2 ; but whether any of those gains carry forward to new setups, which is how al is used in practice. we pick the area of text classification to investigate these concerns. the larger area of nlp has seen a rapid infusion of novel ideas of late. today, a practitioner has easy access to a variety of powerful classifiers via packages such as scikit-learn @xcite , spacy @xcite and hugging face @xcite , and text representations, such as universal sentence encoding (use) @xcite , minilm @xcite and mpnet @xcite . this makes it a fertile ground for testing al's utility. in all this, our motivation is not to disapprove of al as an area for research, but to motivate the inclusion of multiple practical challenges in future studies. contributions: our primary contribution is a rigorous empirical analysis of the learning behavior of al techniques over multiple text classification pipelines, that is targeted towards answering the questions asked above. additionally, we open source an al evaluation framework 3 , to enable researchers to not only reproduce our analysis, but also to rigorously evaluate their own contributions. ## previous work critique of al is not new. @xcite criticize al for its unpredictable (for a task) warm-up times, i.e., a minimum number of labeled instances before which gains over random sampling are evident. @xcite point out problems with al simulations. @xcite identify key issues leading to a lack of realistic al evaluations and propose solutions that they apply to image classification. @xcite study al empirically but focus on the interesting notion of successor models, i.e., future models that would use the labeled data collected via al using a specific model. @xcite examine the empirical effectiveness of al, but they don't evaluate on nlp tasks. @xcite is an empirical study of al effectiveness similar in spirit to ours, but they focus on deep bayesian methods. @xcite study sampling biases in deep al, but their study is limited to one prediction model -fasttext.zip @xcite duciblity tests for the al techniques we benchmark here, and were able to replicate reported results -see §d. ## batch active learning -overview in this work, we specifically study the batch al setting for text classification. here, a qs identifies a batch of b unlabeled points, at each iteration t, for t iterations. a model m t , that is trained on the accumulated labeled pool, is produced at the end of each iteration. the first iteration uses a seed set of s randomly sampled points (although other strategies may be used). we note that that m t should be produced using a model selection strategy (we use a hold-out set here), and must also be calibrated (we use platt scaling @xcite ). the former ensures that m t doesn't overfit to the labeled data, which is likely in the initial iterations due to small quantities. the latter is required since many query strategies rely on uncertainty/confidence scores produced by m t . unfortunately, in our experience, multiple implementations/studies miss one or both of these steps. to avoid any ambiguity, we provide pseudo-code for this al setting in algorithm 1 in §a. ## experiment setup in this section, we describe our experiment setup in detail. ## results we are now ready to look at the results of our experiments. ## summary and conclusion after extensive evaluation of different al algorithms, we are forced to conclude that it is difficult to practically benefit from al. gains from qses are inconsistent across datasets, prediction pipelines and text representations. in fact, between qses and prediction pipelines, the latter seems to have a greater influence on the relative improvement over random ( §5.3). the only general pattern we see is that positive relative improvements become likely as labeled instances accumulate; but these improvements are too small to be broadly useful ( §5.1). another reason as to why it is hard to derive any practical advice is that we lack the tools, theoretical or empirical, to identify a settings-specific warm-start size; when do we stop labeling to realize gains, however small? further, we noted in §5.2 that using al in an "always on" mode can actually perform worse than random sampling. the use of roberta as the prediction pipeline is the only (isolated) case where we see consistent positive relative improvements. our hypothesis as to why is that an end-to-end classifier has a more coherent view of the overall distribution, and therefore informativeness of a sample. but, obviously, we can't discount the role that roberta's specific pre-training might play here, and further experimentation is required to disentangle their respective influences. even in this case, we point out that (1) it provides further evidence for the argument that the qs alone does not decide outcomes, and (2) while positive, the gains aren't considerable, with δ ≈ 1% (see figure 3 (a)-(f) ). although extensive, this study may be considered "limited" relative to real-world variances, e.g., many more choices of classifiers, datasets, which leads us to suspect that the true picture is probably more dismal. what might we do to make the field of al more useful? we feel the biggest problem in al use is that practitioners have to blindly guess what specific al technique will work best for their problem. as a field we need to embrace a broader discourse where the success of a technique needs to be tied to fundamental properties of datasets, e.g., topological features @xcite , and predictors, e.g., vc dimension @xcite , that are identifiable in an unsupervised manner in novel settings.