A ctive L earning for BERT : A n E mpirical S tudy (2020)

Abstract: Real world scenarios present a challenge for text classification, since labels are usually expensive and the data is often characterized by class imbalance. Active Learning (AL) is a ubiquitous paradigm to cope with data scarcity. Recently, pre-trained NLP models, and BERT in particular, are receiving massive attention due to their outstanding performance in various NLP tasks. However, the use of AL with deep pre-trained models has so far received little consideration. Here, we present a large-scale empirical study on active learning techniques for BERT-based classification, addressing a diverse set of AL strategies and datasets. We focus on practical scenarios of binary text classification, where the annotation budget is very small, and the data is often skewed. Our results demonstrate that AL can boost BERT performance, especially in the most realistic scenario in which the initial set of labeled examples is created using keyword-based queries, resulting in a biased sample of the minority class. We release our research framework, aiming to facilitate future research along the lines explored here.

## introduction automatic text classification is a well studied problem in natural language processing (nlp), with great practical importance and numerous real world applications @xcite . there are two major hurdles to developing effective text classifiers in practice, as well as to developing classifiers in other domains -the lack of labeled data, and class imbalance @xcite . text classifiers often require high quantities of labeled data for model training. however, collecting such labeled data is a notoriously expensive and time-consuming process, and shortage of labeled data is exacerbated when the desired class has a * these authors equally contributed to this work. a classical approach for coping with limited annotation resources is active learning (al) @xcite . in this paradigm, one assumes that unlabeled data are abundant, and the goal is to focus the expensive labeling process on the most informative instances. many al strategies have been proposed, aiming to minimize the labeling burden, or if taken from a different perspective -maximize the value of labeling a small set of examples. importantly, the usefulness of an al strategy naturally depends on the classification scheme with which it is coupled. a successful al approach for a naive bayes classifier may not be that effective for a modern deep-learning algorithm such as cnn, and vice versa. a more recent relevant development is the introduction of pre-trained nlp models (cf. @xcite , which have been shown to substantially improve state-of-the-art results in numerous nlp tasks. a prominent example is the bert model @xcite , which has received massive attention from the nlp research community since its inception. however, the use of al with deep pre-trained models for text classification -and bert in particular -has so far received surprisingly little consideration. thus, while recent papers have demonstrated the value of al for various deep-learning text classification schemes @xcite @xcite , the potential of al combined with bert is yet to be explored. first, given the unique properties of pre-trained models, and the expectation that such models will yield adequate performance even with small amounts of training data, it is unclear a priori whether -and to what extent -established al paradigms can further enhance their classification performance. moreover, more recent deep al strategies, such as core-set @xcite and dropout @xcite , were developed in the vision domain for cnns. the value of these strategies on top of the bert transformer architecture remains unclear. our goal in this work is threefold. we study the potential of (i) various al strategies; (ii) in conjunction with bert, an arguably outstanding text classification scheme; (iii) within a highly challenging -yet common -real-world scenario of class imbalance and scarce labeled data. to address this goal, we conduct a systematic study, considering traditional and advanced al strategies coupled with bert for a wide range of datasets. we focus on three scenarios: a balanced setting, serving as a reference, where the prior of the class of interest is not too small; the more challenging imbalanced setting, where the class prior @xmath0; and finally, the imbalanced-practical setting, which is similar to the imbalanced one, but takes a step further towards a truly practical setup, in which there is no access to an unbiased positive sample. instead, we assume the user has access to a biased sample, hopefully enriched with positive examples, obtained by issuing simple queries of keywords associated with the positive class. our results convey that al strategies can boost bert performance, under the challenging setting of a small annotation budget and highly skewed data, especially in the more practical real-world settings. we release our research framework foot_0 , including access to all datasets, an implementation of multiple al strategies, and an associated automatic evaluation framework, aiming to facilitate further research along the lines explored here. ## related work al has been widely used in many fields to successfully decrease the labeling effort involved in the training process. a good summary of active learning works prior to the advances in deep learn-ing can be found in @xcite . advances in deep learning have given rise to extensive research into deep active learning, which aims to adapt the classic al framework to the special properties of dnns. deep al presents some specific challenges. since dnns are computationally heavy, training a new model whenever a single training sample is added is highly impractical. this requires a shift to batch mode active learning, where a batch of examples is queried at every iteration. moreover, the tendency of the softmax layer to over-confidence has led to the development of various uncertaintybased strategies tailored to the special properties of dnns @xcite . most of the works in deep active learning focus on image classification with convolutional neural networks @xcite @xcite . recent papers have demonstrated the value of deep active learning for text classification @xcite @xcite , but in general did not study al for bert. one exception is @xcite who applied an ensemble of al strategies to bert for the task of intent classification. however, this work focuses on a single task, and does not address the effect of small and imbalanced data. additionally, @xcite and @xcite focused on particular variants of bert (biobert and bert-crf) and studied a single or two specific tasks, with a small collection of al strategies. to the best of our knowledge, this work is the first to systematically explore advanced strategies like core-set @xcite , dropout @xcite , expected gradient length @xcite and discriminative active learning @xcite for bert, in various settings and a diversity of tasks. ## results we report results for the al strategies ( §3.3) in three experimental scenarios ( §3.2). following the standard in the field, we use accuracy as the classification metric for the balanced scenario, and f1 6 for the imbalanced and imbalanced-practical scenarios, where the prior for positives is relatively low. figure 1 depicts the classification quality (accuracy or f1) per iteration for each dataset, for the relevant scenarios. for clarity of presentation, we only plot the random baseline and three strategies that represent the different approaches. as can be seen in the full figure in the appendix (figure 3 ), the other strategies behave similarly. in most datasets, all al strategies performed better than the random baseline, even in cases where the baseline results were already very good, e.g., ag's news and subjectivity. interestingly, the largest improvements were observed in the imbalanced-practical scenario. here, the al strategies improve the f1 of the random baseline by a large margin of 4 -8% on average. these results demonstrate that al can indeed enhance bert results when the annotation budget is small, especially for datasets having a low prior for positive examples, as is the case in many real-world settings. to check the significance of the differences, we calculate the wilcoxon p-value 7 for every al strategy compared to the random baseline, per scenario, and perform a bonferroni correction to adjust for the multiple strategies examined. to calculate the p-value for a strategy s per scenario, we compare the classification metric for all pairs (s dik , r dik ) such that r is the random baseline results, @xmath1, where d is the set of datasets in-balanced imbalanced imbalanced-practical cluded in the scenario, @xmath2...5) is the iteration index, and @xmath3...5) is the experiment number. as can be seen in while al strategies improve over the random baseline, apparently no single strategy consistently outperforms all its counterparts. this finding echoes @xcite , who studied al for text classification and sequence tagging in non-bert models, and demonstrated the brittleness and inconsistency of al results. for significance analysis, we calculate the p-value for every pair of strategies per scenario in a similar manner to the one described above per strategy versus random, correcting for the multiple pairs examined, and indeed find no overall significant performance difference between any pair of al strategies. note, however, that some al strategies are more efficient than others with respect to runtime -see as may be expected, using a seed with positive labels obtained by a query, which is naturally biased towards instances that satisfy the query, typically results in an initial model with lower f1, compared to starting with an unbiased set of positive examples (compare iteration 0 per dataset in figure 1 between the imbalanced-practical and imbalanced scenarios). interestingly, though, after several iterations, the al strategies seem to bridge the gap and end up with similar classification performance in both scenarios. we further examined whether the increase in f1 for the imbalanced scenarios is driven by an increase in precision or recall. we find that in the imbalanced-practical scenario, the improvement in f1 is completely dominated by an increase in recall, supporting the notion that the al strategies enable the model to extrapolate and generalize beyond the biased sample obtained by the query 8 . in contrast, in the imbalanced scenario the increase in f1 is mostly driven by an increase in precision. for the precision and recall curves, see figures 4 and 5 in the appendix. to conclude, our analyses suggest two results that were not trivial to begin with. applying al to bert can further boost the performance of this top performing model. furthermore, even when 8 the low recall of the queries can be seen in ## analysis we perform a comparative analysis of the different al strategies, aiming to better understand their relative advantages and disadvantages, and provide some insights that may lead to improved al strategies in future work. to enable an appropriate comparison, this analysis is performed after the initial bert model is trained and each al strategy has selected 50 examples for labeling. correspondingly, all strategies select examples from the same unlabeled set u while using outputs from the same bert model. we measure two batch properties which are known in the literature to impact al effectiveness: diversity: choosing a batch of diverse examples is often better than choosing one containing very similar and perhaps redundant examples. following @xcite , we define the diversity of a set b as: representativeness: a known issue with al strategies, especially the uncertainty-based ones, is their tendency to select outlier examples that do not properly represent the overall data distribution. we thus examine the representativeness of the selected batches. we rely on the knn-density measure proposed by @xcite , in which the density of an example is quantified by the average distance between the example in question and its k most similar examples (i.e., k nearest neighbors) within u , based on the [cls] representations as above. an example with high density degree is less likely to be an outlier. we define the representativeness of a batch as one over the average knn-density of its instances using the euclidean distance with @xmath4. the diversity and representativeness of the different strategies are depicted in figure 2 , where for each scenario we average results over all datasets and seed selections. as expected, the batch-aware strategies, dal and core-set, which were designed to increase diversity, are characterized by the most diverse batches, with dal achieving the highest diversity values, demonstrating the success of using mini-queries @xcite to reduce redundancy of the selected examples. in contrast, the other strategies tend to select less diverse batches, i.e., they are prone to choose redundant examples, especially in the imbalanced-practical scenario. thus, combining these approaches with methods that encourage diversity (e.g., @xcite @xcite can potentially lead to further improvement in their resultant prediction performance. in terms of representativeness, dal, which is a representativeness-driven method, again consistently leads across the scenarios. in contrast, the tendency of the greedy core-set version to select outliers @xcite , is indeed reflected in its relatively low representativeness scores. interestingly, this is not the case for the imbalanced-practical scenario. this result can be attributed to the high bias of l towards query matches, which results in poor representativeness of l, which in turn leaves the main "responsibility" for representing the dataset on the batch examples selected from u. a deeper investigation of this result is left for further investigation. other strategies have low representativeness scores compared to dal, implying that they can be improved by combining them with techniques for encouraging representativeness and avoiding outliers. a popular approach for improving classification quality is combining several, preferably complementary, al strategies. in order to find pairs of strategies with high synergistic potential, we measured the overlap between the batches selected by each pair of strategies. our analysis shows that for all pairs of strategies, the expected batch overlap is relatively low, and does not exceed 15%. in general, overlap was higher in the imbalanced scenarios, probably due to the general incentive to select positive examples, which are rare in the data. also, the overlap within the uncertainty-based strategies was generally quite high. nevertheless, the highest overlap between batches was between egl (which is not an uncertainty-based approach) and lc. we leave for future work to try a combination of strategies with low overlap as a way to improve classification even further. ## conclusions the recent emergence of pre-trained models, with bert as a prominent example, is reshaping the nlp arena @xcite . the promise embodied in these models is their ability to exploit massive unlabeled textual data to learn versatile, arguably universal language representations. these representations, in turn, are proven to be effective for a multitude of downstream nlp tasks. a parallel line of research, dating back nearly three decades, is the notion of al, aiming to minimize labeling burden within the supervised learning paradigm. the pairing of these two influential threads raises non-trivial questions. for example, bert arguably attains excellent performance with relatively little labeled data, used to fine-tune this pre-trained model for a concrete task. it is not obvious to begin with, to what extent al strategies can be used to outperform this already high bar. to the best of our knowledge, the present work provides the first systematic study in this context, while focusing on the prevalent problem of text classification. moreover, we further focus our attention on a scenario well known to many practitioners -and notoriously difficult from a learning perspective -that of building a classifier when the class of interest is scarce in the data at hand. aiming to further bridge the gap between research and practice, in our imbalanced-practical mode we simulate a user within this challenging scenario, armed only with simple queries to define the labeled seed that will bootstrap the al process. our results demonstrate the potential of al on top of bert, especially in this latter scenario. notably, a training data seed resulting from a simple query is expected to capture only limited, and perhaps somewhat obvious, aspects of the class under consideration. our study shows that the initial bert model indeed suffers from poor prediction performance, mainly due to low recall values. however, while the random al baseline is limited in its ability to help bert emerge from this poor initial model, al strategies turn out to be very helpful. using the al pipeline, bert improves its recall by a large margin, generalizing beyond the narrow data it was initially exposed to. this work focused on various binary classification tasks. a natural future direction is to conduct a similar empirical investigation of al over bert in the context of multi-class classification and regression tasks. it would also be interesting to investigate the realm of larger annotation budgets, and more recent bert variants @xcite . finally, the present work focused on existing al strategies, which were mostly developed in the vision domain for cnns. the development of novel al methods, that are tailored for pre-trained models such as bert, seems like an important direction for future work. we hope that the experimental results and analyses reported here, as well as the release of the research framework we developed, would be instrumental for these and other future studies.