Active Learning for Sequence Tagging with Deep Pre-trained Models and B ayesian Uncertainty Estimates (2021)

Abstract: Annotating training data for sequence tagging of texts is usually very time-consuming. Recent advances in transfer learning for natural language processing in conjunction with active learning open the possibility to significantly reduce the necessary annotation budget. We are the first to thoroughly investigate this powerful combination for the sequence tagging task. We conduct an extensive empirical study of various Bayesian uncertainty estimation methods and Monte Carlo dropout options for deep pre-trained models in the active learning framework and find the best combinations for different types of models. Besides, we also demonstrate that to acquire instances during active learning, a full-size Transformer can be substituted with a distilled version, which yields better computational performance and reduces obstacles for applying deep active learning in practice.

## introduction in many natural language processing (nlp) tasks, such as named entity recognition (ner), obtaining gold standard labels for constructing the training dataset can be very time and labor-consuming. it makes the annotation process expensive and limits the application of supervised machine learning models. this is especially the case in such domains as biomedical or scientific text processing, where crowdsourcing is either difficult or prohibitively expensive. in these domains, highly-qualified experts are needed to annotate data correctly, which dramatically increases the annotation cost. active learning (al) is a technique that can help to reduce the amount of annotation required to train a good model by multiple times @xcite . opposite to exhaustive and redundant manual annotation of the entire corpus, al drives the annotation process to focus the expensive human expert time only on the most informative objects, which contributes to a substantial increase in the model quality. al is an iterative process that starts from a small number of labeled seeding instances. in each iteration, an acquisition model is trained on the currently annotated dataset and is applied to the large pool of unannotated objects. the model predictions are used by the al query strategy to sample the informative objects, which are then further demonstrated to the expert annotators. when the annotators provide labels for these objects, the next iteration begins. the collected data can be used for training a final successor model that is used in a target application. during al, acquisition models have to be trained on very small amounts of the labeled data, especially during the early iterations. recently, this problem has been tackled by transfer learning with deep pre-trained models: elmo @xcite , bert @xcite , elec-tra @xcite , and others. pre-trained on a large amount of unlabeled data, they are capable of demonstrating remarkable performance when only hundreds or even dozens of labeled training instances are available. this trait suits the al framework but poses the question about the usefulness of the biased sampling provided by the al query strategies. in this work, we investigate al with the aforementioned deep pre-trained models and compare the results of this combination to the outcome of the models that do not take advantage of deep pretraining. the main contributions of this paper are the following: the remainder paper is structured as follows. section 2 covers relevant works on al for sequence tagging. in section 3, we describe the sequence tagging models. section 4 describes the al strategies used in the experiments. in section 5, we discuss the experimental setup and present the evaluation results. finally, section 6 concludes the paper. ## related work al for sequence tagging with classical machine learning algorithms and a feature-engineering approach has a long research history, e.g. @xcite @xcite . more recently, al in conjunction with deep learning has received much attention. in one of the first works on this topic, @xcite note that practical deep learning models that can be used in al should be computational efficient both for training and inference to reduce the delays in the annotators' work. they propose a cnn-cnn-lstm architecture with convolutional character and word encoders and an lstm tag decoder, which is a faster alternative to the widely adopted lstm-crf architecture @xcite with comparable quality. they also reveal disadvantages of the standard query strategy -least confident (lc), and propose a modification, namely maximum normalized log-probability (mnlp). siddhant and lipton (2018) experiment with bayesian uncertainty estimates. they use cnn-cnn-lstm and cnn-bilstm-crf @xcite networks and two methods for calculating the uncertainty estimates: bayes-by-backprop @xcite and the mc dropout @xcite . the experiments show that the variation ratio @xcite has substantial improvements over mnlp. in contrast to them, we additionally experiment with the bayesian active learning by disagreement (bald) query strategy proposed by @xcite and perform a comparison with variation ratio. there is a series of works that tackle al with a trainable policy model that serves as a query strategy. for this purpose, imitation learning is used in @xcite al. (2018); vu et al. (2019); brantley et al. (2020) , while in @xcite , the authors use deep reinforcement learning. although the proposed solutions are shown to outperform other heuristic algorithms with comparably weak models (basic crf or bert without fine-tuning) in experiments with a small number of al iterations, they can be not very practical due to the high computational costs of collecting training data for policy models. other notable works on deep active learning include @xcite , which proposes an al algorithm based on a bootstrapping approach @xcite and @xcite , which concerns the problem of the mismatch between a model used to construct a training dataset via al (acquisition model) and a final model that is trained on it (successor model). deep pre-trained models are evaluated in the al setting for ner by @xcite . however, they perform the evaluation only on the specific biomedical datasets and do not consider the bayesian query strategies. @xcite conduct an empirical study of al with pre-trained bert but only on the text classification task. brantley et al. (2020) use pre-trained bert in experiments with ner, but they do not fine-tune it, which results in suboptimal performance. in this work, we try to fill the gap by evaluating deep pre-trained models: elmo and various transformers, in the al setting with practical query strategies, including bayesian, and on the widely-used benchmarks in this area. ## sequence tagging models we use a tagger based on the conditional random field model @xcite , two bilstm-crf taggers @xcite with different word representation models, and taggers based on state-of-the-art transformer models. ## active learning query strategies we experiment with four query strategies for the selection of training instances during al. random sampling is the simplest query strategy possible: we just randomly select instances from the unlabeled pool for annotation. in this case, there is no active learning at all. uncertainty sampling @xcite ) methods select instances according to some probabilistic criteria, which indicates how uncertain the model is about the label that was given to the instance. the baseline method is least confident (lc): the instances are sorted in the ascending order of probabilities of the most likely tag sequence. let y i be a tag of a token i that can take one class c out of c values, let x j be a representation of a token j in an input sequence of length n. then the lc score can be formulated as follows: this score favors longer sentences since long sentences usually have a lower probability. maximization of probability is equivalent to maximizing the sum of log-probabilities: to make lc less biased towards longer sentences, @xcite propose a normalization of the log-probability sum. they call the method maximum normalized log-probability (mnlp). the mnlp score can be expressed as follows: in our experiments, we use this normalized version of the uncertainty estimate since it has been shown to be slightly better than the classical lc @xcite , and it is commonly applied in other works on active learning for ner. following @xcite , we implement extensions for the transformer-based and bilstm-based sequence taggers applying the mc dropout technique. gal and ghahramani (2016a) showed that applying a dropout at the prediction time allows us to consider the model as a bayesian neural network and calculate theoretically-grounded approximations of uncertainty estimates by analyzing its multiple stochastic predictions. @xcite and @xcite we experiment with variation ratio (vr) @xcite : a fraction of models, which predictions differ from the majority: let p cm i be a probability of a tag c for a token i that is predicted by a m-th stochastic pass of a model with the mc dropout. then the bald score can be approximated according to the following expression @xcite : although bald can be considered similar to vr, it potentially can give better uncertainty estimates than vr since it leverages the whole probability distributions produced by the model. however, this method has not been tested in the previous works on active learning for sequence tagging. for transformer models, we have two variants of the monte carlo dropout: the mc dropout on the last layer before the classification layer (mc last), and on all layers (mc all). we note that calculating uncertainty estimates in the case of mc all requires multiple stochastic passes, and in each of them, we have to perform inference of the whole transformer model. however, if we replace the dropout only in the classifier, multiple recalculations are needed only for the classifier, while it is enough to perform the inference of the massive transformer "body" only once. therefore, in this case, the overhead of calculating the bayesian uncertainty estimates can be less than 1% (in the case of 10 stochastic passes for electra according to the number of parameters) compared to deterministic strategies like mnlp. the bilstm-crf model has two types of dropout: the word dropout that randomly drops entire words after the embedding layer and the locked dropout @xcite ) that drops the same neurons in the embedding space of a recurrent layer for a whole sequence. therefore, for the bilstm-crf taggers, we have three options: replacing the locked dropout (mc locked), replacing the word dropout (mc word), and replacing both of them (mc all). we should note that obtaining bayesian uncertainty estimates does not require the recalculation of word embeddings (in our case, elmo). ## conclusion in this work, we investigated the combination of al with sequence taggers that take advantage of deep pre-trained models. in the al setting, these sequence taggers substantially outperform the models that do not use deep pre-training. we show that al and transfer learning is a very powerful combination that can help to produce remarkably performing models with just a small fraction of the annotated data. for the conll-2003 corpus, the combination of the best performing pre-trained model and al strategy achieves 99% of the score that can be obtained with training on the full corpus, while using only 20% of the annotated data. for the ontonotes corpus, one needs just 16%. we performed a large empirical study of al query strategies based on the monte carlo dropout in conjunction with deep pre-trained models and are the first to apply bayesian active learning by disagreement to sequence tagging tasks. bayesian active learning by disagreement achieves better results than the variation ratio for transformers. however, we find that the variation ratio is slightly better for the elmo-bilstm-crf model. it is reasonable to use both mc dropout-based query strategies when only the last dropout layer works in a stochastic mode during the inference. this makes this type of query strategies suitable for practical usage due to little computational overhead. finally, we demonstrate that it is possible to reduce the computational overhead of al with deep pre-trained models by using a smaller distilled version of a transformer model for acquiring instances. in the future work, we are seeking to extend the empirical investigation of deep pre-trained models in active learning to query strategies that aim at better handling the batch selection of instances for annotation.