Cold-start Active Learning through Self-supervised Language Modeling (2020)

Abstract: Active learning strives to reduce annotation costs by choosing the most critical examples to label. Typically, the active learning strategy is contingent on the classification model. For instance, uncertainty sampling depends on poorly calibrated model confidence scores. In the cold-start setting, active learning is impractical because of model instability and data scarcity. Fortunately, modern NLP provides an additional source of information: pre-trained language models. The pre-training loss can find examples that surprise the model and should be labeled for efficient fine-tuning. Therefore, we treat the language modeling loss as a proxy for classification uncertainty. With BERT, we develop a simple strategy based on the masked language modeling loss that minimizes labeling costs for text classification. Compared to other baselines, our approach reaches higher accuracy within less sampling iterations and computation time.

## introduction labeling data is a fundamental bottleneck in machine learning, especially for nlp, due to annotation cost and time. the goal of active learning (al) is to recognize the most relevant examples and then query labels from an oracle. for instance, policymakers and physicians want to quickly fine-tune a text classifier to understand emerging medical conditions @xcite . finding labeled data for medical text is challenging because of privacy issues or shortage in expertise @xcite . using al, they can query labels for a small subset of the most relevant documents and immediately train a robust model. modern transformer models dominate the leaderboards for several nlp tasks @xcite . yet the price of adopting * work done while visiting national taiwan university. another issue is that traditional al algorithms, like uncertainty sampling @xcite , falter on deep models. these strategies use model confidence scores, but neural networks are poorly calibrated @xcite . high confidence scores do not imply high correctness likelihood, so the sampled examples are not the most uncertain ones @xcite . plus, these strategies sample one document on each iteration. the singledocument sampling requires training the model after each query and increases the overall expense. these limitations of modern nlp models illustrate a twofold effect: they show a greater need for al and make al more difficult to deploy. ideally, al could be most useful during low-resource situations. in reality, it is impractical to use because the al strategy depends on warm-starting the model with information about the task @xcite . thus, a fitting solution to al for deep classifiers is a cold-start approach, one that does not rely on classification loss or confidence scores. to develop a cold-start al strategy, we should extract knowledge from pre-trained models like bert @xcite . the model encodes syntactic properties @xcite , acts as a database for general world knowledge @xcite , and can detect out-of-distribution examples @xcite . given the knowledge already encoded in pre-trained models, the annotation for a new task should focus on the information missing from pretraining. if a sentence contains many words that perplex the language model, then it is possibly unusual or not well-represented in the pre-training data. thus, the self-supervised objective serves as a surrogate for classification uncertainty. we develop alps (active learning by processing surprisal), an al strategy for bert-based models. 1 while many al methods randomly choose an initial sample, alps selects the first batch of data using the masked language modeling loss. as the highest and most extensive peaks in europe are found in the alps, the alps algorithm finds examples in the data that are both surprising and substantial. to the best of our knowledge, alps is the first al algorithm that only relies on a selfsupervised loss function. we evaluate our approach on four text classification datasets spanning across three different domains. alps outperforms al baselines in accuracy and algorithmic efficiency. the success of alps highlights the importance of self-supervision for cold-start al. ## preliminaries we formally introduce the setup, notation, and terminology that will be used throughout the paper. pre-trained encoder pre-training uses the language modeling loss to train encoder parameters for generalized representations. we call the model input @xmath0, which is a sequence of tokens w from a vocabulary v with sequence length l. given weights w , the encoder h maps x to a d-dimensonal hidden representation h(x; w ). we use bert @xcite as our data encoder, so h is pre-trained with two tasks: masked language modeling (mlm) and next sentence prediction. the embedding h(x; w ) is computed as the final hidden state of the [cls] token in x. we also refer to h(x; w ) as the bert embedding. ## the uncertainty-diversity dichotomy this section provides background on prior work in al. first, we discuss two general al strategies: uncertainty sampling and diversity sampling. then, we explain the dichotomy between the two concepts and introduce badge @xcite , a sota method that attempts to resolve this issue. finally, we focus on the limitations of badge and other al strategies to give motivation for our work. @xcite describes uncertainty and diversity as the "two faces of al". while uncertainty sampling efficiently searches the hypothesis space by finding difficult examples to label, diversity sampling exploits heterogeneity in the feature space @xcite @xcite . uncertainty sampling requires model warmstarting because it depends on model predictions, whereas diversity sampling can be a cold-start approach. a successful al strategy should integrate both aspects, but its exact implementation is an open research question. for example, a naïve idea is to use a fixed combination of strategies to sample points. nevertheless, @xcite experimentally show that this approach hampers accuracy. badge optimizes for both uncertainty and diversity by using confidence scores and clustering. this strategy beats uncertainty-based algorithms @xcite , sampling through bandit learning @xcite , and coreset @xcite , a diversity-based method for convolutional neural networks. ## a self-supervised active learner cold-start al is challenging because of the shortage in labeled data. prior work, like badge, often depend on model uncertainty or inference, but these measures can be unreliable if the model has not trained on enough data (section 3.2.1). to overcome the lack of supervision, what if we apply self-supervision to al? for nlp, the language modeling task is self-supervised because the label for each token is the token itself. if the task has immensely improved transfer learning, then it may reduce generalization error in al too. for our approach, we adopt the uncertaintydiversity badge framework for clustering embeddings that encode information about uncertainty. however, rather than relying on the classification loss gradient, we use the mlm loss to bootstrap uncertainty estimates. thus, we combine uncertainty and diversity sampling for cold-start al. ## active sentence classification we evaluate alps on sentence classification for three different domains: sentiment reviews, news articles, and medical abstracts ( ## related work active learning has shown success in tasks, such as named entity recognition @xcite , word sense disambiguation @xcite , and sentiment analysis @xcite . @xcite are the first to adapt prior al work to deep learning. however, popular heuristics @xcite for querying individual points do not work as well in a batch setting. since then, more research has been conducted on batch al for deep learning. @xcite propose the first work on al for neural text classification. they assume that the classifier is a convolutional neural network and use expected gradient length @xcite to choose sentences that contain words with the most label-discriminative embeddings. besides text classification, al has been applied to neural models for semantic parsing @xcite , named entity recognition @xcite , and machine translation @xcite . alps makes use of bert, a model that excels at transfer learning. other works also combine al and transfer learning to select training data that reduce generalization error. @xcite mea-sures domain divergence from the source domain to select the most informative texts in the target domain. @xcite use al to query points for a target task through matching conditional distributions. additionally, combining word-level and document-level annotations can improve knowledge transfer @xcite . in addition to uncertainty and diversity sampling, other areas of deep al focus on bayesian approaches @xcite and reinforcement learning @xcite . an interesting research direction can integrate one of these approaches with alps. ## conclusion transformers are powerful models that have revolutionized nlp. nevertheless, like other deep models, their accuracy and stability require fine-tuning on large amounts of data. al should level the playing field by directing limited annotations most effectively so that labels complement, rather than duplicate, unsupervised data. luckily, transformers have generalized knowledge about language that can help acquire data for fine-tuning. like badge, we project data into an embedding space and then select the most representative points. our method is unique because it only relies on self-supervision to conduct sampling. using the pre-trained loss guides the al process to sample diverse and uncertain examples in the cold-start setting. future work may focus on finding representations that encode the most important information for al.