On the Importance of Effectively Adapting Pretrained Language Models for Active Learning (2022)

Abstract: Recent active learning (AL) approaches in Natural Language Processing (NLP) proposed using off-the-shelf pretrained language models (LMs). In this paper, we argue that these LMs are not adapted effectively to the downstream task during AL and we explore ways to address this issue. We suggest to first adapt the pretrained LM to the target task by continuing training with all the available unlabeled data and then use it for AL. We also propose a simple yet effective fine-tuning method to ensure that the adapted LM is properly trained in both low and high resource scenarios during AL. Our experiments demonstrate that our approach provides substantial data efficiency improvements compared to the standard fine-tuning approach, suggesting that a poor training strategy can be catastrophic for AL.

## introduction active learning (al) is a method for training supervised models in a data-efficient way @xcite . it is especially useful in scenarios where a large pool of unlabeled data is available but only a limited annotation budget can be afforded; or where expert annotation is prohibitively expensive and time consuming. al methods iteratively alternate between (i) model training with the labeled data available; and (ii) data selection for annotation using a stopping criterion, e.g. until exhausting a fixed annotation budget or reaching a pre-defined performance on a held-out dataset. data selection is performed by an acquisition function that ranks unlabeled data points by some informativeness metric aiming to improve over random selection, using either uncertainty @xcite @xcite @xcite , diversity @xcite @xcite , or both @xcite @xcite . previous al approaches in nlp use taskspecific neural models that are trained from scratch at each iteration @xcite @xcite @xcite . however, these models are usually outperformed by pretrained language models (lms) adapted to end-tasks @xcite , making them suboptimal for al. only recently, pretrained lms such as bert @xcite have been introduced in al settings @xcite @xcite @xcite . still, they are trained at each al iteration with a standard fine-tuning approach that mainly includes a pre-defined number of training epochs, which has been demonstrated to be unstable, especially in small datasets @xcite @xcite . since al includes both low and high data resource settings, the al model training scheme should be robust in both scenarios. 2 to address these limitations, we introduce a suite of effective training strategies for al ( §2). contrary to previous work @xcite @xcite that also use bert @xcite , our proposed method accounts for various data availability settings and the instability of fine-tuning. first, we continue pretraining the lm with the available unlabeled data to adapt it to the task-specific domain. this way, we leverage not only the available labeled data at each al iteration, but the entire unlabeled pool. second, we further propose a simple yet effective fine-tuning method that is robust in both low and high resource data settings for al. we explore the effectiveness of our approach on five standard natural language understandings tasks with various acquisition functions, showing that it outperforms all baselines ( §3). we also conduct an analysis to demonstrate the importance of effective adaptation of pretrained models for al ( §4). our findings highlight that the lm adaptation strategy can be more critical than the actual data acquisition strategy. ## adapting & fine-tuning pretrained models for active learning given a downstream classification task with c classes, a typical al setup consists of a pool of unlabeled data d pool , a model m, an annotation budget b of data points and an acquisition function a(.) for selecting k unlabeled data points for annotation (i.e. acquisition size) until b runs out. the al performance is assessed by training a model on the actively acquired dataset and evaluating on a held-out test set d test . adaptation (tapt) inspired by recent work on transfer learning that shows improvements in downstream classification performance by continuing the pretraining of the lm with the task data @xcite we add an extra step to the al process by continuing pretraining the lm (i.e. task-adaptive pretraining tapt), as in @xcite . formally, we use an lm, such as bert @xcite , p(x; w 0 ) with weights w 0 , that has been already pretrained on a large corpus. we fine-tune p(x; w 0 ) with the available unlabeled data of the downstream task d pool , resulting in the task-adapted lm p tapt (x; w ′ 0 ) with new weights w ′ 0 (cf. line 2 of algorithm 1). ## experiments & results data we experiment with five diverse natural language understanding tasks: question classification ## conclusion we have presented a simple yet effective training scheme for al with pretrained lms that accounts for varying data availability and instability of finetuning. specifically, we propose to first continue pretraining the lm with the available unlabeled data to adapt it to the task-specific domain. this way, we leverage not only the available labeled data at each al iteration, but the entire unlabeled pool. we further propose a method to fine-tune the model during al iterations so that training is robust in both low and high resource data settings. our experiments show that our approach yields substantially better results than standard fine-tuning in five standard nlp datasets. furthermore, we find that the training strategy can be more important than the acquisition strategy. in other words, a poor training strategy can be a crucial impediment to the effectiveness of a good acquisition function, and thus limit its effectiveness (even over random sampling). hence, our work highlights how critical it is to properly adapt a pretrained lm to the low data resource al setting. as state-of-the-art models in nlp advance rapidly, in the future we would be interested in exploring the use of larger lms, such as gpt-3 @xcite and flan @xcite . these models have achieved impressive performance in very low data resource settings (e.g. zero-shot and few-shot), so we would imagine they would be good candidates for the challenging setting of active learning.