A c T une: Uncertainty-Based Active Self-Training for Active Fine-Tuning of Pretrained Language Models (2022)

Abstract: Although fine-tuning pre-trained language models (PLMs) renders strong performance in many NLP tasks, it relies on excessive labeled data. Recently, researchers have resorted to active fine-tuning for enhancing the label efficiency of PLM fine-tuning, but existing methods of this type usually ignore the potential of unlabeled data. We develop AcTune, a new framework that improves the label efficiency of active PLM fine-tuning by unleashing the power of unlabeled data via self-training. AcTune switches between data annotation and model self-training based on uncertainty: the unlabeled samples of high-uncertainty are selected for annotation, while the ones from low-uncertainty regions are used for model self-training. Additionally, we design (1) a region-aware sampling strategy to avoid redundant samples when querying annotations and (2) a momentum-based memory bank to dynamically aggregate the model’s pseudo labels to suppress label noise in self-training. Experiments on 6 text classification datasets show that AcTune outperforms the strongest active learning and self-training baselines and improves the label efficiency of PLM fine-tuning by 56.2% on average. Our implementation is available at https://github.com/yueyu1030/actune .

## introduction fine-tuning pre-trained language models (plms) has achieved much success in natural language processing (nlp) @xcite @xcite . one benefit of plm fine-tuning is the promising performance it offers when consuming only a few labeled data @xcite . however, there are still significant gaps between few-shot and fully-supervised plm fine-tuning in many tasks. besides, the performance of few-shot plm finetuning can be sensitive to different sets of training data @xcite . therefore, there is a crucial need for approaches that make plm finetuning more label-efficient and robust to selection of training data, especially for applications where labeled data are scarce and expensive to obtain. towards this goal, researchers have recently resorted to active fine-tuning of plms and achieved comparable performance to fully-supervised methods with much less annotated samples @xcite @xcite . nevertheless, they usually neglect unlabeled data, which can be useful for improving label efficiency for plm fine-tuning @xcite . to incorporate unlabeled data into active learning, efforts have been made in the semi-supervised active learning literature @xcite @xcite . however, the query strategies proposed in these works can return highly redundant samples due to limited representation power, resulting in suboptimal label efficiency. moreover, they usually rely on pseudo-labeling to utilize unlabeled data, which requires greater (yet often absent) care to denoise the pseudo labels, otherwise the errors could accumulate and hurt model performance. this issue can be even more severe for plms, as the fine-tuning process is often sensitive to different weight initialization and data orderings @xcite . thus, it still remains open and challenging to design robust and label efficient method for active plm fine-tuning. to tackle the above challenges, we propose ac-tune, a new method that improves the label efficiency and robustness of active plm fine-tuning. based on the estimated model uncertainty, ac-tune tightly couples active learning with selftraining in each learning round: (1) when the average uncertainty of a region is low, we trust the model's predictions and select its most certain predictions within the region for self-training; (2) when the average uncertainty of a region is high, indicating inadequate observations for parameter learning, we actively annotate its most uncertain samples within the region to improve model performance. different from existing al methods that only leverage uncertainty for querying labels, our uncertainty-driven self-training paradigm gradually leverages unlabeled data with low uncertainty via self-training, while reducing the chance of error propagation triggered by highly-uncertain mislabeled data. to further boost model performance for ac-tune, we design two techniques to improve the query strategy and suppress label noise, namely region-aware sampling (rs) and momentum-based memory bank (mmb). inspired by the fact that existing uncertainty-based al methods often end up with choosing uncertain yet repetitive data @xcite , we design the region-aware sampling technique to promote both diversity and representativeness by leveraging the representation power of plms. specifically, we first estimate the uncertainties of the unlabeled data with plms, then cluster the data using their plm representations and weigh the data by the corresponding uncertainty. such a clustering scheme partitions the embedding space into small sub-regions with an emphasis on highly-uncertain samples. finally, by sampling over multiple highuncertainty regions, our strategy selects data with high uncertainty and low redundancy. to rectify the erroneous pseudo labels derived by self-training, we design a simple but effective way to select low-uncertainty data for selftraining. our method is motivated by the fact that fine-tuning plms suffer from instability issuesdifferent initializations and data orders can lead to large variance in model performance @xcite @xcite . however, previous approaches only select pseudo-labeled data based on the prediction of the current round and are thus less reliable. in contrast, we maintain a dynamic memory bank to save the predictions of unlabeled samples for later use. we propose a momentum updating method to dynamically aggregate the predictions from preceding rounds @xcite and select lowuncertainty samples based on aggregated prediction. as a result, only the samples with high prediction confidence over multiple rounds will be used for self-training, which mitigates the issue of label noise. we highlight that our active self-training approach is an efficient substitution to existing al methods, requiring little extra computational cost. our key contributions are: (1) an active selftraining paradigm actune that integrates selftraining and active learning to minimize the labeling cost for fine-tuning plms; (2) a region-aware querying strategy to enforce both the informativeness and the diversity of queried samples during al; (3) a simple and effective momentum-based method to leverage the predictions in preceding rounds to alleviate the label noise in self-training and ( 4 ) experiments on 6 benchmarks demonstrating actune improves the label efficiency over existing self-training and active learning baselines by 56.2%. ## related work active learning. active learning has been widely applied to various nlp tasks @xcite @xcite . so far, al methods can be categorized into uncertainty-based methods @xcite , diversity-based methods @xcite and hybrid methods @xcite . @xcite offer an empirical study of active learning with plms. very recently, there are also several works attempted to query labeling functions for weakly-supervised learning @xcite @xcite . in our study, we leverage the power of unlabeled instances via self-training to further promote the performance of al. 2020 ) exploit the most-certain samples from the unlabeled with pseudo-labeling to augment the training set. so far, most of the ssal approaches are designed for cv domain and it remains unknown how this paradigm performs with plms on nlp tasks. in contrast, we propose actune to effectively leverage unlabeled data during finetuing plms for nlp tasks. self-training. self-training is one of the earliest and simplest approaches to semi-supervised learning @xcite . it first generates pseudo labels for high-confidence samples, then fits a new model on pseudo labeled data to improve the generalization ability. however, it is known to be vulnerable to error propagation @xcite @xcite . to alleviate this, we adopt a simple momentum-based method to select high confidence samples, effectively reducing 1429 the pseudo labels noise for active learning. note that although mukherjee and awadallah (2020); rizve et al. (2021) also leverage uncertainty information for self-training, their focus is on developing better self-training methods, while we aim to jointly query high-uncertainty samples and generate pseudo-labels for low-uncertainty samples. the experiments in sec. 3 show that with appropriate querying methods, actune can further improve the performance of self-training. ## conclusion and discussion in this paper, we develop actune, a general active self-training framework for enhancing both label efficiency and model performance in fine-tuning pre-trained language models (plms). we propose a region-aware sampling approach to guarantee both the uncertainty the diversity for querying labels. to combat the label noise propagation issue, we design a momentum-based memory bank to effectively utilize the model predictions for preceding al rounds. empirical results on 6 public text classification benchmarks suggest the superiority of actune to conventional active learning and semi-supervised active learning methods for fine-tuning plms with limited resources. there are several directions to improve actune. first, since our focus is on fine-tuning pre-trained language models, we use the representation of [cls] token for classification. in the future work, we can consider using prompt tuning @xcite , a more dataefficient method for adopting pre-trained language models on classification tasks to further promote the efficiency. also, due to the computational resource constraints, we do not use larger pre-trained language models such as roberta-large @xcite which shown even better performance with only a few labels @xcite . moreover, we can explore more advanced uncertainty estimation approach @xcite into actune to further improve the performance. last, apart from the text classification task, we can also extend our work into other tasks such as sequence labeling and natural language inference (nli). <0.1s <0.1s entropy @xcite 461s 646s bald @xcite 4595s 6451s alps @xcite 488s 677s badge @xcite 554s 1140s cal @xcite 493s 688s revival @xcite ) 3240s oom actune + entropy 477s 733s w/ rs for active learning 15.8s 44.9s w/ mmb for self-training 0.12s 0.18s actune + cal 510s 735s w/ rs for active learning 16.6s 46.4s w/ mmb for self-training 0.12s 0.18s