Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysis (2023)

Abstract: Developed to alleviate prohibitive labeling costs, active learning (AL) methods aim to reduce label complexity in supervised learning. While recent work has demonstrated the benefit of using AL in combination with large pre-trained language models (PLMs), it has often overlooked the practical challenges that hinder the effectiveness of AL. We address these challenges by leveraging representation smoothness analysis to ensure AL is feasible, that is, both effective and practicable. Firstly, we propose an early stopping technique that does not require a validation set – often unavailable in realistic AL conditions – and observe significant improvements over random sampling across multiple datasets and AL methods. Further, we find that task adaptation improves AL, whereas standard short fine-tuning in AL does not provide improvements over random sampling. Our work demonstrates the usefulness of representation smoothness analysis for AL and introduces an AL stopping criterion that reduces label complexity.

## introduction the notorious data hungriness of deep learning models emphasizes the importance of efficient and effective label acquisition. however, the labeling process is often tedious and expensive, ultimately slowing the development of labeled datasets and resulting in subpar models. evolved out of a practical necessity, active learning (al; @xcite is a special family of machine learning algorithms designed to reduce label complexity -the number of labels that a learning algorithm requires to achieve a given performance @xcite -and thus minimize labeling costs. an al method aims to select the most informative examples, which can be particularly useful when unla-beled data are abundant, but the labeling is costly or requires substantial expertise. the striking success of deep learning has motivated the use of traditional al techniques for training deep neural networks (dnns) and the development of novel al methods suited specifically to dnns. in natural language processing (nlp), al has been shown to outperform a random selection of examples in many nlp tasks @xcite @xcite . before the widespread adoption of large pre-trained language models (plms), a typical al approach to training deep models was to use task-specific neural models trained from scratch in each al step @xcite . since plms fine-tuned to downstream tasks outperform standard neural models, plms have supplanted most of them, and researchers have begun to investigate the feasibility of al for plms @xcite . recent work in al experimented with several training regimes, such as plm adaptation and specific fine-tuning techniques @xcite . in particular, task-adaptive pretraining (tapt) has emerged as a cost-effective method for performance improvement complementary to al @xcite . tapt uses additional pre-training on the unlabeled training set via masked language modeling and selfsupervision. in theory, combining al with adapted plms should produce greater reductions in label complexity than either of the methods in isolation. however, since research on combining al with plms is still in its infancy, whether it can work consistently better than random selection in realistic conditions remains an open question. one of the challenges in combining al and plms is that, although al is conceptually simple and promises efficiency gains, there are a host of practical challenges in deploying it in realistic conditions @xcite . the situation is further aggravated by the fact that most al research overlooks these challenges and resorts to unrealistic evaluation setups and resources. one of the most pervasive problems stems from using a hold-out set during training (e.g., a validation set for regularization by early stopping). in real applications, hold-out sets are unlikely to be available, as building them would require additional labeling effort the al is meant to reduce in the first place. another major problem is the flawed evaluation of al methods: typically, an al method is compared against random selection as the baseline, but the two training regimes are not kept identical, which confounds the measured effect of al. in addition to the above-mentioned problems, there is the important practical question of when to stop the acquisition of labels, i.e., how to define the al stopping criterion. ## related work our work builds on several strands of research, including practical challenges in al, combining al with plms, and different training setups for al acquisition models. practical challenges in al. despite the success of al for many nlp tasks, studies have identified a number of practical challenges hindering the broader deployment of al @xcite . the most obvious problem is the unavailability of a labeled validation set, an essential resource in model training typically used for hyperparameter optimization and regularization via early stopping. moreover, in realistic al conditions, a labeled test set is also unavailable, making a held-out evaluation of the underlying model's quality impossible. previous work mostly used model confidence or training error stability to evaluate the acquisition model and derive an al stopping criterion based on that estimation @xcite @xcite . however, these criteria have not been widely adopted as they often require tuning for specific datasets and tasks. we mitigate this by developing a taskagnostic al stopping criterion that detects the points of the largest reduction in label complexity compared to random selection. al with plms. only recently have large plms been coupled with al. early work concentrated mainly on the transformer architecture @xcite utilizing a simple training setup. more concretely, the predominant approach was to use a standard fine-tuning technique with a fixed number of training epochs, fine-tuning the model from scratch in each al step @xcite @xcite @xcite . however, @xcite and @xcite showed that fine-tuning in low-resource setups (scenarios with little training data) tends to be very unstable, especially when training for only a few epochs. this instability poses a serious issue, as al often implies a low-resource setting. moreover, fine-tuning is often sensitive to weight initialization and data ordering @xcite . this instability of plm fine-tuning also makes the al results unstable. we address the instability issue by proposing an early stopping technique without a validation set, and we show that combining plms with al is feasible. ## representation in besov space due to their remarkable flexibility and adaptivity, deep learning models have gained significant traction. to explain these phenomena, researchers have leveraged function space theory to develop approximation and estimation error analysis @xcite . our work relies on a particular type of analysis based on the theory of besov spaces. ## preliminaries in this section, we describe our experimental setup, detailing the datasets, models, al methods, and evaluation metrics. ## active sample smoothness in section 5, we analyzed the besov smoothness of layer representations of plms. in this section, we take a step further and examine the smoothness at the instance level. instead of using the representations on the training set, we computed the besov smoothness as the average across layers on the unseen (selected but not yet trained on) active sample, acquired by the al method. in contrast to the seen training examples, we argue that the besov smoothness on unseen examples can be interpreted as the amount of information the model could obtain from that sample. more precisely, the lower the smoothness of an active sample, the more informative it is for the model. in contrast, smooth samples are already well-represented and thus not as resource-effective as their less smooth counterparts. figure 3 : the relationship between an active sample and random sample smoothness. figure 3a shows how the smoothness of samples retrieved by al (red) relates to the smoothness of random samples (violet) with fitted regression lines. the smoothness values are calculated as the average across layers. figure 3b shows the corresponding al performance curve. the gray dotted line indicates the intersection of active and random sample smoothness, which signals the beginning of diminishing returns of al. we show the results for bert in the eta b regime on the trec-6 dataset as an illustration. we observe very similar patterns in other datasets. since all of the al methods display similar behavior in eta b regime, we show only mc to avoid clutter (cf. figure 5 in appendix for other datasets). best viewed on a computer screen. we compare the besov smoothness of actively acquired samples against random samples. we consistently observe two patterns, showcased by figure 3 . first, the smoothness of random samples is uniform throughout the al steps. the second pattern occurs in the trend of al sample smoothness. in the early al steps, al sample smoothness is low, indicating sharp representations that require smoothing (by learning). as the al procedure progresses, the acquisition model improves, and the active samples' smoothness increases. we interpret this as the model slowly consuming the information from the data pool, eventually reaching a state of "information depletion", i.e., a state in which the remaining unlabeled data provides no additional value to the model. ## conclusion in our paper, we leverage representation smoothness analysis to improve the effectiveness of active learning (al). in realistic conditions, we show that al with pre-trained language models (plms) is effective when combined with task adaptation, while standard short fine-tuning often fails. we address the problem of unavailable resources (labeled holdout sets) by developing the besov early stopping technique (beast) that does not require a validation set. for al to be feasible, it must be both effective and practicable. beast meets both feasibility requirements: it improves al performance over random sampling and reduces the variance of the performance scores across al steps (effectiveness) while not requiring additional labeled data (practicability). moreover, beast improves the performance of plms even in standard fine-tuning without al, which makes it potentially useful in zero-shot and few-shot setups where a validation set could also be unavailable. we further show the usefulness of representation smoothness analysis for al by devising a simple and effective al stopping criterion. we corroborate the hypothesis from previous research in that the effectiveness of al is influenced more by the training regime rather than the al method itself. we believe that the relationship between plms' generalization properties, label complexity, and representation smoothness is an exciting avenue for al, and we hope our results will motivate further research in that direction.