ALLSH : Active Learning Guided by Local Sensitivity and Hardness (2022)

Abstract: Active learning, which effectively collects informative unlabeled data for annotation, reduces the demand for labeled data. In this work, we propose to retrieve unlabeled samples with a local sensitivity and hardness-aware acquisition function. The proposed method generates data copies through local perturbations and selects data points whose predictive likelihoods diverge the most from their copies. We further empower our acquisition function by injecting the select-worst case perturbation. Our method achieves consistent gains over the commonly used active learning strategies in various classification tasks. Furthermore, we observe consistent improvements over the baselines on the study of prompt selection in prompt-based few-shot learning. These experiments demonstrate that our acquisition guided by local sensitivity and hardness can be effective and beneficial for many NLP tasks.

## introduction crowdsourcing annotations @xcite has become a common practice for developing nlp benchmark datasets. rich prior works @xcite @xcite show that the time-consuming and expensive manual labeling in crowdsourcing annotations are not an annotation artifact but rather core linguistic phenomena. active learning (al) is introduced to efficiently acquire data for annotation from a (typically large) pool of unlabeled data. its goal is to concentrate the human labeling effort on the most informative data in hopes of maximizing the model performance while minimizing the data annotation cost. popular approaches to acquiring data for al are uncertainty sampling and diversity sampling. uncertainty sampling selects data that the model predicts with low-confidence @xcite code is available at https://github.com/ szhang42/allsh @xcite . diversity sampling selects batches of unlabeled examples that are prototypical of the unlabeled pool to exploit heterogeneity in the feature space @xcite . different from these two perspectives, recent works focus on the informativeness of the selected data. for example, @xcite acquire informative unlabeled data using the training dynamics based on the model predictive log likelihood. @xcite construct contrastive examples in the input feature space. however, these methods either ignore the local sensitivity of the input features or take no consideration of the difficulty of the learning data. consequently, they may ignore examples around the decision boundary, or select hard-to-train or even noisy examples. their performance may further suffer under some practical settings, such as those with imbalanced labels and when there is a very limited annotation budget. in this work, we determine the informativeness by considering both the local sensitivity and learning difficulty. for local sensitivity, we take the classical definition from @xcite , which is widely used in both classic machine learning problems (e.g. @xcite @xcite @xcite and recent deep learning settings (e.g. @xcite @xcite . specifying a local region r region (x) around an example x, we assume in our prior that all examples in r region (x) have the same labels. 2 if the examples in r region (x) give us different labels, we say the local region of x is sensitive. data augmentation has been chosen as the way to create label-equivalent local regions in many recent works (e.g., @xcite . we utilize data augmentation as a tool to capture the local sensitivity and hardness of inputs and present allsh: active learning guided by local sensitivity and hardness. through various designs on local perturbations, allsh selects unlabeled data points from the pool whose predictive likelihoods diverge the most from their augmented copies. this way, allsh can effectively ensure the informative and local-sensitive data to have correct humanannotated labels. figure 1 illustrates the scheme of the proposed acquisition strategy. we conduct a comprehensive evaluation of our approach on datasets ranging from sentiment analysis, topic classification, natural language inference, to paraphrase detection. to measure the proposed acquisition function in more realistic settings where the samples stem from a dissimilar input distribution, we (1) set up an out-of-domain test dataset and (2) leak out-of-domain data (e.g., adversarial perturbations) into the selection pool. we further expand the proposed acquisition to a more challenging setting: prompt-based few-shot learning @xcite , where we query a fixed pre-trained language model via a natural language prompt containing a few training examples. we focus on selecting the most valuable prompts for a given test task (e.g., selecting 4 prompts for one given dataset). we adapt our acquisition function to retrieve prompts for the gpt-2 model. furthermore, we provide extensive ablation studies on different design choices for the acquisition function, including the designs of augmentations and divergences. our method shows consistent gains in all settings with multiple datasets. with little modification, our data acquisition can be easily applied to other nlp tasks for a better sample selection strategy. our contributions are summarized as follows: ## method in this section we present in detail our proposed method, allsh (algorithm 1). ## experimental settings in the prompt-based few-shot learning, we follow @xcite to use sst-2 @xcite ) for sentiment analysis, trec (voorhees and tice, 2000) for question classification, and rte @xcite for recognizing textual entailment. see appendix a for more details of the data. ## experiments we evaluate the performance of our acquisition and learning framework in this section. we bold the best results within random, entropy, badge, cal, and the proposed allsh (ours) in tables. then, we bold the best result within each column block. all experimental results are obtained with five independent runs to determine the variance. see appendix a for the full results with error bars. ## analysis can we use our proposed acquisition in the imbalance setting? extreme label imbalance is an important challenge in many non-pairwise nlp tasks @xcite @xcite would different augmentations make meaningful difference? we test if our results are sensitive to the choice of augmentation: tf-idf and backtranslation. for tf-idf, we compare the random sample augmentation and worst-case augmentation (wca). tf-idf and backtranslation generate diverse paraphrases while preserving the semantics of the original sentences. select-worst case augments the inputs by incorporating the approximate adversarial perturbations. we also observe that wca achieves the highest gains on two datasets. this confirms our discussion in section 2.3 that select-worst case is capable of imposing local sensitivity. what is the influence of the choice of divergence? we select different divergences in the statistical distance family and study their abilities in encoding different information. corresponding to section 2.3, we present the results in we experiment on the kl divergence, jsd, and α-divergence @xcite with the α value set as -0.5 or 0.5. we notice that for our case the difference between different divergences is small. a possible reason is that the number of class categories is small and therefore the choice of divergence does not have a large influence. can we use the proposed acquisition with extremely few labeled data? we have presented the results under very limited annotation budgets in the uncertainty-based methods @xcite have become the most common strategy. instead of only considering uncertainty, diversity sampling has also become an alternative direction. recent works @xcite annotation budgeting annotation budgeting with learning has long been studied @xcite . @xcite study the tradeoff between collecting multiple labels per example versus annotating more examples. on the other hand, different labeling strategies such as providing fine-grained rationales @xcite , active learning @xcite , and the training dynamics approach @xcite are studied. except standard classification, class-imbalance @xcite or noisy label cases @xcite have also been explored. we utilize active learning to explore the labeling strategies and aim to select the most informative data for annotations. ## conclusion our work demonstrates the benefits of introducing local sensitivity and learning from easy to hard into the acquisition strategy. the proposed acquisition function shows noticeable gains in performance across classification tasks and prompt-based fewshot learning. in this work, we conduct the detailed study with the proposed acquisition strategy in different settings, including imbalanced and extremely limited labels. we also verify the impact of different choice of designs such as the choice of divergence and augmentations. to summarize, the proposed allsh is effective and general, with the potential to be incorporated into existing models for various nlp tasks.