Investigating Multi-source Active Learning for Natural Language Inference (2023)

Abstract: In recent years, active learning has been successfully applied to an array of NLP tasks. However, prior work often assumes that training and test data are drawn from the same distribution. This is problematic, as in real-life settings data may stem from several sources of varying relevance and quality. We show that four popular active learning schemes fail to outperform random selection when applied to unlabelled pools comprised of multiple data sources on the task of natural language inference. We reveal that uncertainty-based strategies perform poorly due to the acquisition of collective outliers, i.e., hard-to-learn instances that hamper learning and generalisation. When outliers are removed, strategies are found to recover and outperform random baselines. In further analysis, we find that collective outliers vary in form between sources, and show that hard-to-learn data is not always categorically harmful. Lastly, we leverage dataset cartography to introduce difficulty-stratified testing and find that different strategies are affected differently by example learnability and difficulty.

## introduction in recent years, active learning (al) @xcite has emerged as a promising avenue for dataefficient supervised learning @xcite . al has been successfully applied to a variety of nlp tasks, such as text classification @xcite @xcite @xcite , entity recognition @xcite @xcite , part-of-speech tagging @xcite and neural machine translation @xcite @xcite . however, these works share a major limitation: they often implicitly assume that unlabelled training data comes from a single source 1 (houlsby 1 throughout the paper, we use the term "source" to de-figure 1 : the pool of unlabelled data consists of a source s, or multiple sources i s i , for the singlesource and multi-source al setting, respectively. the indomain test set follows the same distribution of sources. @xcite @xcite @xcite . we refer to this setting as singlesource al. the single-source assumption is problematic for various reasons @xcite . in real-life settings we have no guarantees that all unlabelled data at our disposal will necessarily stem from the same distribution, nor will we have assurances that all examples are of consistent quality, or that they bear sufficient relevancy to our task. for instance, quality issues may arise when unlabelled data is collected through noisy processes with limited room for monitoring individual samples, such as web-crawling @xcite . alternatively, one may have access to several sources of unlabelled data of decent quality, but incomplete knowledge of their relevance to the task-at-hand. for instance, medical data may be collected from various different physical sites (hospitals, clinics, general practitioners) which may differ statistically from the targte distribution due to e.g. differences in patient demographics. ideally, al methods should be robust towards these conditions in order to achieve adequate solutions. in this work, we study whether existing al methods can adequately select relevant data points in a multi-source scenario for nlp. we examine robustness by evaluating al performance in both in-domain (id) and out-of-domain (ood) settings, while we also conduct an extensive analysis to interpret our findings. a primary phenomenon of interest here concerns collective outliers: data points which models struggle to learn due to high ambiguity, the requirement of specialist skills or labelling errors @xcite . while such outliers were previously found to disrupt several al methods for visual question answering @xcite , their impact on textbased tasks remains under-explored. given the wide body of work on the noise and biases that pervade nli @xcite @xcite @xcite @xcite , we may reasonably assume popular nli datasets to suffer from collective outliers as well. concretely, our contributions can be summarised as follows: (1) we apply several popular al methods in the under-explored multi-source, poolbased setting on the task of nli, using robertalarge @xcite as our acquisition model, and find that no strategy consistently outperforms random selection ( §5). (2) we seek to explain our findings by creating datamaps to explore the actively acquired data ( §6.1) and show that uncertainty-based acquisition functions perform poorly due to acquisition of collective outliers ( §6.2). (3) we examine the effect of training data difficulty on downstream performance ( §7.1) and after thorough experiments we find that uncertaintybased al methods recover or even surpass random selection when hard-to-learn data points are removed from the pool ( §7.2). (4) finally, we introduce difficulty-stratified testing and show that the learnability of acquired training data affects different strategies differently at test-time ( §7.3). our code is publicly available at https://github. com/asnijders/multi_source_al . ## single & multi-source active learning we assume a warm-start, pool-based al scenario @xcite with access to a pool of unlabelled training data, d pool , and a seed dataset of labelled examples, d train . during each iteration i of al, we first train a model m with d train and then use it in conjunction with some acquisition function a to select a new batch of unlabelled examples d batch from d pool for labelling. upon labelling, these examples are removed from d pool and added to d train , after which a new round of al begins. single-source al for single-source al, we assume that d pool is a set of unlabelled data that stems from a single source s, i.e. d @xmath0. multi-source al for multi-source al, we assume that d pool comprises a union of distinct ## experimental setup data we perform experiments on natural language inference (nli), a popular classification task to gauge a model's natural language understanding @xcite . we construct the unlabelled pool from three distinct datasets: snli @xcite , anli @xcite and wanli @xcite . we consider mnli @xcite as an out-of-domain set to evaluate the transferability of actively acquired training sets. for more details see appendix a.2. experiments we apply al with two distinct end-goals: in-domain (id) generalization, where the same source(s) are used for both the unlabelled pool d pool and the test set d test , and out-ofdomain (ood) generalization, where we evaluate on the test set of an external source of which no training data was present in the unlabelled pool. constructing multi-source pools for multisource al, the unlabelled pool comprises the union of the snli, anli and wanli training sets. during model selection, we similarly assume the union of snli, anli and wanli validation sets. for both training and validation data we down-sample sources to the minority source size to obtain pools with even shares per source. we sub-sample training data for each source to reduce experiment runtimes. this yields a pool of 60k unlabelled training examples comprised of 20k shares of each source. ## dataset cartography for al diagnosis our findings in both single and multi-source al settings, for both in-domain and out-of-domain evaluations, showed poor performance of all algorithms ( §5). we therefore aim to investigate if the answer for the observed al failure may lie in the presence of so-called collective outliers: examples that models find hard to learn as a result of factors, such as high ambiguity, underspecification, requirement of specialist skills or labelling errors @xcite . collective outliers can be identified through dataset cartography @xcite , a post-hoc modelbased diagnostic tool which plots training examples along a so-called learnability spectrum. ## stratified analysis on data difficulty we begin our analysis of collective outliers by exploring impossible data points in the three nli datasets we use; snli, anli and wanli. we denote as impossible data points those that yield a confidence value in the range 0 ≤ @xmath1.25 ( §6.1). we provide some examples in ## conclusion and future work our work highlights the challenge of successfully applying al when the (training) pool comprises several sources which span various domains in the task of nli. similar to @xcite , we show that uncertainty-based strategies, such as mcme and bald, perform poorly ( §5) due to acquisition of collective outliers which impede successful learning ( §6.2). however, these strategies recover when outliers are removed ( §7.2). practically, this suggests that uncertainty-based methods may fare well under more carefully curated datasets and labelling schemes, while alternative strategies (e.g. diversity-based) may be preferable in cases with poorer quality guarantees (e.g. data collection with limited budget for annotation verification). next, we find that performance outcomes between strategies differ for test data of various difficulties ( §7.3). on the one hand, this complicates strategy selection: it is unclear whether a strategy that performs well on hard data but poorer on easy data is preferable to a strategy with opposite properties. on the other hand, knowing which strategies work well for test data of a certain difficulty may be advantageous when the difficulty of the test set is known, in out-of-domain settings @xcite . lastly, in contrast with @xcite and @xcite , we have shown that cases exist in which training examples in the hardto-learn region do not hamper learning but are in fact pivotal for achieving good generalization ( §7.1). consequently, there may be value in refining existing cartography-based methods such that they can discriminate between useful and harmful hard-to-learn data. more broadly, our findings underscore the potential of understanding these phenomena for other nlp tasks and datasets.