Cartography Active Learning (2021)

Abstract: We propose Cartography Active Learning (CAL), a novel Active Learning (AL) algorithm that exploits the behavior of the model on individual instances during training as a proxy to find the most informative instances for labeling. CAL is inspired by data maps, which were recently proposed to derive insights into dataset quality (Swayamdipta et al., 2020). We compare our method on popular text classification tasks to commonly used AL strategies, which instead rely on post-training behavior. We demonstrate that CAL is competitive to other common AL methods, showing that training dynamics derived from small seed data can be successfully used for AL. We provide insights into our new AL method by analyzing batch-level statistics utilizing the data maps. Our results further show that CAL results in a more data-efficient learning strategy, achieving comparable or better results with considerably less training data.

## introduction active learning (al) is a widely-used method to tackle the time-consuming and expensive collection and manual labeling of data. in recent years, many al strategies were proposed. the simplest and most widely used is uncertainty sampling @xcite , where the learner queries instances that it is most uncertain about. uncertainty sampling is myopic: it only measures the information content of a single data instance. alternative al algorithms instead focus on selecting a diverse batch @xcite @xcite or to estimate the uncertainty distribution of the learner @xcite . however, these methods are usually limited in their notion of informativeness, which is tied to post-training model uncertainty and batch diversity. recently, @xcite introduced data maps, to visualize the behaviour of the model on individual instances during training (training dynamics). the plotted data maps (figure 1 ) reveal distinct regions in a dataset: groups of ambiguous instances useful for high performance and linked to high informativeness, easy-to-learn instances which aid optimization, and hard-to-learn instances which frequently correspond to mislabeled or erroneous instances. we propose cartography active learning (cal), which automatically selects the the most informative instances that contribute optimally to model learning. to do so, we leverage a largely ignored source of information: insights derived during training, i.e., training dynamics derived from limited data maps (see section 4) to choose informative instances at the boundary of ambiguous and hard-to-learn instances. we hypothesize that this region is where the model will learn the most from. data maps provide the additional benefit that we can use them to measure informativeness of a batch with straightforward metrics and visualize dataset properties. these distinct regions in the data maps have their own respective statistics. therefore, as a second research question we investigate whether data map statistics help to assess why some al algorithms work better than others. contributions in this paper, our contributions are twofold. (1) we present cartography active learning, a novel al algorithm that exploits data maps for al. we compare our results against other competitive and widely used al algorithms and outperform them in early al iterations. (2) additionally, we leverage the data maps to inspect what instances al methods select. we show that our approach optimally selects informative instances avoiding only hard-to-learn and easy-tolearn cases, which leads to better al and comparable or better results than full dataset training. al has seen many usage scenarios in the natural language processing (nlp) field @xcite @xcite . the perspective of al is that if a model is allowed to select the data from which it will learn the most, it will achieve comparable (or better) performance with less training instances @xcite , and at the same time addressing the costly labeling process with a human annotator. a popular scenario is pool-based active learning @xcite @xcite , which assumes a small set of labeled data l and a large pool of unlabeled data u. most al algorithms start similarly: a model is fit to l to get access to p θ (y | x), then apply a query strategy to get the best scored instance from u, label this instance and add it to l in an iterative process. common strategies a commonly used query strategy is uncertainty sampling @xcite . in this approach, the learner queries the instances which it is least certain about. there are two popular approaches. ## cartography active learning the key idea of cal is to use model-independent measures, from fitting the model on the seed data l, by using data maps @xcite for al. data maps help identify characteristics of instances within the broader trends of a dataset by leveraging their training dynamics (i.e., the behavior of a model during training, such as mean and standard deviation of confidence and correctness with respect to the gold label). these modeldependent measures reveal distinct regions in a data map, by and large, reflecting instance properties (see figure 1 and details below on easy-to-learn, ambiguous, and hard-to-learn instances). training dynamics encapsulate information of data quality that has been largely ignored in al: the sweet spot of instances at the boundary of hard-to-learn and ambiguous instances, which are quick to label while providing informative samples, as shown in full data training @xcite . mapping the data formally, the training dynamics of instance i are defined as the statistics calculated over e epochs. these statistics are then used as the coordinates in the plot. the following statistics are calculated, confidence, variability, and correctness, following the notation of @xcite : confidence 2 (equation 1 ) is the mean model probability of the gold label (y * i ) across epochs. where p θ (e) is the model's probability with parameters θ (e) at the end of the e th epoch. then, variability (equation 2 ) is calculated as the standard deviation of p θ (e) (y * i | x i ), the spread across epochs e. last, correctness (equation 3 ) is denoted as the fraction of times the model correctly labels instance x i across epochs e. given the aforementioned training dynamics and the obtained statistics per instance, we plot the data maps for both agnews @xcite and trec @xcite , using all training data (figure 1 ). the data map is based on a multi-layer perceptron (mlp). as shown by @xcite , data maps identify three distinct regions: easy-to-learn, ambiguous, and hard-to-learn. the easy-to-learn instances are consistently predicted correctly with high confidence, these instances can be found in the upper region of the plot. the ambiguous samples have high variability and the model is inconsistent in predicting these correctly (middle region). the instances that are (almost) never predicted correctly, and have low confidence and variability, are referred to as hard-to-learn cases. this confirms findings by swayamdipta data maps from seed data given the distinct regions for data selection in the full data map in figure 1 , we first investigate whether regions are still identifiable if we have little amounts of training data, as this is a prerequisite for cal. figure 2 shows this for 1,000 training samples of agnews and 500 of trec. we can see that the data points are more scattered, where the easy-to-learn and ambiguous samples are mixed. however, it seems the hard-to-learn region can still qualitatively be distinguished from the other regions. ## experimental setup we focus on pool-based active learning. once trained on a seed set l, we begin the simulated al loop by iteratively selecting instances based on the scoring of an acquisition function. we take the top-50 instances, following prior work @xcite . the selected instances are shown the withheld label and added to the labeled set l and removed from u. we evaluate the performance of the trained model on a predefined held-out test set. we run 30 al iterations, over five random seeds, and report averages over these runs. in our al setup, we consider two popular text classification tasks, namely agnews @xcite and trec @xcite . the ag-news task entails classifying news articles into four classes: world, sports, business, science/technology. for trec, the task is to categorize questions into one of six categories based on the subject of the question, such as questions about locations, persons, concepts, et cetera. statistics of the data can be found in ## results & analysis we plot the accuracy of the al algorithms (section 4.2) on each dataset in figure 3 . for agnews and trec, all al strategies except dal outperform the random baseline. cal is statistically dominant over bald (agnews) and dal (agnews, trec), and competitive with lc and entropy ( ## conclusion in this paper, we introduced a new al algorithm, cartography active learning. the al objective is transformed into a binary classification task @xcite , where we optimize for selecting the most informative data with respect to a model by leveraging insights from data maps @xcite . data maps help to identify distinct regions in a dataset based on training dynamics (hard-to-learn and easy-tolearn/ambiguous instances), which have shown to play an important role in model optimization and stability in full dataset training @xcite . we use these insights in low-data regimes and propose cal. specifically for the task of visual question answering, contemporary work by @xcite use data maps for analysis into al acquisition functions and the effect of outliers. in contrast, this work instead applies data maps directly for al by training a discriminator on limited seed data maps to select the most informative instances. we show empirically that our method is competitive or significantly outperforms various popular al methods, and provide intuitions on why this is the case by using training dynamics. we initialize the main model with english d @xmath0, we sum the embeddings over the sequence of tokens as motivated by @xcite . for agnews we impose a maximum sequence length of 200 and a batch size of 64. for trec, a maximum sentence length of 42 and a batch size of 16. we run both models for 30 epochs, with no early stopping. per al iteration, we do a weight reset on all models. we average our results over five randomly generated seeds @xcite @xcite @xcite . all experiments were ran on an nvidia ® a100 sxm4 40 gb gpu and an amd epyc ™ 7662 64-core cpu. specifically for cal, a single al batch (50) selection iteration takes 11 seconds on average assuming trec. for agnews, one al iteration takes 62 seconds on average. both runtimes are with respect to the models depicted in section 4.3 and hardware mentioned above.