A nchor AL : Computationally Efficient Active Learning for Large and Imbalanced Datasets (2024)

Abstract: Active learning for imbalanced classification tasks is challenging as the minority classes naturally occur rarely. Gathering a large pool of unlabelled data is thus essential to capture minority instances. Standard pool-based active learning is computationally expensive on large pools and often reaches low accuracy by overfitting the initial decision boundary, thus failing to explore the input space and find minority instances. To address these issues we propose AnchorAL. At each iteration, AnchorAL chooses class-specific instances from the labelled set, or *anchors*, and retrieves the most similar unlabelled instances from the pool. This resulting *subpool* is then used for active learning. Using a small, fixed-sized subpool AnchorAL allows scaling any active learning strategy to large pools. By dynamically selecting different anchors at each iteration it promotes class balance and prevents overfitting the initial decision boundary, thus promoting the discovery of new clusters of minority instances. Experiments across different classification tasks, active learning strategies, and model architectures AnchorAL is *(i)* faster, often reducing runtime from hours to minutes, *(ii)* trains more performant models, *(iii)* and returns more balanced datasets than competing methods.

## introduction the abundance of web-scale textual data 1 has contributed to the success of generalist language models pretrained as multi-purpose foundation models and fine-tuned to solve downstream natural language processing tasks @xcite . the data used during fine-tuning critically affects their downstream abilities @xcite , especially in real-world applications where performance on rare concepts, or minority classes, is critical @xcite . standard pool-based active learning (al), however, struggles with large and imbalanced pools. first, al can be too computationally expensive due to its iterative nature. specifically, the inference costs of repeatedly evaluating a model on every unlabelled instance in each iteration can be prohibitive, especially given the size of modern language models @xcite ; furthermore, the annotation costs (i.e., the man-hours per annotation) can drastically increase due to the annotators' waiting time between iterations. second, al can be as good as random selection due to the imbalance: diversity-based strategies can be ineffective when minority and majority instances are not easily separable in high-dimensional spaces @xcite , while uncertainty-based strategies can fail to explore the input space and discover minority clusters because they tend to keep refining the initial decision boundary @xcite ; hybrid strategies suffer from similar limitations. one line of prior work has proposed to subset the pool in each iteration before running an al strategy to speed up instance selection @xcite . however, existing pool filtering methods either do not address class imbalance or are computationally inefficient. therefore, it is still an open question how to scale al on large datasets while addressing the imbalance. in this paper, we propose a new method, an-choral, designed to jointly address the learning and computational challenges of applying al to large and imbalanced datasets. anchoral works by filtering the pool before running any al strategy, thus reducing the inference costs and the annotators' waiting time in each iteration. crucially, the filtering process is designed to promote the exploration of the input space and the discovery of mi-nority instances while keeping instance selection time constant, regardless of the original pool size. specifically, at each iteration, anchoral chooses class-specific instances from the labelled set, which we term anchors. then, each unlabelled instance is scored based on its average distance from the anchors and the most similar are used to form the subset of the pool, which we term subpool. while any similarity measure works, we use the semantic representation capabilities of language models (e.g., @xcite and measure similarity based on cosine distance between instance representations. using a small, fixed-sized subpool an-choral allows scaling any active learning strategy to large pools while keeping the annotators' waiting time constant across iterations independently of the size of the original pool. optionally, the maximum size of the subpool can be set by the user, thus making anchoral suitable for different computational budgets. moreover, by dynamically selecting different anchors at each iteration it prevents overfitting the initial decision boundary, thus promoting the discovery of new clusters of minority instances; the class-specific anchors promote class balance even for al strategies that do not explicitly account for imbalance (fig. 2 ). we test the effectiveness of anchoral across 4 text classification tasks, both binary and multiclass, 3 al strategies (i.e., uncertainty, diversity, and hybrid), and 6 models with different sizes and architectures (i.e., encoder, decoder, and encoderdecoder). experiments show that anchoral is the fastest method, reducing the total selection time from hours to minutes; (often) the best-performing, reaching higher performance in less time and with fewer annotations; and discovers the most minority instances resulting in more balanced labelled sets. ## related work in this section, we provide the background on imbalanced learning and present the challenges of applying al to imbalanced datasets. then, we discuss approaches to address these challenges and scale al to large pools. ## methodology we consider the standard pool-based al setting for classification tasks @xcite wherein we are given a (large) pool of unlabelled data u and access to an oracle f ⋆ that given an instance x returns its true foot_0 label @xmath0, ..., c}. the goal is to induce the best possible classifier f within a fixed annotation budget b. the annotation process happens iteratively over t iterations. we assume access to a small initial labelled set d 0 with at least one instance per class used to bootstrap an initial classifier f 0 . in each iteration, we train a new version of the classifier f t on the available labelled instances d t . then, the al strategy φ uses the model to inform the selection of the set of instances to annotate b t ⊂ u t such that |b t | =⌊b/t ⌋. finally, the selected instances are annotated by the oracle, removed from the pool, and added to the labelled set; and the cycle repeats. ## experimental setup we mimic a realistic and interactive annotation setting @xcite @xcite characterised by a budget of 5k foot_4 annotations and label 25 instances per iteration. 7 we start with an initial type model # params encoder bert-base @xcite ) 110m bert-tiny @xcite 4.4m albert-base @xcite 12m deberta-base @xcite 86m decoder gpt-2 @xcite 117m enc-dec t5-base @xcite 220m baselines. we compare anchoral with two pool filtering methods. randsub @xcite samples m instances uniformly at random from the pool; we set @xmath1. foot_8 seals @xcite limits the pool to the set of @xmath2: after every instance is selected, its k-nearest neighbours are added to the pool. moreover, we compare against standard al (no-op) which considers the entire pool and random sampling (random). ## results cost efficiency. broadly, the overall annotation cost can be divided into the cost of the compute needed to run inference on the pool and the cost of the human annotation. pool filtering methods reduce both cost components by considering only a subset of the pool in each iteration: fewer instances require less compute and are faster to process which makes annotators spend less time waiting for instances to label resulting in more annotations within the same time budget. anchoral is the fastest method overall, reducing the total annotators' waiting time ("time" column) from hours to less than 5 minutes and labelling the most instances within the time budget ("budget" column). anchoral is faster because it returns a smaller subpool than the other methods (fig. 2 panel a). for example, anchoral @xmath3, by design, the subpool grows across iterations and for randsub a larger subpool is required to achieve good performance. in theory, randsub can be made arbitrarily fast by choosing a small m but it comes with high performance costs. for example, setting @xmath4, similar to anchoral, results in a dramatic performance drop ( performance. anchoral is the best-performing method for both minority and majority classes per time-("overall" columns) and budget-unit ("budget-matched" columns) across models and al strategies. the only exception is agnews-bus where it slightly underperforms seals on the minority class(es). we hypothesise that this might be an artefact of the downsampling procedure we applied to this task to make it imbalanced: minority instances may be near the initial decision boundary, making exploration of the input space detrimental. furthermore, we notice that randsub is a stronger baseline than reported by @xcite . we hypothesise that this discrepancy depends on the fact that we train the entire model rather than only the classification head. ## analysis in this section, we study anchoral's anchoring mechanism. this mechanism is the core component allowing anchoral to reach higher performance despite using a fixed-sized, small subpool. everything else being equal, if the anchoring mechanism is turned off the performance is dramatically affected, as shown in the last row of ## conclusions we propose anchoral, a novel pool filtering method designed to scale al to large pools while addressing class imbalance. anchoral uses the semantic representation capabilities of language models to explore the input space and create a fixed-sized, smaller, more balanced, and different subpool in each iteration. by running the al strategy on the subpool, anchoral promotes the discovery of minority instances, prevents overfitting to the initial labelled set, and obtains a constant instance selection time, independently of the original pool size.