Class Balancing for Efficient Active Learning in Imbalanced Datasets (2024)

Abstract: Recent developments in active learning algorithms for NLP tasks show promising results in terms of reducing labelling complexity. In this paper we extend this effort to imbalanced datasets; we bridge between the active learning approach of obtaining diverse andinformative examples, and the heuristic of class balancing used in imbalanced datasets. We develop a novel tune-free weighting technique that canbe applied to various existing active learning algorithms, adding a component of class balancing. We compare several active learning algorithms to their modified version on multiple public datasetsand show that when the classes are imbalanced, with manual annotation effort remaining equal the modified version significantly outperforms the original both in terms of the test metric and the number of obtained minority examples. Moreover, when the imbalance is mild or non-existent (classes are completely balanced), our technique does not harm the base algorithms.

## introduction pre-trained language models (plms) and masked language models @xcite @xcite have revolutionized nlp by supplying meaningful contextual embeddings for tokens and sentences. these models pushed sota on many tasks, proving especially effective for text classification tasks @xcite . for many of these tasks, given enough labeled data for finetuning a plm, satisfactory results can be achieved. on the other hand, due to the rapid growth in textual datasets, for many domains and tasks it is often the case that while a vast amount of textual data is available, only a small portion of the text-instances belong to a specific class of interest @xcite @xcite . this sparsity makes annotation challenging because naive data sampling methods do not produce enough examples belonging to a class of interest. it is often possible to use some filtering heuristic before performing manual annotations, so that the percentage of data belonging to the class in question will rise. this approach however is not scalable, since it requires hand-crafting for each specific task or even class. in addition, when filtering is imperfect, the plm may never observe an important cluster of examples, hurting its generalization capabilities. therefore, it is essential to minimize the amount of human annotations needed in order to acquire labeled data that will be sufficient for the model to reach a satisfying performance. this calls for the use of active learning (al) techniques. there are many works applying al techniques for plm fine-tuning @xcite @xcite but most do not specifically consider the imbalanced dataset case. standard al objectives include obtaining samples which the model is less certain about and increasing the diversity of the chosen sample set. existing al methods @xcite aimed to handle imbalanced datasets suggest doing so by alternating between some approach designed to obtain positive examples foot_0 , and another approach with standard al objectives. the shortcoming of this alternating approach is that the exact balance of objectives, i.e., how many samples should be chosen aimed to be positive vs diverse, etc., is dependent on the imbalance of the dataset. as a result, existing methods either require a balancing hyperparameter that cannot be tuned in a practical foot_1 setting, or need to learn the correct balancing ratio over time, making the methods efficient only after a large amount of labelling. in this work we take a different approach that modifies an al algorithm to favor positive exam-ples in a way that is self-tuned to obtain an equal amount of positive and negative samples. the core idea of our approach is to debias the inherent skew in the imbalanced dataset by utilizing the inversepropensity score of the probability of a sample to be positive, according to the model obtained thus far. the simplest form of our algorithm is to sample according to these scores. intuitively, once the model is reasonable, and this happens quickly in the era of pre-trained models, the amount of positive samples is in the same order of magnitude as the negative samples. we show that these scores can be used not only as a tool to modify the uniform sampling approach, but can be combined with other algorithms providing their own weights to the data points @xcite , or clustering approaches aimed to maximize diversity @xcite . we demonstrate that for the imbalanced setting, our adaptation of sota al algorithms outperform their original counterparts in multiple datasets. we show that this behavior is consistent across various levels of data imbalance and that in the balanced setting, our adaptation is comparable to the original al algorithms. the latter property is crucial from a practical perspective as the imbalance ratio of a dataset is rarely known in advance and can be difficult to estimate. concluding, we showcase a weakness of recent al approaches when dealing with imbalanced datasets and devise a novel self-tuned reweighting solution that complements existing (including sota) al algorithms, improving their performance on imbalanced datasets. we experiment on four imbalanced datasets from different tasks and domains. we publish our code and imbalanced datasets for reproducibilty and to encourage future research in this area foot_2 . ## related work conventional active learning. there is a myriad of works on active learning in different settings @xcite . herein, we consider a pool-based active learning scenario where the algorithm has access to a large unlabeled dataset u that can be labeled through human annotations. as previously mentioned, our solution modifies existing active learning algorithms in order to better deal with imbalanced datasets. we identify three types of active learning algorithms: random, embedding-based, and score-based algorithms. random sampling, the simplest approach, generates a distribution over u from which k distinct samples are randomly selected (e.g., uniform sampling). embedding-based algorithms embed unlabeled samples in a high dimensional space. a subset of samples is then selected for annotation using a clustering algorithm based on their embedding, to increase the sample diversity. badge @xcite ) is an embedding-based algorithm where the sample gradient from the last layer of the model, taken from the log loss of the predicted label, is used as an embedding. alps @xcite adopts a similar approach, but uses an embedding that captures the language model uncertainty of the different tokens in the sentence. score-based algorithms attach a model score to each sample. the most common score-based approach is least confidence (lewis and gale, 1994) which selects the top-k samples with the highest model uncertainty. a more recent work called dal @xcite trains a classifier to detect new samples dissimilar from the already labeled data. active learning in skewed datasets. compared to active learning, relatively only a few studies have specifically addressed the issue of class imbalance. a common approach in handling class imbalance is to skew the active learning algorithm towards ranking positive samples higher. odal @xcite is an extension of dal that targets the cold-start setting using an outlier detection algorithm until a single positive sample is found. hal @xcite suggests a hybrid active learning algorithm that switches between selecting 'exploitation' that chooses points where the model is uncertain and 'exploration' that either chooses samples uniformly at random or according to how far they are from already chosen samples, yet this approach requires a hyperparameter determining how to balance exploration and exploitation, that is crucial to the performance, and setting it is said to be an open problem. recent works try to combine conventional active learning algorithms with positive-skewed sampling. c lin (2018) explore a method that combines multiple strategies for selecting new examples such as standard active learning, sampling in a skewed way towards positive examples, and generating examples of the minority type (via manual annotators). they apply a multi-armed bandit (mab) algorithm (ucb) to balance between the strategies. similarly, bmp @xcite defines two types of policies, one samples from the positive class and the other uses a standard al algorithm such as random or ve @xcite , and uses a mab algorithm to dynamically allocate batches to the different policies. it is folklore knowledge that mab algorithms require a large number of rounds (at least 10s) before they can learn anything meaningful foot_3 , and indeed the number of rounds in the mentioned papers are 55 or more. having a large number of rounds can be computationally expensive as sota active learning algorithms require applying a large ml model on all examples, as well as time consuming in a human annotation setting. we thus focus on a small number of rounds (say, 5), making these works less applicable. ## setting we consider the pool-based active learning setting, where we are given a pool of unlabeled samples u. at each iteration t, some subset s t ⊂ u of size k is selected for annotation. this subset, sometimes together with s 1 , . . . , s t-1 , is used to train a model m t . the subset s t is selected by a sampling algorithm a, which is usually dependent on the model m t-1 . a formal description of this process is given in algorithm 1. the goal in this setting is to select a subset d ⊂ u to be labeled, such that a model trained on d optimizes some metric on a separate test set. we focus on the binary classification problem, as it includes multiple common and important imbalanced scenarios such as phishing attempts, fraud, etc. @xcite @xcite . ## our approach consider some iteration t where we have at our disposal a model m t-1 trained on samples labelled in previous iterations. 5 for a sample @xmath0, we consider m t-1 (x) as an approximation of the probability of x being a positive example. if we uti-algorithm 1 pool-based active learning input: unlabeled data pool u, number of samples per iteration k, number of iterations t , sampling algorithm a, classifier model training algorithm m and seed samples s. our goal is to construct a distribution p over u such that the resulting random variable m t-1 (x) is uniformly distributed in [0, 1]. to motivate this idea, consider a case where the model is perfectly calibrated, meaning the probability it provides is correct. here, if sampled according to p, half of the examples, in expectation, will be positive (in observation 1 we provide an additional bound under a weaker assumption). another advantage of such a distribution appears even in the case of balanced datasets. consider a case where many of the examples are easy to classify. it is likely that in an early stage easy samples will be concentrated near 0 and 1 in the distribution of m t-1 (x). this means that our sample strategy will upweight exactly the problematic points where the model is uncertain, such that we obtain the property of biasing towards informative examples. the distribution p described above can be achieved by weighting each sample inversely to its propensity results in a uniform sample. thus, we call this method invprop. in terms of implementation, we discretize the interval [0, 1] into m bins uniformly, meaning [0, 1 m ), [ 1 m , 2 m ), . . . , [ m-1 m , 1], as can be seen in figure 1(a) . for bin i that contains b i samples we give each sample a probability proportional to 1 b i (see figure 1(b) ). the following lemma demonstrates the robustness of invprop-weights to the exact, and unknown, degree of imbalance in the data. ## conclusions in this paper, we tackled the problem of al in imbalanced datasets. we propose a novel weightingtechnique, invprop , and apply it to three recent al algorithms (badge, apls and dal) as well as to naive random sampling. we show results for the posprob weighting scheme in addition to invprop and compare to a uniform weighting baseline on four datasets. we show that invprop-weighting consistently finds more positive examples, and leads to better or comparable performance compared to other weighting schemes on all tested datasets and al algorithms. we also test various imbalance settings showing that invprop outperforms the uniform baseline across all imbalance ratios which represent some imbalance. in addition, on a balanced dataset the uniform baseline outperforms our method in only one metric, while being comparable in the rest. compared to posprob, our solution proved to be more robust to changes in the imbalance ratio. when testing extreme scenarios where the imbalance is less than 0.5%, all tested algorithms perform worse than the simple random sampling baseline, suggesting that al algorithms are not advantageous given an uninformative model. concluding, we show our novel weighting scheme improves several sota al algorithms on various datasets and imbalance ratios.