Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification (2022)

Abstract: To obtain a large amount of training labels inexpensively, researchers have recently adopted the weak supervision (WS) paradigm, which leverages labeling rules to synthesize training labels rather than using individual annotations to achieve competitive results for natural language processing (NLP) tasks. However, data imbalance is often overlooked in applying the WS paradigm, despite being a common issue in a variety of NLP tasks. To address this challenge, we propose Adaptive Ranking-based Sample Selection (ARS2), a model-agnostic framework to alleviate the data imbalance issue in the WS paradigm. Specifically, it calculates a probabilistic margin score based on the output of the current model to measure and rank the cleanliness of each data point. Then, the ranked data are sampled based on both class-wise and rule-aware ranking. In particular, the two sample strategies corresponds to our motivations: (1) to train the model with balanced data batches to reduce the data imbalance issue and (2) to exploit the expertise of each labeling rule for collecting clean samples. Experiments on four text classification datasets with four different imbalance ratios show that ARS2 outperformed the state-of-the-art imbalanced learning and WS methods, leading to a 2%-57.8% improvement on their F1-score.

## introduction deep learning models rely heavily on high-quality yet expensive, labeled data. owing to this considerable cost, the weak supervision (ws) paradigm has increasingly been used to reduce human efforts @xcite . this approach synthesizes training labels with labeling rules to significantly improve the efficiency of creating training sets and have achieved competitive results in natural language processing (nlp) @xcite @xcite . however, existing methods leveraging the ws paradigm to perform nlp tasks mostly focus on reducing the noise in training labels brought by labeling rules, while ignoring the common and critical problem of data imbalance. in fact, in a preliminary experiment performed as part of the present work (fig. 1 ), we found that the ws paradigm may amplify the imbalance ratio of the dataset because the synthesized training labels tend to have more imbalanced distribution. to address this issue, we propose ars2 as a general model-agnostic framework based on the ws paradigm. ars2 is mainly divided in two steps, including (1) warm-up, in which stage noisy data is used to train the model and obtain a noise detector; to select the samples most likely to be clean, we adopt a selection strategy based on small-loss, which is a very common method that has been verified to be effective in many situations @xcite @xcite . specifically, deep neural networks, have strong ability of memorization @xcite , will first memorize labels of clean data and then those of noisy data with the assumption that the clean data are of the majority in a noisy dataset. data with small loss can thus be regarded as clean examples with high probability. inspired by this approach, we propose probabilistic margin score (pms) as a criterion to judge whether data are clean. instead of using the confidence given by a model directly, a confidence margin is used for better performance @xcite . we also performed a comparative experiment on the use of margin versus the direct use of confidence, as described in sec. 3.3. sample selection based on weak labels can lead to severe class imbalance. consequently, models trained using these imbalanced subsets can exhibit both superior performance on majority classes and inferior performance on minority classes @xcite . a reweighted loss function can partially address this problem. however, performance remains nonetheless limited by noisy labels, that is, data with majority-class features may be annotated as minority-class data incorrectly, which misleads the training process. therefore, we propose a sample selection strategy based on class-wise ranking (cr) to address imbalanced data. using this strategy, we can select relatively balanced sample batches for training and avoid the strong influence of the majority class. to further exploit the expertise of labeling rules, we also propose another sample selection strategy called rule-aware ranking (rr). we use aggregated labels as pseudo-labels in the ws paradigm and discards weak labels. however, the annotations generated by rules are likely to contain a considerable amount of valid information. for example, some rules yield a high proportion of correct results. the higher the pms, the more likely the labeling result of the rules is to be close to the ground truth. using this strategy, we can select batches with clean data for training and avoid the influence of noise. the primary contributions of this work are summarized as follows. ## conclusion in this article, we have proposed ars2 as a method based on the ws paradigm to reduce both noise and the impact of natural imbalances in data. we have proposed pms to evaluate the level of noise in training data. to reduce the impact of noisy data on training, we have proposed two ranking strategies based on pms, including cr and rr. finally, adaptive sampling is performed on the data based on this ranking to clean the data. we have also presented experimental results on eight different datasets, which demonstrate that ars2 outperformed traditional ws and loss modification methods.