A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing (2023)

Abstract: Many natural language processing (NLP) tasks are naturally imbalanced, as some target categories occur much more frequently than others in the real world. In such scenarios, current NLP models tend to perform poorly on less frequent classes. Addressing class imbalance in NLP is an active research topic, yet, finding a good approach for a particular task and imbalance scenario is difficult. In this survey, the first overview on class imbalance in deep-learning based NLP, we first discuss various types of controlled and real-world class imbalance. Our survey then covers approaches that have been explicitly proposed for class-imbalanced NLP tasks or, originating in the computer vision community, have been evaluated on them. We organize the methods by whether they are based on sampling, data augmentation, choice of loss function, staged learning, or model design. Finally, we discuss open problems and how to move forward.

## introduction class imbalance is a major problem in natural language processing (nlp), because target category distributions are almost always skewed in nlp tasks. as illustrated by figure 1 , this often leads to poor performance on minority classes. which categories matter is highly task-specific and may even depend on the intended downstream use. developing methods that improve model performance in imbalanced data settings has been an active area for decades (e.g., @xcite @xcite @xcite , and is recently gaining momentum in the context of maturing neural approaches (e.g., @xcite @xcite @xcite . the problem is exacerbated when classes overlap in the feature space @xcite . for example, in patent classification, technical categories differ largely in frequency, and the concepts mentioned in the different categories can be very similar. on a large variety of nlp tasks, transformer models such as bert @xcite outperform both their neural predecessors and traditional models @xcite @xcite . performance for minority classes is also often higher when using self-supervised pre-trained models (e.g., @xcite , which parallels findings from computer vision @xcite . however, the advent of bert has not solved the class imbalance problem in nlp, as illustrated by figure 1 . @xcite find that on synthetically imbalanced named entity datasets with 0 500 1000 # of samples majority classes having thousands of examples, at least 25 instances are required to predict a class at all, and 100 examples to learn to predict it with some accuracy. despite the relevance of class imbalance to nlp, related surveys only exist in the computer vision domain @xcite . incorporating methods addressing class imbalance can lead to performance gains of up to 20%. yet, nlp research often overlooks how important this is in practical applications, where minority classes may be of special interest. our contribution is to draw a clear landscape of approaches applicable to deep-learning (dl) based nlp. we set out with a problem definition (sec. 2), and then organize approaches by whether they are based on sampling, data augmentation, choice of loss function, staged learning, or model design (sec. 3). our extensive survey finds that re-sampling, data augmentation, and changing the loss function can be relatively simple ways to increase performance in class-imbalanced settings and are thus straightforward choices for nlp practitioners. 1 while promising research directions, staged learning or model modifications often are implementation-wise and/or computationally costlier. moreover, we discuss particular challenges of non-standard classification settings, e.g., imbalanced multi-label classification and catch-all classes, and provide useful connections to related computer vision work. finally, we outline promising directions for future research (sec. 4). scope of this survey. we focus on approaches evaluated on or developed for neural methods. work from "traditional" nlp (e.g., @xcite @xcite as well as natural language generation (e.g., @xcite and automatic speech recognition (e.g., @xcite are not ad1 we provide practical advice on identifying potentially applicable class imbalance methods in the appendix (figure 3 ). dressed in this survey. other types of imbalances such as differently sized data sets of subtasks in continual learning @xcite or imbalanced regression @xcite are also beyond the scope of this survey. in sec. 3.5, we briefly touch upon the related area of few-shot learning @xcite . related surveys. we review imbalance-specific data augmentation approaches in sec. 3.2. @xcite give a broader overview of data augmentation in nlp, @xcite provide an overview of low-resource nlp, and ramponi and plank (2020) discuss neural domain adaptation. ## problem definition class imbalance refers to a classification setting in which one or multiple classes (minority classes) are considerably less frequent than others (majority classes). more concrete definitions, e.g., regarding the relative share up to which a class is seen as a minority class, depend on the task, dataset and labelset size. much research focuses on improving all minority classes equally while maintaining or at least monitoring majority class performance (e.g., @xcite @xcite . we next discuss prototypical types of imbalance (sec. 2.1) and then compare controlled and real-world settings (sec. 2.2). ## methods for addressing class imbalance in nlp in this section, we survey methods that either have been explicitly proposed to address classimbalance issues in nlp or that have been empirically shown to be applicable for nlp problems. we provide an overview of which methods are applicable to a selection of nlp tasks in appendix a. ## insights and future directions we have provided a comprehensive, concise and structured overview of current approaches to dealing with class imbalance in dl-based nlp. what works (best)? as there is no established benchmark for class-imbalanced settings, evaluation results are hard to compare across papers. in general, re-sampling or changing the loss function may lead to small to moderate gains. for data augmentation approaches, the reported performance increases tend to be larger than for re-sampling or new loss functions. the effects of staged training or modifications of the model vary drastically, ranging from detrimental to very large performance gains. hence, re-sampling, data augmentation, and changing the loss function are straightforward choices in class-imbalanced settings. approaches based on staged learning or model design may sometimes outperform them, but often come with a higher implementation or computational cost. for a practical decision aid and potential application settings of some class imbalance methods, see figure 3 in appendix b and how should we report results? much nlp research only reports aggregate statistics @xcite , making it hard to judge the impact on improvements by class, which is often important in practice. we thus argue that nlp researchers should always report per-class statistics, e.g., as in figure 1 . open-sourcing spreadsheets with the exact numbers would enable the community to compare systems more flexibly from multiple angles, i.e., with respect to whichever class(es) matter in a particular application scenario, and to re-use this data in research on class imbalance. reviewers should also value works that analyze performance for relevant minority classes rather than focusing largely only on overall accuracy improvements. a main hindrance to making progress on class imbalance in computer vision and nlp alike is that experimental results are often hard to compare @xcite . a first important step would be to not restrict baselines to methods of the same type, e.g., a new data augmentation approach should not only compare to other data augmentation methods, but also to using loss functions for class imbalance. establishing a shared and systematic benchmark of a diverse set of class-imbalanced nlp tasks would be highly beneficial for both researchers and practitioners. how can we move forward? most work on class-imbalanced nlp has focused on single-label text classification. finding good solutions for multilabel settings is still an open research challenge. class imbalance also poses problems in nlp tasks such as sequence labeling or parsing, and we believe that the interaction of structured prediction models with methods to address class imbalance is a promising area for future research. moreover, we need to study how class imbalance methods affect prediction calibration in order to provide reliable confidence estimates. in general, inspiration for new model architectures could for example be drawn from approaches developed for few-shot learning @xcite . recently, prompting @xcite has emerged as a new paradigm in nlp, which performs strongly in real-world few-shot settings @xcite . methods that improve worst-case performance under distribution shift (e.g., @xcite might also be applied to improve minority-class performance.