Balancing via Generation for Multi-Class Text Classification Improvement (2020)

Abstract: Data balancing is a known technique for improving the performance of classification tasks. In this work we define a novel balancing-viageneration framework termed BalaGen. BalaGen consists of a flexible balancing policy coupled with a text generation mechanism. Combined, these two techniques can be used to augment a dataset for more balanced distribution. We evaluate BalaGen on three publicly available semantic utterance classification (SUC) datasets. One of these is a new COVID-19 Q&A dataset published here for the first time. Our work demonstrates that optimal balancing policies can significantly improve classifier performance, while augmenting just part of the classes and under-sampling others. Furthermore, capitalizing on the advantages of balancing, we show its usefulness in all relevant BalaGen framework components. We validate the superiority of BalaGen on ten semantic utterance datasets taken from real-life goaloriented dialogue systems. Based on our results we encourage using data balancing prior to training for text classification tasks.

## introduction imbalanced datasets pose a known difficulty in achieving ultimate classification performance as classifiers tend to be biassed towards larger classes @xcite @xcite . moreover, identifying samples that belong to under-represented classes is of high importance in many real-life domains such as fraud detection, disease diagnosis, and cyber security. although the imbalanced data classification problem is well-defined, and has been researched extensively over the last two decades (estabrooks et al., 2004; batista et al., 2004; ramyachitra and manikandan, 2014; zhu et al., 2017; buda et al., * equal contribution 2018), there has been considerably less work devoted to balancing textual datasets. we propose a novel balancing-via-generation framework, termed balagen, to improves textual classification performance. balagen uses a balancing policy to identify over-and under-represented classes. it then uses controlled text generation, coupled with a weak labeling mechanism to augment the under-represented classes. additionally, it applies under-sampling to decrease the overrepresented classes. our analysis is focused on semantic utterance classification (suc) @xcite @xcite . suc is a fundamental, multi-class, highly imbalanced textual classification problem. for example, it is widely used for intent (class) detection in goaloriented dialogue systems @xcite , and for frequently asked question (faq) retrieval @xcite @xcite . correctly identifying scarce utterances is of great importance in many real life scenarios. for example, consider a scenario in which a user converses with the dialogue system in an online shop @xcite . for the store owner, the task of correctly identifying the buying-intent utterances is paramount. however, the number of utterances related to searching for products is expected to be significantly higher, thus biasing the classifier toward this intent. we analyzed balagen's capabilities on two publicly available suc datasets. in addition, we introduce a new dataset called covid-19 q&a (cqa), which contains answers to questions frequently asked by the public during the pandemic period. analysis of this new dataset further demonstrates improved performance using our approach. our contribution is thus four-fold: i) we present balagen, a balancing-via-generation framework for optimizing classification performance on imbalanced multi-class textual datasets. @xcite we analyze different factors that affect balagen's performance, including quality of generated textual data, weak supervision mechanisms, and balancing of bala-gen's internal components. iii) we validate our approach on 3 publicly available datasets and a collection of 10 suc datasets used to train real-life goal-oriented dialogue systems. iv) we contribute a new covid-19 related suc dataset. ## related work in imbalanced classification, also known as the "class imbalance problem", classifiers tend to bias towards larger classes @xcite . this challenge, has garnered extensive research over the past decades @xcite @xcite . the range of approaches to solve this issue depends on the type of data and the target classifier @xcite @xcite . @xcite divide classification improvements over imbalanced datasets into five levels: data, algorithmic, cost sensitive, feature selection and ensemble. we focus our review on the data level and specifically on textual dataset balancing. primary data-level methods vary the number of samples in the dataset via re-sampling. we follow the common terminology and refer to a method that adds samples to a dataset, as over-sampling, and to a method that removes samples as undersampling. sample-copy, i.e. duplicating existing samples, is the most straightforward over-sampling method and random-selection is the most straightforward under-sampling method. while these methods were shown to be effective to some extent for data balancing, they are insufficient when it comes to solving the problem @xcite . traditional and well researched feature-based over-sampling techniques generate new samples via feature manipulation @xcite . most of these techniques are based on the synthetic minority oversampling technique (smote) @xcite or the adaptive synthetic (adasyn) approach @xcite . these approaches create synthetic samples by manipulating the feature values of existing samples. however, the latest deep learning (dl) models do not have an explainable features layer to manipulate. although the embedding layer may be perceived as the dl analogy to the traditional feature layer, this layer is of high dimension and is not easy to interpret and manipulate while preserving the original class label. thus, local changes to the embedding values of textual datasets does not yield the expected results. in contrast to feature-based over-sampling techniques, data augmentation generates additional samples through transformations applied directly to the data. for example, easy data augmentation (eda) @xcite is a naïve yet effective text augmentation technique based on synonym replacement using wordnet @xcite , random insertion, random swap, and random deletion of words. language model-based markov chain (mc) @xcite is another example of a word level second-order model that was shown to improve textual data-balancing @xcite . additional research works includes structure preserving word replacement using a language model @xcite , recurrent neural language generation for augmentation @xcite , and various parapharasing methods as done in @xcite . recently, transformer-based pre-trained architectures @xcite have been developed and successfully applied to a wide set of natural language generation (nlg), processing and understanding tasks. examples of these include generative pre-trained (gpt) @xcite , which is a right-to-left language model based on the transformer's decoder architecture @xcite , bert @xcite , bart @xcite and t5 @xcite . these attention-based architectures are capable of generating human-level high-quality text, making them a compelling choice for textual data augmentations. specifically, cbert @xcite improves eda by using bert synonym prediction. additional advanced transformer-based methods control the generation process by providing an existing sample, designated class label, or both. these methods were shown to be beneficial for data augmentation @xcite . however, these methods suffer from several drawbacks: first, they were only shown to be successful on small sized datasets (five samples per class or 1% of the dataset). second, the augmentation process was shown to be error prone as the generated samples do not always preserve the class label of the original data. third, as we show in this work, naïvely using these methods to generate a constant number of samples for each class in the dataset, as done in previous work, does not realize their full potential for improving textual classification tasks. other approaches for data balancing can include weak-labeling of available unlabeled data @xcite , or even active learning @xcite . however, both of these approaches require additional domain data which is not always available. notably, some approaches aim at assuring interpretability of generated samples @xcite . however, balagen takes a different aproach -aiming to improve performance without consideration of textual validity/interpretability of generated sentences as done in @xcite . thus, only class perseverance and ability to contribute to accuracy are considered. to the best of our knowledge, this is the first work to explore the use of transformer-based augmentation techniques directly towards data balancing to improve textual classification tasks. ## method at the cornerstone of our methodology lie the recent controlled text generation methods, capable of synthesizing high quality samples @xcite . we tested the hypothesis whereby enhancing these generation methods with a new balancing technique, which differentially add and remove samples from classes, can result in a significant improvement to classifier accuracy. to overcome the well-known drawback of oversampling via text generation, i.e., class label preservation is not guaranteed @xcite , we employed a weak labeling mechanism which is used to select generated samples that have a high probability of preserving their class label. we further refer to weak labelers simply as labelers. in the rest of this section, we describe the steps of our balagen approach. we refer to the step numbers according to the enumeration in the pseudocode given in algorithm 1 and the schematic flow diagram shown in figure 1 . balancing policy: a balancing policy π(•), generally, aims to reach a specific distribution of the samples among the classes, by adding and removing samples. in step (1) we use policies that determine a band [b low , b high ], which within the set of classes are considered well-represented (w r). consequently, the set of classes smaller than b low are referred to as under-represented (u r) and should be further over-sampled, e.g., via augmentation. classes larger than b high are considered over-represented (or) and will be under-sampled. in the following, let c i be the index of i th class after sorting the classes by their size (i.e., the number of samples) in an ascending order. given that n is the number of classes, |c n | is the size of the largest class. in figure 2 we describe several types of balancing policies supported by balagen. while there may be many approaches to determine the w r band, here we employ the following percentile approach: given the parameters β low and β high , we set b low such that β low % of the classes belong to the u r set and set b high such that β high % of the classes belong to the or set. note that β low + β @xmath0. balancing the train set of the generator and weak-labelers: in step (2) we compose a balanced dataset d s used to train the generator and the labeler(s). the under-sampling method is executed on the or classes targeting the b high threshold, while the oversampling method is executed on the u r classes targeting the b low threshold. this step aims to reduce class biases of the generator and labelers. formally, os and us denote over and under sampling functions, respectively. each accept two parameters: a dataset d to perform on and the threshold b. ## discussion and future work in this work we present balagen, a balancing-viageneration framework. we show that balancing textual datasets via generation is a promising technique. furthermore our analysis reveals that the optimal balancing policy depends on the quality of the generated samples, the weak supervision mechanism applied, and the training of balagen's internal component. i.e., the generator and labelers. in balagen we assume that each sample contributes the same gain to its class accuracy. a possible enhancement of balagen could take into account not only the number of samples in each class, but also their quality. alternatively, balancing policies could also consider class accuracy. additional enhancements for balagen could include employing more advanced under-sampling technique such as data cleaning @xcite , cluster-based under-sampling @xcite , or other distribution based techniques @xcite . balagen can also be used to explore setting β @xmath1. additional enhancements may also include investigating more sophisticated weak labeling ensemble mechanisms. we focused our evaluation on the semantic utterance classification (suc) domain which is characterized by highly imbalanced data. however, it is desirable to validate the applicability of our general balancing approach on other textual domains.