D eep ADA :An Attention-Based Deep Learning Framework for Augmenting Imbalanced Textual Datasets (2022)

Abstract: In this paper, we present an attention-based deep learning framework, DeepADA, which uses data augmentation to address the class imbalance problem in textual datasets. The proposed framework carries out the following functions:(i) using MPNET-based embeddings to extract keywords out of documents from the minority class, (ii) making use of a CNN-BiLSTM architecture with parallel attention to learn the important contextual words associated with the minority class documents’ keywords and provide them with word-level characteristics derived from their statistical and semantic features, (iii) using MPNET, replacing the key contextual terms derived from the oversampled documents that match to a keyword with the contextual term that best fits the context, and finally (iv) oversampling the minority class dataset to produce a balanced dataset. Using a 2-layer stacked BiLSTM classifier, we assess the efficacy of the proposed framework using the original and oversampled versions of three Amazon’s reviews datasets. We contrast the proposed data augmentation approach with two state-of-the-art text data augmentation methods. The experimental results reveal that our method produces an oversampled dataset that is more useful and helps the classifier perform better than the other two state-of-the-art methods. Nevertheless, we discover that the oversampled datasets outperformed their original ones by a wide margin.

## introduction textual datasets from different domains generally suffer from the issue of class imbalance, where instances from the majority class outnumber the instances from the minority class by a huge margin. in such a case, the classifier cannot perform well on the minority class dataset; as a result, the minority class instances go undetected. most research works handle this issue by simply using a random oversampling algorithm without injecting additional knowledge into the minority class dataset. as a result, the resulting model is highly susceptible to overfitting. nevertheless, a data augmentationbased minority class oversampling approach to handle a class imbalance in a textual dataset has not been well investigated. generally, the augmentation techniques generate undesirable documents that do not share the similar distribution of syntax, semantics, and pragmatics of the original data. when it comes to text data augmentation, class-indicating words (keywords) play a significant role @xcite . we must selectively augment the text, considering the significance of different words. in this process -(i) we should be able to identify the important class-indicating words so that the newly generated document semantically revolves around the class of the original document, (ii) we should be able to characterize the important words that aid the identification class indicating words and ultimately improves the generalization ability. deep learning models have emerged as effective classification models and are successful in many domains @xcite @xcite , and have robust pattern learning ability and are widely successful for classification tasks. this paper presents a deep learning-based text data augmentation approach that exploits different deep learning techniques to create a balanced dataset by augmenting the newly generated documents to the minority class dataset to improve the detection efficacy of the classification algorithms on the minority class. the proposed approach first identifies keywords from the minority class data points utilizing a ranking-based weighted approach. using an attention mechanism, it exploits the identified keywords to extract important contextual words from minority-class documents. in this process, it recognizes the word roles using statistical correlation to measure word occurrence frequencies respective to text categories and semantic similarity to measure word semantics respective to text categories, which helps to find the words semantically similar to text labels. finally, it utilizes important contextual words to enrich the minority class dataset. the proposed approach oversamples the minority class dataset by generating new documents based on important contextual words and augmenting them to the minority class dataset. the proposed approach seems interpretable and improves the performance of the deep learning classifiers over the augmented datasets. the rest of the paper is organized as follows. section 2 presents a brief review of the existing literature on text data augmentation. section 3 presents a detailed description of the proposed attentionbased text data augmentation approach. section 4 presents the experimental setup and evaluation results. it also presents a comparative analysis of the proposed approach with two state-of-the-art text data augmentation approaches. finally, section 5 concludes the paper with future research directions. ## related works researchers have come up with many approaches including words substitution-based @xcite @xcite , paraphrasing-based @xcite , and text generation-based @xcite solutions. in @xcite , authors introduced easy data augmentation (eda), a widely used wordreplacement based data augmentation method with four basic randomization operations -(i) replacement, (ii) insertion, (iii) swap, and (iv) deletion. they have visualized that these simple operations can improve a classifier's performance on text classification tasks. in @xcite , authors proposed contextual augmentation for labeled sentences to predict words from a wide range of substitute words, learned using a label-conditional bidirectional language model. in @xcite , authors showed that augmenting n-grams from a minority class document to itself that includes minority class keywords using latent dirichlet allocation (lda), if any, in the document, can improve the cnn's ability to identify the minority class instances. in @xcite , authors proposed to identify substitute words according to their context, which, apart from the similar meaning, also cares whether the candidates fit in the surrounding context and labels. in @xcite , authors exploited data augmentation to automatically create more labeled training data to fine-tune a language model to derive each aspectopinion pair's sentiment. in @xcite , authors insisted on creating new linguistic patterns for text data augmentation using large pre-trained language models. ## proposed attention-based deep learning framework in this section, we discuss the proposed attentionbased deep learning framework, deepada, for data augmentation to improve the performance of classifiers on imbalanced text datasets. we evaluate the proposed approach over 3 amazon reviews dataset, with statistics as shown in ## conclusion and future work this paper presents a deep learning-based text data augmentation approach, deepada, to address the class imbalance issue of classifying textual datasets. the oversampled dataset generated using deepada can be helpful for deep learning models that extract patterns from the data. experiments on different datasets show that deepada significantly outperforms the state-of-the-art methods. the ablation study in the form of the baseline deepada b reveals that statistical correlation and semantic similarity are essential for effective word selection. the performance observed on the minority class dataset, and the macro-averaged performance over the 3 datasets indicates that the classifier acquires stronger generalization ability when trained on oversampled datasets generated using deepada. exploring more word-level features and extensive study on various transformer-based language models to generate more qualitative oversampled datasets seems a promising future direction of research.