Data Augmentation for Multiclass Utterance Classification – A Systematic Study (2020)

Abstract: Utterance classification is a key component in many conversational systems. However, classifying real-world user utterances is challenging, as people may express their ideas and thoughts in manifold ways, and the amount of training data for some categories may be fairly limited, resulting in imbalanced data distributions. To alleviate these issues, we conduct a comprehensive survey regarding data augmentation approaches for text classification, including simple random resampling, word-level transformations, and neural text generation to cope with imbalanced data. Our experiments focus on multi-class datasets with a large number of data samples, which has not been systematically studied in previous work. The results show that the effectiveness of different data augmentation schemes depends on the nature of the dataset under consideration.

## introduction in modern conversational systems, classifying incoming user utterances is among the most crucial processes. this is particularly evident for automated customer service systems: if the underlying demand can successfully be classified based on the user's description, a known solution can directly be provided to them. a weak classifier may miscategorize a request, resulting in customer dissatisfaction. a considerable cause for low-performing classification is a lack of sufficient training data for certain categories, which manifests as the problem of imbalanced data distributions. classifying utterances is particularly challenging, as people may choose various different forms of expressing their ideas and thoughts. thus, very different utterances may reflect the same underlying intent. recent findings on query variation in search systems suggest substantial potential for considering language diversity in nlp applications @xcite @xcite . motivated by these observations, we study how to improve utterance classification results by drawing on utterance variation. unfortunately, soliciting clean human-generated data can be expensive and difficult to scale. in this study, we instead consider automated utterance generation schemes to augment the original dataset during the training process, in order to: (1) mitigate the data imbalance, and (2) improve the classification effectiveness. since automatically generated data is cheap and easy to obtain, we can use it as a way of augmenting existing data, which may improve our classification model's effectiveness and robustness. in this study, we conduct a thorough investigation of current data augmentation approaches to mitigate the imbalanced data problem, including simple resampling, word-level transformations, and neural text generation. subsequently, an optimal balanced spot is sought to achieve a better classification result. contributions: the main contributions of this paper are twofold. first, we conduct a comprehensive survey regarding state-of-the-art data augmentation approaches for multi-class text classification. second, our extensive empirical analysis compares the effectiveness of such data augmentation approaches, showing the value of adopting variational autoencoding techniques along with hybrid combinations of oversampling and undersampling. ## data augmentation strategies empirically, it is well-known that the scarcity of data can hamper the effectiveness of machine learning models, particularly data-hungry deep neural networks. hence, data augmentation techniques have become ubiquitous in certain fields to alleviate this issue. a substantial number of studies have focused on applying data augmentation to facilitate classification tasks in computer vision. in the vision domain, very simple data augmentation tricks are able to engender significant performance gains. some operations, such as rotation, translation, and flipping have come to be routinely invoked for both handwritten document recognition @xcite as well as regular image classification tasks @xcite . in addition to these macroscopic operations, @xcite further leverage rgb channel intensity alterations to reduce overfitting caused by data insufficiency. for speech recognition tasks, data augmentation is as well commonly invoked to train more robust models, and is mainly applied at the audio signal level @xcite . unlike images and speech, text cannot naturally be regarded as a continuous signal that can be perturbed arbitrarily, given its discrete units of semantic meaning. to faithfully retain these features, paraphrases are an intuitive and in some sense ideal way to expand a dataset by incorporating alternative expressions for the existing sentences. however, human rephrasing is too expensive and unrealistic, whereas machine paraphrasing currently has its limitations, e.g., it only works on specific tasks @xcite and a specific paraphrase corpus may be required @xcite . to cope with the generalization issue of data augmentation, a number of universal approaches have been proposed, and we thoroughly evaluate several such universal data augmentation approaches, which can be classified into three types: simple re-sampling, wordlevel transformations, and neural text generation. in the following, we explicitly introduce each method, and for vae-based neural generation modeling, we additionally introduce some new adaptions to facilitate the task of learning a classification. ## experiments in this section, we describe the datasets used in the experiments, followed by an elaboration of our experimental setup and pertinent implementation details. subsequently, we provide our quantitative experimental results as well as a detailed analysis. ## conclusion this paper presents a survey and systematic experimental framework to investigate state-ofthe-art data augmentation schemes for text classification, considering both utterance classification and ordinary multi-class text categorization. we carried out a thorough set of experiments to compare the effectiveness of different strategies. our results highlight the potential of using recent neural generative models as a method to facilitate classification for large datasets. in particular, we vae methods were found to be comparable to if not better than previous stateof-the-art approaches such as eda. our experiments further show that an optimal balanced spot is able to further improve the classification results. finally, based on our detailed analyses regarding multiclass imbalance, we argue that the imbalance issue cannot be reduced to merely considering the relative imbalance in the number of data samples. rather, more focus should be placed on the absolute counts and the feature representations within each class.