Quantifying the Evaluation of Heuristic Methods for Textual Data Augmentation (2020)

Abstract: Data augmentation has been shown to be effective in providing more training data for machine learning and resulting in more robust classifiers. However, for some problems, there may be multiple augmentation heuristics, and the choices of which one to use may significantly impact the success of the training. In this work, we propose a metric for evaluating augmentation heuristics; specifically, we quantify the extent to which an example is “hard to distinguish” by considering the difference between the distribution of the augmented samples of different classes. Experimenting with multiple heuristics in two prediction tasks (positive/negative sentiment and verbosity/conciseness) validates our claims by revealing the connection between the distribution difference of different classes and the classification accuracy.

## introduction machine learning approaches have been shown to be capable of making accurate predictions in many well-known problem domains with an abundance of training data. this heavy reliance on the availability of the data, however, may hamper the application of machine learning approaches to resourcelimited problem domains, where a sizable training data are not always available. there is a growing body of research on training under resource scarcity, and data augmentation is one of such techniques. it aims to reconcile the data requirement of the machine learning approaches by applying a general (e.g., randomly remove a word) or domain-inspired heuristic (e.g., replace an adjective with an antonym) to the (limited) existing data in order to generate more training samples. one challenge for data augmentation is in choosing the most appropriate heuristic for the application in question. there may be many domainindependent augmentation heuristics, and a domain expert may come up with many different domaininspired heuristics; but the choices of which examples from these heuristics to use may have a significant impact on the success of the trained model. a straightforward approach to choose an augmentation heuristic is to actually perform the classification experiment on all possible augmented datasets and then chose the best performing one(s) based on the evaluative results. however, this approach may not be computationally practical when there are too many augmentation heuristic options. in this paper, we propose an alternative heuristic evaluation approach based on the idea that a good heuristic should aim to generate "hard to distinguish" samples for different classes. we further argue that the generation quality of "hard to distinguish" examples could be quantified as the difference between the distribution of the augmented samples that a heuristic generates for different classes. to calculate the distribution difference, we proposed to use pre-trained off-the-shelf embeddings to convert sentences into class distributions, then calculate the kl-divergence between them and used that as a metric to evaluate the "hard to distinguish" examples that a heuristic produces. we validate our proposed heuristic evaluation approach by experimenting with multiple heuristics and augmented datasets for two classification tasks: predicting whether a sentence expresses positive or negative sentiment and predicting whether a sentence is verbose or concise. results suggest that quantifying the "hard to distinguish" example generation quality of the heuristics as the difference between class distribution of the augmented examples, could be served as an effective metric for choosing a suitable augmented dataset for a classification task. ## data augmentation data augmentation is a technique for generating additional training data by applying a heuristic transformation to the existing training examples. for example, an existing image could by rescaled or flipped to get more images with the same label to expand the size and diversity of the training dataset and thus train a more reliable and accurate model @xcite @xcite . in general, data augmentation could be formulated as equation 1, where h is a heuristic function that transforms the datapoint and label pair of (x, y) to a new augmented sample (x, ŷ). the majority of existing data augmentation approaches are label-preserving, which relaxes the equation 1 as (x, y) = h(x, y) = (h(x), y); this means, if x belong to some class a, augmented x also belong to class a. for example, using a synonym replacement heuristic, a sentence with positive sentiment could be augmented into a new example, while preserving the overall positive sentiment. label-preserving data augmentation requires existing labeled samples for every class that is needed to be augmented. data augmentation can be non-label-preserving as well, where the label itself might also transform using function h y that expands equation 1 as: this means, while x belongs to class a, x might not necessarily belong to class a. for example, by replacing the most positive word(s) of a sentence with positive sentiment with an antonym, the sentence's sentiment may become negative. non-labelpreserving data augmentation is not bound to the assumption of having labeled samples for instances of all classes and samples from one class may be enough to generate instances of other classes. given a classification task, there may be multiple heuristics and data augmentation approaches that allow us to transform existing samples to new ones, but the choice of heuristic may significantly impact the success of the task. in this paper, we aim to answer the key question: "which heuristic and data augmentation approach is more appropriate for a classification task?" in section 3, we propose a low-cost approach to quantify the evaluation of different heuristics and the resulting augmented datasets for classification tasks. we believe our proposed approach could be a contribution to the nlp community because data augmentation has been shown to be useful for many nlp applications, with researchers proposing many different approaches for text data augmentation; for example, @xcite @xcite @xcite @xcite used embedding-based lexical substitution approach, @xcite used random noise injection, including random word insertion, deletion, or sentence shuffling, (luque, 2019) used instance crossover by combining halves of tweets, @xcite adapt the mixup approach @xcite ) to text by interpolating the distributed representation of different sentences, @xcite @xcite used back-translation, and @xcite @xcite ) used (deep) generative models to augment more training examples. however, with all these textual augmentation options, trying all of them for a (classifier) training task might be impractical, and to our best knowledge, there is not a guideline for how to choose between them for a task. ## quantification of heuristics suitability a straightforward approach to assess which heuristic and data augmentation approach is more appropriate for the task is to try every heuristic to generate an augmented dataset, then train a classifier on each and check the final classification performance @xcite . the training process in this brute-force approach, however, may be time-consuming and resource-intensive, especially in complex training scenarios. alternatively, we may try to identify qualities that make a heuristic effective. intuitively, a good heuristic ought to generate augmented samples that are the most similar to the original data distribution. however, this approach may overlook the additional generalization benefit that may come from diverse augmented training examples. moreover, this approach may not be possible for problem domains with limited resources, where original labeled data is not available for all classes, and one may have to use non-label-preserving heuristics to augment examples for all classes. on the other hand, from the classification task perspective, a good heuristic should aim to generate near-miss examples (samples of class b hard to distinguish from a). we believe, the "hard to distinguish" samples can be quantified by finding a way to compute the difference between the samples of different classes, to sever as an guideline for choosing between different heuristic approaches. let us assume samples of class a are drawn from distribution a, which should be different from distribution b that samples of class b are drawn from. the difference between distribution a and b can be calculated as the kl-divergence (kld) @xcite ) from b to a as: d kl (a||b). kld calculates how probability distribution a is different from the reference probability distribution b as the amount of information gained if samples of b are used instead of samples of a. thus, a lower d kl (a||b) means distribution a is more similar to distribution b, so samples of class a are harder to distinguish from samples of class b. therefore, the extent to which "hard to distinguish" samples can be generated by heuristic h could be quantified as d kl (a h ||b h ), where a h and b h indicate the samples of class a and b augmented using heuristic h, and equation 2 could be used to identify which heuristic is generating "harder to distinguish" samples and so more suitable for the classification task. finally, to transform sentences from their discrete word representation into a continuous distribution representation, we utilize a few of the numerous pre-trained embeddings that nowadays are the de facto approach for encoding sentences into vector space @xcite @xcite . we examine the applicability of our proposed approach by studying two classification tasks: sentiment analysis, as a resource-rich problem domain that allows experimenting with both labelpreserving and non-label-preserving heuristics, and verbosity analysis, as a resource-limited problem domain that the absence of sizable labeled data limits the options to non-label-preserving heuristics. ## augmented datasets in this section, we go over some heuristic options for augmenting training corpora for sentiment analysis and verbosity detection domains. ## experiments the key questions for validating our proposed approach for quantifying the evaluation of heuristic textual data augmentation methods are: to measure the accuracy of sentiment and verbosity classification in answering q1, we trained an lstm @xcite and a cnn @xcite classifier on each the augmented dataset. the classification result for each task and augmented dataset is reported in section 5.2. the lstm and cnn models are trained on augmented corpora separately for each task; the sentiment classifiers are evaluated on a held-out portion of the ypd, and the verbosity classifiers are evaluated on spc. none of the sentences of the held-out ypd and spc are used during the creation of the augmented datasets. to answer q2, we use two pre-trained encoder models: universal sentence encoder (use) @xcite and bidirectional encoder representations from transformers (bert) @xcite , both of which are transformer-based encoder of greater-than-word length text, to transform the sentences into a continuous space so that we can treat them as class distributions and measure their similarity. ## conclusion this paper presents an approach for evaluating the suitability of augmentation heuristics for classifications task via "hard to distinguish" example generation capacity of the heuristics through analyzing the difference of class distribution of the augmented examples. experimental results suggest our proposed heuristic evaluation approach could be a low-cost yet effective way of measuring the suitability of an augmented heuristic for a classification task.