Good-Enough Example Extrapolation (2021)

Abstract: This paper asks whether extrapolating the hidden space distribution of text examples from one class onto another is a valid inductive bias for data augmentation. To operationalize this question, I propose a simple data augmentation protocol called “good-enough example extrapolation” (GE3). GE3 is lightweight and has no hyperparameters. Applied to three text classification datasets for various data imbalance scenarios, GE3 improves performance more than upsampling and other hidden-space data augmentation methods.

## introduction text classification is a fundamental task in nlp for which modern architectures have achieved high performance when training data is sufficient @xcite . in many applied settings where data collection and annotation is limited, however, a common challenge is data imbalance @xcite , in which training data from certain categories is scarce. a classic example of such a scenario is intent classification, where developers may wish to update a conversational agent to be able to classify new intents, but the amount of training data for these new intents lags behind that of existing ones @xcite . one common method for mitigating the weaknesses of limited training data is data augmentation, a paradigm that is become increasingly seductive in the nlp landscape (see @xcite . while data augmentation may be of only incremental utility when training data is sufficient @xcite , it can be particularly helpful for mitigating data scarcity in low-resource settings such as few-shot classification @xcite or, as this paper will soon explore, data-imbalanced text classification. in this paper, i propose a simple data augmentation protocol called good-enough foot_0 example extrapolation (ge3) for the class-imbalanced scenario. as shown in figure 1, ge3 extrapolates the hidden-space distribution of examples from one class onto another class. ge3 has no hyperparameters, is model-agnostic, and requires little computational overhead, making it easy-to-use. in empirical experiments, i apply ge3 to intent classification, newspaper headline classification, and relation classification, finding that in a variety of class-imbalanced scenarios, ge3 substantially outperforms upsampling and other hidden state augmentation techniques. so for a given classification task, inputs from the same category will have some distribution (i.e., cluster) in hidden space, where the distance between distributions represents the relationship between categories, and the distribution of points within the same category models some random variable. ge3 leverages this intuition by extrapolating the distribution of points in the same category, which models some random variable, from one category onto another. figure 2 illustrates this intuition in a hypothetical instance from the huffpost dataset of news headlines classification, where the hidden-space relationship between two examples in the travel category can be extrapolated to form a new example in the health category. method. formally, i describe ge3 as follows. given a text classification task with k output classes {c j } k @xmath0, denote the n c training examples in class c as {x i c } nc @xmath1. for each class c, let µ(c) = 1 nc nc @xmath2. ge3 generates augmented examples by extrapolating the data distribution from a source class c s to a target class c t in hidden space. and so for each hidden space representation x i cs in the source class, i generate a corresponding augmented example xct in the target class: in total, for each class in the training set, i can generate a set of extrapolated points from every other class, augmenting the size of the original training set by a factor of k. i then train the classification model on the union of original data and extrapolated examples. notably, this extrapolation method operates without any hyperparameters, as augmented examples are generated via distributions from other classes instead of a noising function (c.f. augmentation techniques that usually have a strength parameter @xcite ). ## experimental setup i evaluate the proposed hidden space extrapolation protocol in several data imbalance scenarios on three diverse text classification datasets. ## results finally, one of the reasons that ge3 improves performance more than other techniques could be that each class gets extrapolated examples from every other class, and if classes have unique distributions, then these extrapolated examples are valuable as additional training data. therefore, i perform an ablation study using a variable n aug , which restricts the number of other classes a given class can receive extrapolated examples from. for instance, if n @xmath3, then any given class may only receive extrapolated examples from two other random classes, even if there are 63 other classes (as is the case in fewrel). i also perform a similar ablation for gaussian and uniform noise, in which i generate augmented examples until each class has n aug • n many training examples. figure 3 shows these results. for uniform and gaussian noise, additional augmented examples did not further improve performance after around n @xmath4. for ge3, on the other hand, improvement continued to increase as n aug increased (although the marginal improvement decreases for each n aug ). this result confirms the intuition that extrapolations from more classes provided additional value during training. hidden space augmentation. a growing direction in data augmentation has proposed to augment data in hidden space instead of at the input-level. in computer vision, devries and taylor (2017) explored noising, interpolation, and extrapolation, and mixup @xcite combines pairs of examples. these methods have since been adopted to nlp-chen et al. (2020) modify mixup to improve semi-supervised text-classification, and @xcite explore various hidden space augmentation techniques for few-shot intent classification, which i evaluated as baselines. whereas the extrapolation technique used by @xcite and @xcite (which i call "within-extrapolation" in this paper) extrapolates the hidden space between a pair of points in the hidden space, ge3 extrapolates the hidden space distribution of one class onto another class. example extrapolation. in vision, @xcite used a modified auto-encoder to synthesize new examples from category after seeing a few examples from it, improving few-shot object recognition. perhaps most similar to this work, @xcite train t5 @xcite to, given some examples of a class as an input sequence, generate additional examples. because ge3 operates in hidden space, it is simpler and more computationally accessible compared with fine-tuning t5 for each classification task. ## discussion the motivation for this work emerged from a mixture of failed experiments (i tried to devise an algorithm to select better augmented sentences in hidden space) in addition to an admiration for the elegance of the ex2 @xcite . in hindsight, it would have been helpful to compare the performances of these two techniques in the same setting (notably, whereas i artificially restrict the sample size for certain classes in this paper, ex2 uses the original data distributions of the datasets, which is a harder setting to show improvements from data augmentation). i would be remiss not to mention at least one weakness that i see in my own work. there has been an influx of recent work proposing various augmentation techniques for different nlp tasks, and due to the lack of standardized evaluation datasets and models, many papers foot_3 do not perform a full comparison with respect to relevant baselines. this paper circumvents comparing with many data augmentation baselines (e.g., @xcite ) by focusing on the question of whether hiddenspace example extrapolation is a valid inductive bias (and not whether it is the best augmentation technique). hence, although i find example extrapolation to be a nice idea, i should concede that the particular ge3 operationalization of example extrapolation should undergo more comprehensive comparison with baselines before i can recommend it as a go-to augmentation technique. in summary, i have proposed a data augmentation protocol called ge3, which extrapolates the hidden space distribution of one class onto another. the empirical experiments in this paper suggests that example extrapolation in hidden space is a valid inductive bias for data augmentation. moreover, ge3 is appealing because it has no hyperparameters, is model agnostic, and is lightweight. if example extrapolation is an idea deserving of further exploration by our field, i hope this paper adds a leaf to the tree of knowledge in that space.