When Chosen Wisely, More Data Is What You Need: A Universal Sample-Efficient Strategy For Data Augmentation (2022)

Abstract: Data Augmentation (DA) is known to improve the generalizability of deep neural networks. Most existing DA techniques naively add a certain number of augmented samples without considering the quality and the added computational cost of these samples. To tackle this problem, a common strategy, adopted by several state-of-the-art DA methods, is to adaptively generate or re-weight augmented samples with respect to the task objective during training. However, these adaptive DA methods: (1) are computationally expensive and not sample-efficient, and (2) are designed merely for a specific setting. In this work, we present a universal DA technique, called Glitter, to overcome both issues. Glitter can be plugged into any DA method, making training sample-efficient without sacrificing performance. From a pre-generated pool of augmented samples, Glitter adaptively selects a subset of worst-case samples with maximal loss, analogous to adversarial DA. Without altering the training strategy, the task objective can be optimized on the selected subset. Our thorough experiments on the GLUE benchmark, SQuAD, and HellaSwag in three widely used training setups including consistency training, self-distillation and knowledge distillation reveal that Glitter is substantially faster to train and achieves a competitive performance, compared to strong baselines.

## introduction the undeniable importance of data in deep learning @xcite and the costly process of data annotation has propelled researchers into leveraging data augmentation (da) in a broad range of applications from computer vision @xcite to natural language processing (nlp) including machine translation @xcite , language understanding @xcite @xcite , and question answering @xcite @xcite . da is shown to be effective in improving generalization of deep neural networks @xcite and in increasing the number of training samples especially in low resource data regimes @xcite . nonetheless, in nlp, the discrete nature of text poses additional complexity to da as generating semantically viable text from another text is challenging @xcite . da methods can be broadly categorized into task-aware and task-agnostic methods. taskagnostic da methods essentially generate augmented text regardless of the task at hand and often do not warrant additional training or fine-tuning. they can be based on some hand-crafted heuristics @xcite , backtranslation @xcite , or token replacement from a pre-trained language model @xcite @xcite . even though deploying task-agnostic methods is straightforward, these methods do not take into account any task-specific information, and thus, their performance is usually limited. on the other hand, task-aware da methods are capable of generating augmented samples, conditioned on the downstream task objective @xcite @xcite . these methods adapt augmented examples specifically for a task in that they construct augmented examples, sometimes partly, during training. despite their advantages, they often incur additional training costs, resulting in a prohibitively slow and a computationally expensive training. in general, the central problems surrounding da techniques in nlp can be summarized as follows: first, da methods are mostly not sample-efficient in that they add arbitrary number of augmented samples to the training data and naively incorporate all of them into training without investigating how many of augmented samples are actually needed. second, although more effective, taskaware methods are notoriously time-consuming to train. this is especially problematic in large-scale datasets such as squad @xcite and mnli @xcite . third, most da methods are not universal as they work solely with a particular setup-e.g., training a singlenetwork @xcite , or training in teacherstudent settings @xcite . overall, the importance of both sample efficiency and training efficiency for da has been often overlooked. motivated by the above problems, in this work, we introduce a universal da method, glitter foot_0 , which can be plugged into any da method to make them sample-efficient, and task-aware without sacrificing performance. specifically, given a pool of augmented samples that are generated offline, our proposed method follows a minimax approach @xcite to select a small subset with maximal expected loss (maximization step) during training. without any further adjustments to the training algorithm, the task objective can be optimized for this selected subset (minimization step). our key contributions in this paper can be summarized as follows: ## methodology in this section, we introduce our task-aware da method, glitter , that aims at using an efficient number of augmented samples without sacrificing performance. our proposed strategy is agnostic to da methods; it can be seamlessly plugged into any da method with any training setting to enforce sample efficiency. existing learning-based da methods train a separate da model and adapt its output for a particular objective function that is entirely task-dependent: in contrast to learning-based da, we propose to generate many augmented candidates using any arbitrary da method prior training, and adaptively select most suitable candidates during training. this procedure does not introduce additional trainable parameters into training, and more importantly, is capable of automatically ignoring unnecessary augmented examples. let (x i , y i ) n @xmath0, y)} represent training data such that a pair x @xmath1. suppose a pool of k augmented examples, x (i) = {x k (i)} k @xmath2, are sampled from some da model for each training example (x i , y i ) ∈ (x , y). note that glitter imposes no restrictions on how to augment training data; augmented samples can be generated via a single or even multiple da models. ## ablation study and discussion in this section, we aim to answer the following questions: for this purpose, we conduct a detailed analysis on 4 glue tasks-i.e., sst, mrpc, qnli, and rte. we trained models based on vanilla-da and glitter using self-kd and tested them on the development set (the dev setting). ## conclusion in this work, we proposed a universal da technique, namely glitter, that can be freely applied to any da technique to enforce sample efficiency without introducing additional parameters or changing the training procedure. we extensively evaluated glitter on a broad range of nlu tasks and in various widely used settings including consistency training, self-distillation and knowledge distillation and demonstrated substantial efficiency gains without compromising effectiveness. extending glitter to auto-regressive models for machine translation and abstractive summarization is an interesting direction for future work.