Data Augmentation for Text Generation Without Any Augmented Data (2021)

Abstract: Data augmentation is an effective way to improve the performance of many neural text generation models. However, current data augmentation methods need to define or choose proper data mapping functions that map the original samples into the augmented samples. In this work, we derive an objective to formulate the problem of data augmentation on text generation tasks without any use of augmented data constructed by specific mapping functions. Our proposed objective can be efficiently optimized and applied to popular loss functions on text generation tasks with a convergence rate guarantee. Experiments on five datasets of two text generation tasks show that our approach can approximate or even surpass popular data augmentation methods.

## introduction end-to-end neural models are generally trained in a data-driven paradigm. many researchers have proposed powerful network structures to fit training data well. it has also become ubiquitous to increase the training data amount to improve model performance. data augmentation is an effective technique to create additional samples in both vision and text classification tasks @xcite @xcite , which perturb samples without changing their labels. for text generation tasks, there can be more types of data perturbation to construct augmented samples, including corrupting the input text @xcite , the output text @xcite , or both @xcite . as such, classification tasks can be regarded as special cases of generation tasks in terms of incorporating data augmentation techniques, and this work mainly discusses text generation tasks. the focus of previous work on text data augmentation has been to design proper augmentation techniques to create augmented samples. some augmentation methods have been proposed for general text tasks. for example, different general replacement operations have been explored to edit words in a text sample, ranging from simple look-up tables @xcite to pretrained masked language models @xcite . @xcite propose to augment text sequences by back-translation. for some generation tasks such as dialogue generation, general augmentation methods may not yield stable improvements and it requires to carefully incorporate the task property to design useful augmented samples @xcite . all these methods need to explicitly construct augmented samples, and the data mapping functions from the original samples to the augmented samples are mostly defined apriori. this motivates us to raise a question, whether we can skip the step to define or choose proper augmented data mapping functions to accomplish effective data augmentation. to answer this question, we aim to formulate the problem of data augmentation for general text generation models without any use of augmented data mapping functions. we start from a conventional data augmentation objective, which is a weighted combination of loss functions associated with the original and augmented samples. we show that the loss parts of the augmented samples can be re-parameterized by variables not dependent on the augmented data mapping functions, if a simple euclidean loss function between the sentence representations is applied. based on this observation, we propose to directly define a distribution on the re-parameterized variables. then we optimize the expectation of the augmented loss parts over this distribution to approximate the original augmented loss parts computed with various augmented data mapping functions. we make different assumptions on the variable distributions and find that our proposed objective can be computed and optimized efficiently by simple gradient weighting. if stochastic gradient descent (sgd) is used, our objective is guaranteed with the convergence rate o(1/ √ t ). our objective can be coupled with popular loss functions on text generation tasks, including the word mover's distance @xcite and the cross-entropy loss. our approach, which utilizes the proposed objective and optimizes it by sgd, has two advantages. first, it provides a unified formulation of various data perturbation types in general text generation models, which sheds a light on understanding the working mechanism of data augmentation. second, the optimization of our approach is simple and efficient. without introducing any new sample during training, we can avoid additional calculation efforts on augmented samples, often with the total size much larger than the original data size. hence, our approach maintains high training efficiency. extensive experiments are conducted to validate the effectiveness of our approach. we mainly use the lstm-based network structure @xcite and perform experiments on two text generation tasks -neural machine translation and single-turn conversational response generation. results on five datasets demonstrate that the proposed approach can approximate or even surpass popular data augmentation methods such as masked language model @xcite and back-translation @xcite . ## related work data augmentation has shown promising improvements on neural models for different text generation tasks such as language modeling @xcite , machine translation @xcite and dialogue generation @xcite . existing text data augmentation methods can be mainly categorized into word-level augmentation and sentence-level augmentation. word-level augmentation methods perturb words within the original sentence. common operations include word insertion and deletion @xcite , synonym replacement @xcite , and embedding mix-up @xcite . masked language models can be used by masking some percentages of tokens at random, and predicting the masked words based on its context @xcite . sentence-level data augmentation is not limited to edit only a few words in the original sentence, but to generate a complete sentence. for example, back-translation is originally proposed to translate monolingual target language data into source language to augment training pairs in machine translation @xcite . it is later extended to paraphrase sentences in any text dataset, in which two translation models are applied: one translation model from the source language to target language and another from the target to the source. ganbased and vae-based models have also achieved impressive results to create entire sentences to augment the training data @xcite . for dialogue generation, retrieved sentences can be good supplement of the original corpus @xcite . both word-level and sentence-level augmentation methods need to define their augmented data mapping functions (i.e. operations to edit words or models to generate sentences) apriori. some works train policies to sample a set of word-level operations @xcite , but the operation candidates are still pre-defined. a few works learn to construct augmented samples and optimize the network jointly @xcite . different from previous work, our goal is not to propose or learn novel augmented data mapping functions. instead, we investigate whether the effectiveness of data augmentation can be achieved while we do not bother to use any specific augmented data mapping function. besides data augmentation, data weighting is another useful way to improve model learning. it assigns a weight to each sample to adapt its importance during training. the sample weights are often carefully defined @xcite or learnt by another network @xcite . data augmentation is often combined with data weighting together to weight the original and augmented samples. ## background we are given original samples @xmath0, y)} with x, y both as text sequences. without loss of generality, a deep generation model is to learn a mapping function f x,y by a deep neural network that outputs y given x. as mentioned in the introduction, text generation tasks mainly have three types of augmented data: let ℓ(f x,y (x), y) denote the loss function to be minimized for each sample. we first use augmented data in the input domain as an example to present the problem formulation and introduce our approach, then later discuss other types of augmented data. data augmentation methods generally apply an augmented loss per sample with its augmented samples: ## our approach in this section, we aim to formulate the problem of data augmentation for general text generation models without any use of augmented data mapping functions. we introduce our approach by assuming that the loss function ℓ is the most simple euclidean distance, i.e. ## loss function in theory, our approach can be applied to any lipschitz smooth loss function that holds the equation (3). in this section, we show another valid loss function in our approach -the word mover's distance (wmd) @xcite , which is previously used in various text generation tasks. next, we discuss the cross entropy loss, in which the proposed objective is not an upper-bound of the data augmentation objective. however, our approach can still converge with the same convergence rate and experimental results in the next section validate the effectiveness of our approach with the cross-entropy loss. ## experiments the proposed approach provides a new paradigm and understanding of data augmentation for text generation. to evaluate that our approach can mimic the effect of data augmentation, we conduct experiments on two text generation tasksneural machine translation and conversational response generation. we compare our approach with two most popular data augmentation methods (one token-level and one sentence-level augmentation method) that can be applied on various text generation tasks: • masked language model (mlm): we use a pretrained bert @xcite and randomly choose 15% of the words for each sentence. bert takes in these masked words to predict these masked positions with new words. we augment one sample from each original training sample. thus the data size increases to twice of the original one. note that we only augment the english side of translation datasets. we set the same weight w of all augmented loss parts used in ℓ aug as a hyper-parameter, and tune it on the development set of each dataset. since euclidean distance is a special case of wmd as discussed in sec 5.1, we show results of all methods with the use of the cross-entropy loss and wd. we mainly use the fairseq @xcite seq2seq implementation as our model. both encoder and decoder are one-layer lstm. the word embedding dimension is 256. attention @xcite is used with a dropout rate of 0.1. all parameters are randomly initialized based on the uniform distribution [-0.1, +0.1]. we use sgd to optimize our models, and the learning rate is started with 1.0. after 8 epochs, we start to halve the learning rate after each epoch. all experiments are run on a single nvidia v100 gpu. code for our experiments are available once our work is accepted. ## conclusions and future work we have proposed an objective of formulating data augmentation without any use of any augmented data mapping function. we show its optimization and provide the corresponding convergence rate. both the l 2 wasserstein distance and the crossentropy loss are discussed with their use in our objective and their corresponding theoretical guarantees. different from previous data augmentation works that need to add manipulated data into the training process, our gradient based approach provides a potential way to obtain performance improvements, which may come from augmented data, without incurring the computational expense. experiments on both neural machine translation and conversational response generation validate the effectiveness of our objective compared to existing popular data augmentation methods: masked language models and back-translation. we believe this work provides a new understanding of data augmentation. our approach can also be useful to a wide range of tasks including text classification tasks, which can be seen as special cases of text generation tasks, and cross-modality generation tasks such as image captioning, in which we can skip the step to use various image augmentation techniques. we would like to point out that some parts of our approach can be improved in the future, which may lead to a better performance and generalization. firstly, current distributions we choose in the re-parameterized loss are relatively simple. some points under current continuous distributions may not correspond to valid text sequences in the original text space, due to the discreteness of natural languages. a possible way is that we change to leverage more informative distributions, such as including prior distributions computed from several augmented samples. secondly, our method is derived under the framework of sgd and it is possible to extend it to the adam framework @xcite @xcite . we also leave the more general version of our work in the future. definition 2. a point θ is called -accurate if ||∇ℓ(θ)|| 2 ≤ . a stochastic iterative algorithm is said to achieve -accuracy in t iterations if e[||∇ℓ(θ t )|| 2 ] ≤ , where the expectation is over the stochasticity of the algorithm. definition 3. we say ℓ ∈ f n has σ-bounded gradients if ||∇ℓ i (θ)|| ≤ σ for all @xmath1. let α t denote the learning rate at iteration t, and w it be the gradient weight assigned to sample i by our approach. by sgd, we have definition 4. we say the positive gradient weight w in our approach is bounded if there exist constants w 1 and w 2 such that w 1 ≤ w @xmath2.