G en A ug: Data Augmentation for Finetuning Text Generators (2020)

Abstract: In this paper, we investigate data augmentation for text generation, which we call GenAug. Text generation and language modeling are important tasks within natural language processing, and are especially challenging for low-data regimes. We propose and evaluate various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. We also examine the relationship between the amount of augmentation and the quality of the generated text. We utilize several metrics that evaluate important aspects of the generated text including its diversity and fluency. Our experiments demonstrate that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods, and that the quality of generations improves to a peak at approximately three times the amount of original data.

## introduction text generation is an important but difficult task within natural language processing (nlp). a major goal is for dialogue agents to generate human-like text. the development of strong pretrained text generators like gpt-2 (radford et al., 2019) has made it easier to perform generation for new domains or task specifications. these models are typically finetuned on downstream tasks such as classification; however, the first stage of their training is language modeling. effective language models are important not only for generation but many nlp tasks. in-domain examples are needed for finetuning. otherwise, the generated text, though fluent english, will not faithfully imbibe domain properties such as the vocabulary preferred, domain shifts in word meaning, and domain distribution over properties such as sentiment. the learned language ## related work there has been work using gpt-2 as a component in the data augmentation process for training classifiers (kumar et al., 2020; papanikolaou and pierleoni, 2020). we investigate augmentation for finetuning gpt-2 itself, and in fact deal with a precondition for the former -without a language model conforming to the domain, generated text would be further from the domain distribution. there is also work on data augmentation for training nlp classifiers such as wei and zou (2019), lu et al. (2006), and kobayashi (2018). we adopt some techniques from wei and zou (2019) for our experiments, but in general, augmentation techniques for classification do not necessarily work well for generation. the distribution learned in the latter case, p (x c |x), x @xmath0, is more complex than the former, p (y|x), @xmath1, due to a higher dimensional output variable (where y is the label set, x c denotes continuation, and |v | refers to the vocabulary). generation of adversarial examples (aves) to evaluate robustness of nlp tasks is another area being investigated. jia and liang (2017) construct aves for span-based qa by adding sentences with distractor spans to passages. zhang et al. (2019b) use word swapping to craft aves for paraphrase detection. unlike these works, we are not concerned with test-time invariance or test-time model behavior on augmented examples, as long as these augmented examples improve training. kang et al. (2018) and glockner et al. (2018) use wordnet relations to construct aves for textual entailment. however, to the best of our knowledge, we are the first ones to explore such methods using wordnet and lexical databases for text data augmentation for generative models. ## conclusion and future work we introduced and investigated genaug: data augmentation for text generation, specifically finetuning text generators, through various augmentation methods. we finetuned gpt-2 on a subset of the yelp reviews dataset, and demonstrated that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods. we also showed that the quality of generated text improves to a peak at approximately three times the amount of original training data. potential future directions include exploring augmentation based on a) linguistic principles like compositionality (andreas, 2020) and b) using more complex lexical resources -e.g. framenet (baker et al., 1998). one can also investigate further augmentation techniques using word replacement such as exploring the contextual augmentation method used in kobayashi (2018). further, methods of improving semantic text exchange (ste) on longer texts can be investigated, which would make it more effective for data augmentation. lastly, there is potential in exploring data augmentation for other domains such as dialogue and related tasks such as style transfer (kang et al., 2019), and investigating interesting aspects of it such as dialogue personalization (li et al., 2020).