A ug NLG : Few-shot Natural Language Generation using Self-trained Data Augmentation (2021)

Abstract: Natural Language Generation (NLG) is a key component in a task-oriented dialogue system, which converts the structured meaning representation (MR) to the natural language. For large-scale conversational systems, where it is common to have over hundreds of intents and thousands of slots, neither template-based approaches nor model-based approaches are scalable. Recently, neural NLGs started leveraging transfer learning and showed promising results in few-shot settings. This paper proposes AugNLG, a novel data augmentation approach that combines a self-trained neural retrieval model with a few-shot learned NLU model, to automatically create MR-to-Text data from open-domain texts. The proposed system mostly outperforms the state-of-the-art methods on the FewshotWOZ data in both BLEU and Slot Error Rate. We further confirm improved results on the FewshotSGD data and provide comprehensive analysis results on key components of our system. Our code and data are available at https://github.com/XinnuoXu/AugNLG .

## introduction large-scale conversational systems provide a natural interface to achieve various daily-life tasks. natural language generation (nlg) is a key component in such a system to convert the structured meaning representation (mr) to the natural language, as shown in figure 1 . in task-oriented dialogue systems, nlg is typically accomplished by filling out a basic set of developer-provided templates, leading to a conversational system generating unnatural, robotic responses. in order to make the system sound more human-like, model-based nlg approaches, in particular neural models, have recently been gaining an increasing traction @xcite . however, neither the template-based approaches nor the model-based approaches are sufficiently scalable for large-scale conversational systems, where it is common to have over hundreds of intents and thousands of slots. with the rise of neural transfer learning for nlp using pretrained lms, recently, neural nlgs started to leverage transfer learning and showed some promising results @xcite @xcite . in particular, @xcite proposed fewshotwoz, the first nlg benchmark test in few-shot learning settings, and achieved a sota performance by leveraging existing mr-to-text data sets via task-specific continued pre-training. despite the improved result, their approach leaves little room for further improvements as mr-to-text data are expensive to obtain for new domains, practically circling back to the same scalability problem after exhausting the existing data. in order to go beyond this restriction, this paper proposes augnlg, a novel data augmentation approach, that automatically creates mr-to-text data from open-domain texts by combining a self-trained neural retrieval model with a few-shot learned nlu model. since our data augmentation approach is orthogonal to the prior transfer learning approaches, one can use our approach in conjunction with other approaches. in experiments, we empirically show that augnlg mostly boosts the performance of both the fine-tuned gpt-2 (ft-gpt) @xcite and sc-gpt @xcite , the continued pretraining approach with existing mr-to-text data, on the fewshotwoz task. furthermore, we construct another fewshot learning testbed, fewshotsgd, out of the schema-guided dialogue (sgd) corpus @xcite and confirm improved results by applying augnlg to the ft-gpt. 1 finally, we provide comprehensive analysis results on the key components of our system to gain detailed insights into the relationship between component-wise behavior and various parameters. ## few-shot transfer learning for nlg the goal of nlg is to translate an mr a into its natural language response @xmath0, . . . , x t , where x i is the ith token in the sequence x and t is the sequence length. a is defined as the combination of intent i and slot-value pairs {(s i , v i )} p @xmath1: given in-domain mr-to-text data @xmath2, x n )} n @xmath3, where n is the number of examples, a statistical neural language model parameterized by θ is adopted to characterize the conditional probability p θ (x|a). by adopting the chain rule on auto-regressive generation, the joint probability of x conditioned on a is decomposed as t @xmath4, a). the training process, i.e. the learning of θ, is then defined as maximizing the log-likelihood of the conditional probabilities over the entire training dataset: in the few-shot learning setup, the number of training examples n is extremely small (e.g. ≤ 50), which easily leads to non-fluent generated sentences with many grammar mistakes or missing pieces of information. in order to combat the data sparseness problem, inspired by prior transfer learning approaches, we introduce a three-step pipeline to gradually evolve a general large-scale language model to a domain-specific nlg model (shown in figure 2 ): (1) pre-training a base language model with massive amounts of text, (2) nlg-specific continued pre-training with auto-augmented mrto-text data, and (3) final fine-tuning with the limited in-domain mr-to-text ground-truth data. specifically, in step (1), we adopt gpt-2 (radford et al., 2019) as our base language model since gpt-2 has demonstrated a remarkable performance on auto-regressive text generation tasks, which is close to mr-to-text generation, in a variety of domains. however, gpt-2 is pre-trained on open-webtext and the language style and topics thereof are quite different from those of daily conversations in a target domain. furthermore, the generation task in nlg is conditioned on the input mr, as opposed to the unconditioned generation of the underlying gpt-2 pre-training task. thus, to bring the model a step closer to the final nlg model in the target domain, in step (2), we continuously pre-train the gpt-2 model on an automatically constructed set of augmented mr-to-text pairs @xmath5, x m )} m @xmath6, where m is the number of augmented examples, which is much larger than the amount of in-domain ground-truth data. data augmentation is achieved by retrieving a large amount of relevant text from reddit @xcite with a self-trained neural retrieval model and then synthesizing mrs with a few-shot learned nlu model. the details of data augmentation is described in section 4. finally, in step (3), we fine-tune the nlg model on a limited amount of in-domain ground-truth mr-to-text pairs d for a final adaptation. ## data augmentation the data augmentation procedure aims to construct a large amount of mr-to-text pairs d from open-domain texts that are relevant to the in-domain ground-truth mr-to-text pairs d. the augmentation process consists of two stages: (1) retrieving keyword-matching utterances and filtering out domain-irrelevant instances, (2) generating synthetic mr annotations. figure 3 illustrates the overall pipeline with some examples. for further analysis and studies, we release the data from all intermediate steps for each domain at https://github.com/xinnuoxu/ augnlg/tree/master/augmented_data . ## results fewshotwoz comparing to sc-gpt, ft-gpt tends to overgenerate and introduces hallucinations. however, augnlg and augnlg-sc managed to generate fluent, natural text while precisely reflecting the the input mr. we further examined 70 randomly sampled utterances generated by augnlg-sc, whose bleu scores are lower than those generated by sc-gpt, in the "hotel", "train" and "taxi" domain to understand some potential factors causing the lower bleu scores we found that the lower bleu scores are mainly driven by bleu penalizing semantically correct paraphrases due to the nature of bleu only checking surface-level matches. some examples of such penalization are provided in ## in-depth analysis in this section, we provide comprehensive analysis results on the key components and parameters of our system to gain detailed insights: (1) intrinsic evaluation on augmented data, (2) influence of nlu quality, and (3) performance trends over varying amounts of augmented data. ## conclusion in this paper, we proposed augnlg, a novel data augmentation approach that combines a self-trained retrieval model with a few-shot learned nlu, to automatically create mr-to-text data from opendomain texts. experimental results verify the effectiveness of our approach by establishing new sota performances on two benchmark tests. more importantly, we showed how our approach complements the previous sota approach, which hinges on unscalable data sources, with unlimited opendomain data. future work includes (1) technical innovations on each component of our system for further performance improvements, (2) exploring self-training on the nlu side too to evolve both the nlu and nlg model at the same time.