Few-shot Natural Language Generation for Task-Oriented Dialog (2020)

Abstract: As a crucial component in task-oriented dialog systems, the Natural Language Generation (NLG) module converts a dialog act represented in a semantic form into a response in natural language. The success of traditional template-based or statistical models typically relies on heavily annotated data, which is infeasible for new domains. Therefore, it is pivotal for an NLG system to generalize well with limited labelled data in real applications. To this end, we present FewshotWOZ, the first NLG benchmark to simulate the few-shot learning setting in task-oriented dialog systems. Further, we develop the SC-GPT model. It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains. Experiments on FewshotWOZ and the large Multi-Domain-WOZ datasets show that the proposed SC-GPT significantly outperforms existing methods, measured by various automatic metrics and human evaluations.

## introduction task-oriented dialog systems are becoming increasingly popular, as they can assist users in various daily activities such as ticket booking and restaurant reservations. in a typical task-oriented dialog system, the natural language generation (nlg) module plays a crucial role: it converts a system action (e.g., often specified in a semantic form selected by a dialog policy) into a final response in natural language. hence, the response should be adequate to represent semantic dialog actions, and fluent to engage users' attention. as the ultimate interface to interacts with users, nlg plays a significant impact on the users' experience. existing methods for nlg can be broadly summarized into two major categories. (i) template-based methods require domain experts to handcraft templates for each domain, and the system fills in slot-values afterward @xcite . thus, the produced responses are often adequate to contain the required semantic information, but not always fluent and nature, hurting users' experiences. (ii) statistical language models such as neural networks @xcite learn to generate fluent responses via training from labelled corpus. one canonical model is semantically conditioned lstm (sc-lstm) @xcite , which encodes dialog acts with onehot representations and uses it as an extra feature to inform the sentence generation process. despite its good performance on simple domains, it requires large amounts of domain-specific annotated data which is not available for many domains in realworld applications. even worse, this renders severe scalability issues when the number of possible combinations of dialog acts grows exponentially with the number of slots in more complex domains. we revisit the current research benchmarks for nlg, and notice that each dialog domain is extensively labelled to favor model training. however, this is in contrast to the real-world application scenarios, where only very limited amounts of labelled data are available for new domains. to simulate such a few-shot learning setting, we have developed a new benchmark dataset, called fewshot-woz, based on the multiwoz @xcite and cambridge nlg datasets @xcite . fewshotwoz consists of dialog utterances from 7 domains. for each domain, we provide less than 50 labeled utterances for finetuning. we believe that fewshotwoz can better inspire research to address the challenge of learning data-hungry statistical models with very limited amounts of labelled data in real-world scenarios. to deal with the challenge of few-shot learning, we develop the sc-gpt model. sc-gpt is a multilayer transformer neural language model, trained in three steps: (i) pre-trained on plain text, similar to gpt-2 @xcite ; (ii) continuously pretrained on large amounts of dialog-act labeled utterances corpora to acquire the ability of controllable generation; (iii) fine-tuned for a target domain using very limited amounts of domain labels. unlike gpt-2, sc-gpt generates semantically controlled responses that are conditioned on the given semantic form, similar to sc-lstm but requiring much less domain labels to generalize to new domains. in summary, our key contributions are three-fold: ## background a typical task-oriented spoken dialog system uses a pipeline architecture, as shown in figure the dialog act emitted by the dialog policy module serves as the input to the nlg, through which a system response in natural language is generated. in this paper, we focus on the nlg component of task-oriented dialog systems, i.e., how to produce natural language responses conditioned on dialog acts. specifically, dialog act a is defined as the combination of intent i and slot-value pairs {(s i , v i )} p @xmath0: the goal of nlg is to translate a into a natural language response @xmath1, • • • , x t ], where t is the sequence length. in figure 1 (b), we show an example of the dialog act: confirm (@xmath2, @xmath3, and the system response dialog act corresponding natural language response is "let me confirm that you are searching for hilton in the center area". ## semantically conditioned gpt we tackle this generation problem using conditional neural language models. given training data of n samples @xmath4, x n )} n @xmath5, our goal is to build a statistical model parameterized by θ to characterize p θ (x|a). to leverage the sequential structure of response, one may further decompose the joint probability of x using the chain rule, casting an auto-regressive generation process as follows: learning θ is performed via maximizing the loglikelihood (mle) of the conditional probabilities in (2) over the entire training dataset: in this paper, we employ the transformers @xcite to parameterize the conditionals in (2). to enable strong generalization and controllable ability for the learned model, we propose the following three-stage procedure as the training recipe. massive plain language pre-training. large models trained on massive training corpus usually generalize better to new domains. inspired by this, we inherit the gpt-2 architecture @xcite as the backbone language model. gpt-2 is an auto-regressive language model that leverages 12-24 layers of masked, multi-head self-attention transformers. gpt-2 is pre-trained on extremely massive text data openwebtext @xcite . it has demonstrated superior performance on characterizing human language data distribution and knowledge transfer. given text prompts, gpt-2 can often generate realistic sentences. ## experiments in this section, we evaluate the proposed sc-gpt on the fewshotwoz and multiwoz datasets to answer two research questions: (i) is sc-gpt an effective model for strong generalization and controlability in dialog response generation? (ii) does fewshotwoz meet the goal of effectively evaluating the generalization of nlg models in the few-shot learning setting? ## experiments in this section, we evaluate the proposed sc-gpt on the fewshotwoz and multiwoz datasets to answer two research questions: (i) is sc-gpt an effective model for strong generalization and controllability in dialog response generation? (ii) does fewshotwoz meet the goal of effectively evaluating the generalization of nlg models in the few-shot learning setting? ## conclusion and future work in this paper, we have made two major contributions towards developing a more pragmatic nlg module in task-oriented dialogue systems: (i) a new benchmark fewshotwoz is introduced to simulate the few-shot learning scenarios with in this few-shot learning setting. the generated utterances are poor in quality and suffer from inaccurate slot rendering. in addition, gpt-2 performs consistently better than sc-lstm in all the domains. it reveals the feasibility of using a pretrained language model for nlg, though only limited annotations are available for fine-tuning. importantly, sc-gpt performs significantly better than gpt and sc-lstm in terms of both bleu and err. in all the domains, sc-gpt reduces the err to a significantly lower level, revealing its strong controllability power. this verifies the importance of pre-training on large annotated dialog data, as sc-gpt learns how to generate utterances specified by the dialog acts accurately.