Template Guided Text Generation for Task-Oriented Dialogue (2020)

Abstract: Virtual assistants such as Google Assistant, Amazon Alexa, and Apple Siri enable users to interact with a large number of services and APIs on the web using natural language. In this work, we investigate two methods for Natural Language Generation (NLG) using a single domain-independent model across a large number of APIs. First, we propose a schema-guided approach which conditions the generation on a schema describing the API in natural language. Our second method investigates the use of a small number of templates, growing linearly in number of slots, to convey the semantics of the API. To generate utterances for an arbitrary slot combination, a few simple templates are first concatenated to give a semantically correct, but possibly incoherent and ungrammatical utterance. A pre-trained language model is subsequently employed to rewrite it into coherent, natural sounding text. Through automatic metrics and human evaluation, we show that our method improves over strong baselines, is robust to out-of-domain inputs and shows improved sample efficiency.

## introduction virtual assistants have become popular in recent years and task-completion is one of their most important aspects. these assistants help users in accomplishing tasks such as finding restaurants, buying sports tickets, finding the weather etc., by providing a natural language interface to many services or apis available on the web. most systems include a natural language understanding and dialogue state tracking module for semantic parsing of the dialogue history. this is followed by a policy module which interacts with the apis, whenever required, and generates the actions to be taken by the system to continue the dialog. in the end, the natural language generation (nlg) module converts these actions into an utterance, which is surfaced to the user. being the user-facing interface of the dialogue system, nlg is one of the most important components impacting user experience. traditional nlg systems heavily utilize a set of templates to produce system utterances. although the use of templates gives good control over the outputs generated by the system, defining templates becomes increasingly tedious as more apis are added. supporting multi-domain conversations spanning multiple apis quickly grows out of hand, requiring expert linguists and rigorous testing to ensure the grammatical correctness and appropriateness of generated utterances. consequently, data-driven generative approaches have gained prominence. such systems require much less effort and can generate utterances containing novel patterns. meanwhile, with the rapid proliferation of personal assistants, supporting large number of apis across multiple domains has become increasingly important, resulting in research on supporting new apis with few labelled examples (few-shot learning). to this end, generative models pre-trained on large amounts of unannotated text have been increasingly successful. in this work, we address the challenges of joint modeling across a large number of domains, and data efficient generalization to new domains and apis for nlg. our contributions are the following: the model rewrites it to a conversational response surfaced to the user. ## related work natural language generation from structured input (nlg) has been an active area of research, facilitated by creation of datasets like wikibio @xcite , e2e challenge @xcite , webnlg @xcite and multiwoz @xcite . neural sequence models have been extensively used in a variety of configurations for nlg in dialogue systems. @xcite proposed a two-step approach: first generating a delexicalized utterance with placeholders for slots and then post-processing it to replace placeholders with values from api results, whereas @xcite highlighted the importance of conditioning responses on slot values. sequence to sequence architectures directly converting a sequential representation of system ac-tions to a system response are also very common @xcite @xcite . domainadaptation and transfer learning in low resource settings has also been an extensively studied problem @xcite @xcite , with recently released datasets like sgd @xcite and fewshotwoz @xcite providing a good benchmark. meanwhile, language models pre-trained on large amount of unannotated text corpus have achieved state-of-the-art performance across several natural language processing tasks @xcite @xcite @xcite , including natural language generation @xcite . our template based approach bears similarities to sentence fusion @xcite , and prototype based text editing @xcite @xcite . however, none of these works tackle text generation from structured data. ## model the goal of nlg is to translate a to a natural language response with the same semantic content. to this end, we first convert the set a into a sequence. then, we finetune a text-to-text transfer transformer (t5) @xcite ) model, which is a pre-trained sequence to sequence transformer, to generate the natural language response using this sequence as input. now, we present three different methods for converting a into a sequence, the last two being our contributions. they are also summarized in figure 2 . ## experimental setup we conduct a series of experiments to compare the three system action representations presented above. we also evaluate nlg in few-shot settings and investigate a few other aspects of the sgd dataset. in each of the experiments reported in this paper, we start with a pre-trained t5-small model foot_1 . it has 6 layers each in the encoder and decoder, with a total of around 60 million parameters. the model is then fine-tuned on the corresponding dataset using a constant learning rate of 0.001 and batch size of 256 for 5000 steps. the checkpoint yielding the highest bleu score on the development set is picked for reporting test set results. during inference, we use beam search with a width of 4 and length penalty α = 0.6. ## action representations we compare the different methods of action representation on multiwoz 2.1 @xcite , the cleaned version of the e2e restaurant corpus @xcite and the schema-guided dialogue (sgd) @xcite dataset. the sgd dataset features a larger number of domains and slots, and the presence of multiple apis per domain (figure 4 ) makes it representative of practical scale-related challenges faced by today's virtual assistants. furthermore, as opposed to the other two datasets, its evaluation sets contain many domains, and consequently slots, which are not present in the training set. even for domains shared between the training and evaluation sets, the evaluation sets contain additional slots in some cases. this focus on zero-shot generalization to new domains and apis makes sgd more challenging than existing nlg benchmarks. ## few-shot nlg virtual assistants need to support a constantly increasing number of domains and apis. in order to keep labelled data costs under control, improving few-shot learning methods is important. in this section, we study the trade-off between the number of annotated training examples and performance of nlg. prior work @xcite @xcite has studied few-shot learning and domain adaptation in a simulated setting by creating small subsets. however, lack of knowledge of the exact data splits makes it difficult to make comparisons to other methods. to remedy this, we create a new canonical split of the sgd dataset as described below. for every domain, we make sure that each dialog act (inform, request etc.) and slot (name, time, price etc.) is represented at least once. however, all combinations of dialog acts and slots may not exist. this benchmark is referred to as fewshotsgd and we make the exact splits publicly available. the exact number of examples in each split is given in ## other experiments in this section, we conduct experiments to explore a few other aspects of our setup on the sgd dataset. for these experiments we use the naive representation, since it is more widely adopted in prior work. we hope that these experiments will guide design choices in the future nlg models. ## conclusion and future work in this work, we proposed schema guided and template guided input representation schemes for task oriented response generation. coupled with pretrained language models, the template guided approach enables zero-shot generalization to new domains with little effort. moreover, we show that it can lead to drastic reduction in annotation costs. we also present the first set of results on the multidomain sgd dataset, which we hope will pave the way for further research in few-shot, zero-shot and multi-domain language generation. while in this paper we use standard pre-trained models, designing pre-training tasks tailored to sentence fusion is an interesting line of future work. we also hope to apply t2g2 to languages other than english. obtaining annotated data in non-english languages is an even bigger challenge, making the sample efficiency of our template rewriting approach especially suited to this setting. another interesting line of future work is to investigate the use of t2g2 for generating user utterances, which could be useful for dialogue data augmentation and user simulation. this requires adding the ability to generate utterances with stylistic variations to capture different user personalities while maintaining consistency in style and vocabulary over a single dialogue. proceedings of the 2015 conference on empirical methods in natural language processing, pages 1711-1721. tsung-hsien wen, david vandyke, nikola mrk si'c, milica gasic, lina m rojas barahona, pei-hao su, stefan ultes, and steve young. 2017. a networkbased end-to-end trainable task-oriented dialogue system. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 1, long papers, pages 438-449. yu wu, furu wei, shaohan huang, yunli wang, zhoujun li, and ming zhou. 2019. response generation by context-aware prototype editing. in proceedings of the aaai conference on artificial intelligence, volume 33, pages 7281-7288. zhilin yang, zihang dai, yiming yang, jaime carbonell, russ r salakhutdinov, and quoc v le. 2019. xlnet: generalized autoregressive pretraining for language understanding. in advances in neural information processing systems, pages 5754-5764. chenguang zhu, michael zeng, and xuedong huang. 2019. multi-task learning for natural language generation in task-oriented dialogue. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 1261-1266.