Schema-Guided Natural Language Generation (2020)

Abstract: Neural network based approaches to data-to-text natural language generation (NLG) have gained popularity in recent years, with the goal of generating a natural language prompt that accurately realizes an input meaning representation. To facilitate the training of neural network models, researchers created large datasets of paired utterances and their meaning representations. However, the creation of such datasets is an arduous task and they mostly consist of simple meaning representations composed of slot and value tokens to be realized. These representations do not include any contextual information that an NLG system can use when trying to generalize, such as domain information and descriptions of slots and values. In this paper, we present the novel task of Schema-Guided Natural Language Generation (SG-NLG). Here, the goal is still to generate a natural language prompt, but in SG-NLG, the input MRs are paired with rich schemata providing contextual information. To generate a dataset for SG-NLG we re-purpose an existing dataset for another task: dialog state tracking, which includes a large and rich schema spanning multiple different attributes, including information about the domain, user intent, and slot descriptions. We train different state-of-the-art models for neural natural language generation on this dataset and show that in many cases, including rich schema information allows our models to produce higher quality outputs both in terms of semantics and diversity. We also conduct experiments comparing model performance on seen versus unseen domains, and present a human evaluation demonstrating high ratings for overall output quality.

## introduction much of the recent work on neural natural language generation (nnlg) focuses on generating a * authors contributed equally and are listed alphabetically. dataset mr reference e2e (novikova et al., 2017b) inform name[the punter], food[indian], pricerange[cheap] the punter offers cheap indian food. laptop (wen et al., 2016) inform name[satellite eu-rus65], type[laptop], memory[4gb], driver-range[medium], isforbusiness[false] the satellite eurus 65 is a laptop designed for home use with 4 gb of memory and a medium sized hard drive only having simple and limited information within these mrs has several shortcomings. model outputs are either very generic or generators have to be trained for a narrow domain and cannot be used for new domains. thus, some recent work has focused on different methods to improve naturalness (zhu et al., 2019) and promote domain transfer (tran and nguyen, 2018; wen et al., 2016). mrs are not unique to the problem of language generation: tasks such as dialog state tracking (rastogi et al., 2019), policy learning (chen et al., 2018), and task completion (li et al., 2017) also require the use of an mr to track context and state information relevant to the task. mrs from these more dialog-oriented tasks are often referred to as a "schemata." while dialog state tracking schemata do not necessarily include descriptions (and generally only include names of intents, slots, and values like traditional mrs), recent work has suggested that the use of descriptions may help with different language tasks, such as zero-shot and transfer learning (bapna et al., 2017). the most recent dialog system technology challenge (dstc8) (rastogi et al., 2019) provides such descriptions and introduces the idea of schema-guided dialog state tracking. to our knowledge, no previous work on nnlg has attempted to generate natural language strings from schemata using this richer and more informative data. in this paper, we propose the new task of schema-guided natural language generation, where we take a turn-level schema as input and generate a natural language string describing the required content, guided by the context information provided in the schema. following previous work on schema-guided language tasks, we hypoth-esize that descriptions in the schema will lead to better generated outputs and the possibility of zeroshot learning (bapna et al., 2017). for example, to realize the mr request(time), domain-specific descriptions of common slots like time can help us realize better outputs, such as "what time do you want to reserve your dinner?" in the restaurant domain, and "what time do you want to see your movie?" for movies. similarly, we note that for dialog system developers, writing domain-specific templates for all scenarios is clearly not scalable, but providing a few domain-specific descriptions for slots/intents is much more feasible. we focus on system-side turns from the dstc8 dataset and, to allow our models to better generalize, we generate natural language templates, i.e., delexicalized surface forms, such as "is there a specific cuisine type you enjoy, such as @xmath0cuisine2, or something else?" from the example schema in our contributions in this paper are three-fold: (1) we introduce a novel task and repurpose a dataset for schema-guided nlg, (2) we present our methods to include schema descriptions in state-of-theart nnlg models, and (3) we demonstrate how using a schema frequently leads to better quality outputs than traditional mrs. we experiment with three different nnlg models (sequence-to-sequence, conditional variational autoencoders, and gpt-2 as a pretrained language model). we show that the rich schema information frequently helps improve model performance on similarity-toreference and semantic accuracy measures across domains, and that it promotes more diverse outputs with larger vocabularies. we also present a human evaluation demonstrating the high quality of our outputs in terms of naturalness and semantic correctness. ## data to create a rich dataset for nnlg, we repurpose the dataset used for the schema-guided state tracking track of dstc8 (rastogi et al., 2019). @xcite we preprocess the data to create our schema-guided natural language (sg-nlg) dataset for training and evaluating our nnlg models. @xcite since we are focused on system turns, we first drop all the user turns. the second step in the preprocessing pipeline is to delexicalize each of the system utterances. the original data is annotated with the spans of the slots mentioned in each turn. we replace these mentions with the slot type plus an increasing index prefixed by the @xmath1cuisine 1. for example, the utterance "is there a specific cuisine type you enjoy, such as mexican, italian, or something else?" becomes "is there a specific cuisine type you enjoy, such as @xmath2cuisine 2 or something else? the third step is to construct the mr corresponding to each system turn. we represent an mr as a triplet: one dialog act with exactly one slot and one value. therefore, an mr that in the original dstc8 dataset is represented as request(@xmath5, italian]) becomes request(@xmath6. note that the mr has been delexicalized in the same fashion as the utterance. similarly, for mrs that do not have a value, e.g., request(city), we introduced the null value resulting in request(@xmath7. we also use the null value to replace the slot in dialog acts that do not require one, e.g., bye() becomes bye(@xmath8. once we generate templates and mr pairs, we add information about the service. in dstc8, there are multiple services within a single domain, e.g., services travel 1 and travel 2 are both part of the travel domain, but have distinct schema. foot_2 dstc8 annotates each turn with the corresponding service, so we reuse this information. our schema also includes user intent. @xcite since only user turns are annotated with intent information, we use the immediately preceding user turn's intent annotation if the system turn and the user turn share the same service. if the service is not the same, we drop the intent information, i.e., we use an empty string as the intent (this only happens in 3.3% of cases). next, we add information extracted from the schema file of the original data. this includes service description, slot descriptions (one description for each slot in the mr), and intent descriptions. these descriptions are very short english sentences (on average 9.8, 5.9 and 8.3 words for services, slots and intents). lastly, we add to each triplet a sentence describing, in plain english, the meaning of the mr. these description are not directly available in dstc8 but are procedurally generated by a set of rules. @xcite for example, the mr con-firm(@xmath9." the intuition behind these natural language mrs is to provide a more semantically informative representation of the dialog acts, slots and values. ## evaluation for each of our three models, we generate a single output for each test instance. ## related work most work on nnlg uses a simple mr that consists of slots and value tokens that only describe information that should be realized, without including contextual information to guide the generator as we do; although some work has described how this could be useful (walker et al., 2018). webnlg (colin et al., 2016) includes structured triples from wikipedia which may constitute slightly richer mrs, but are not contextualized. oraby et al. (2019) generate rich mrs that contain syntactic and stylistic information for generating descriptive restaurant reviews, but do not add in any contextual information that does not need to be included in the output realization. table -to-text generation using rotowire (nba players and stats) also includes richer information, but it is also not contextualized (wiseman et al., 2017; gong et al., 2019). other previous work has attempted to address domain transfer in nlg. dethlefs et al. (2017) use an abstract meaning representation (amr) as a way to share common semantic information across domains. wen et al. (2016) use a "data counterfeiting" method to generate synthetic data from existing domains to train models on unseen domains, then fine-tune on a small set of in-domain utterances. tran et al. (2018) also train models on a source domain dataset, then fine-tune on a small sample of target domain utterances for domain adaptation. rather than fine-tuning models for new domains, our data-driven approach allows us to learn domain information directly from the data schema. ## conclusions in this paper, we present the novel task of schema-guided nlg. we demonstrate how we are able to generate templates (i.e., delexicalized system prompts) across different domains using three stateof-the-art models, informed by a rich schema of information including intent descriptions, slot descriptions and domain information. we present our novel sg-nlg dataset, which we construct by repurposing a dataset from the dialog state tracking community. in our evaluation, we demonstrate how training using our rich schema frequently improves the overall quality of generated prompts. this is true for different similarity metrics (up to 0.43 bleu and 0.61 meteor) that we recognize are weak measures of quality but, more importantly, for semantic metrics (as low as 0.18 average ser), and even for diversity (up to 2.6k bigram vocabulary). moreover, this holds true on both seen and unseen domains in many different settings. we conduct a human evaluation as a more accurate quality assessment, and show how our outputs are rated up to 3.61 out of 5 overall (as compared to 3.97 for references). we observe that different models have different strengths: seq2seq and cvae have higher bleu reference similarity scores, while gpt2 is significantly more diverse and is scored highest overall in human evaluation. for future work, we are interested in exploring how schema-guided nlg can be used in dialog system contexts, where only outputs that have no slot errors and high overall fluency should be selected as responses. we are also interested in improving both the semantic correctness and fluency of our model outputs by introducing improved methods for constrained decoding and language model integration. additionally, we plan to develop more accurate automatic measures of quality, as well as more fine-grained control of domain transfer.