How to Make Neural Natural Language Generation as Reliable as Templates in Task-Oriented Dialogue (2020)

Abstract: Neural Natural Language Generation (NLG) systems are well known for their unreliability. To overcome this issue, we propose a data augmentation approach which allows us to restrict the output of a network and guarantee reliability. While this restriction means generation will be less diverse than if randomly sampled, we include experiments that demonstrate the tendency of existing neural generation approaches to produce dull and repetitive text, and we argue that reliability is more important than diversity for this task. The system trained using this approach scored 100% in semantic accuracy on the E2E NLG Challenge dataset, the same as a template system.

## introduction the goal of task oriented dialogue is to help a user achieve a narrow goal, such as booking a restaurant or movie ticket. the final step of a conversational interface is generating a response to the user; more specifically, performing surface realization of some structured data containing relevant information. research into neural nlg systems for the surface realization task is popular because such systems may have advantages over the dominant rule and template-based systems: neural nlg systems trained on datasets may be both easier to maintain and to scale to new domains, as well as generating more natural responses @xcite . but neural nlg systems are not without problems. they are widely considered too unreliable for business applications; they have a tendency to hallucinate facts, unsupported by the structured data they were given @xcite . a less well known issue is the template-like generation of neural nlg systems @xcite . figure 1 highlights this issue; neural nlg systems (tgen and slug2slug) are far less diverse than the training data (e2e dataset) in their usage of surface forms that express an attribute. intuitively, one might expect that a neural nlg system trained on a dataset with 75 different surface forms to express an attribute would use a wide variety of them -instead we see only the top two most common surface forms in use. we highlight the issue of lack of diversity, not to provide a specific solution to it, but rather to provide some context for our proposal which relates to reliability of neural nlg systems. given that our main goal is reliability, we wondered if there were some way to lean into the blandness and lack of diversity of existing neural systems. we propose a data-oriented and model-agnostic solution. using the e2e nlg challenge dataset @xcite , we experiment with an augmented input sequence that includes the surface form of each attribute-value pair. by including the surface form in the input sequence, we can use a restricted decoding strategy when generating an utterance. this guarantees reliability. by sacrificing a small amount of unconstrained diversity, we are able to achieve 100% semantic accuracy on the e2e dataset. ## frequency of surface forms to compare the diversity of the e2e training data with that of the generated text, we looked at the surface forms used to express each attribute-value pair. this is enabled by a set of regular expressions released by @xcite foot_0 . the regular expressions capture the entire phrase used to express an attribute-value pair, focusing on the content words and attempting as much as possible to leave out the function words, e.g. we counted the surface forms used for a given attribute-value pair, in both the dataset and generated text, and plotted them against each other, see figure 1 and additional figures in the supplementary material. while there was an average of 133 different surface forms for each attribute-value pair in the e2e dataset, the neural systems, on average, only generated 3 of the most common surface forms. this convinced us that the diversity was hardly any better than templates, which by default only use a single surface form per attribute. ## method how can a neural nlg system generate text from a set of attribute-value pairs and ensure that they appear correctly in the generated text? as opposed to templates, which are static, neural nlg systems are statistical generators and provide no inherent guarantees of accuracy. thus we propose augmenting the input sequence with the surface form of each attribute-value pair. this augmented input sequence enables us to restrict the text that is generated in a way that provides guaranteed accuracy. ## experimental setup we performed experiments with the e2e nlg challenge dataset @xcite . it is a task-oriented dialogue dataset, collected using crowd sourcing, focused on the surface realization of attribute-value pairs describing restaurants. ## discussion this is not the first data-focused approach to improving accuracy; @xcite also proposed a constrained decoding strategy. the difference between our decoding strategies lies in the guarantees provided. their approach focuses on an augmented target utterance, as opposed to input sequence, in which special bracket tokens are used to surround surface forms. e.g.