Slot-consistent NLG for Task-oriented Dialogue Systems with Iterative Rectification Network (2020)

Abstract: Data-driven approaches using neural networks have achieved promising performances in natural language generation (NLG). However, neural generators are prone to make mistakes, e.g., neglecting an input slot value and generating a redundant slot value. Prior works refer this to hallucination phenomenon. In this paper, we study slot consistency for building reliable NLG systems with all slot values of input dialogue act (DA) properly generated in output sentences. We propose Iterative Rectification Network (IRN) for improving general NLG systems to produce both correct and fluent responses. It applies a bootstrapping algorithm to sample training candidates and uses reinforcement learning to incorporate discrete reward related to slot inconsistency into training. Comprehensive studies have been conducted on multiple benchmark datasets, showing that the proposed methods have significantly reduced the slot error rate (ERR) for all strong baselines. Human evaluations also have confirmed its effectiveness.

## introduction natural language generation (nlg), as a critical component of task-oriented dialogue systems, converts a meaning representation, i.e., dialogue act (da), into natural language sentences. traditional methods @xcite @xcite are mostly pipeline-based, dividing the generation process into sentence planing and surface realization. despite their robustness, they heavily rely on handcrafted rules and domain-specific knowledge. in addition, the generated sentences of rule-based approaches are rather rigid, without the variance of human language. more recently, neural network based models @xcite extracted from sf hotel @xcite dataset. errors are marked in colors (missing, misplaced). @xcite have attracted much attention. they implicitly learn sentence planning and surface realisation end-to-end with cross entropy objectives. for example, @xcite employ an attentive encoder-decoder model, which applies attention mechanism over input slot value pairs. although neural generators can be trained end-to-end, they suffer from hallucination phenomenon @xcite . examples in in this paper, we propose iterative rectification network (irn) to improve slot consistency for general nlg systems. irn consists of a pointer rewriter and an experience replay buffer. pointer rewriter iteratively rectifies slot-inconsistent generations from knn or data-driven nlg systems. experience replay buffer of a fixed size collects candidates, which consist of mistaken cases, for training irn. leveraging the above observations, we further introduce a retrieval-based bootstrapping to sample pseudo mistaken cases as candidates for enriching the training data. to foster consistency between training objective and evaluation metrics, we use reinforce @xcite to incorporate slot consistency and other discrete rewards into training objectives. extensive experiments show that, the proposed model, knn + irn, significantly outperforms all previous strong approaches. when applying irn to improve slot consistency of prior nlg baselines, we notice large reductions of their slot error rates. finally, the effectiveness of the proposed methods are further confirmed using bleu scores, case analysis and human evaluations. ## architeture figure 1 shows the architecture of iterative rectification network. it consists of two components: a pointer rewriter to produce templates with improved performance metrics and an experience replay buffer to gather and sample training data. the improvements on slot consistency are obtained via an iterative rewriting process. assume, at iteration k, we have a template y (k) that is not slot consistent with input da, i.e., g(y (k) ) = g(x). then, a pointer rewriter iteratively rewrites it as above recursion ends once g(y (k) ) = g(x) or a certain number of iterations is reached. ## training with supervised learning and distant supervision one key idea behind the proposed irn model is to conduct distant supervision on the actions of template copy and generation. we diagram its motivation in figure 2 . during training, only candidate y and its reference z are given. the exact actions that convert template y to z have to be inferred from the two templates. here we use simple rules for the inference. firstly, the rules check if reference token z j exists in the candidate y. the output is a label d c consisting of 1s and 0s, representing whether tokens in the reference template are existent/absent in the candidate. secondly, the rules locate the orig-inal position d l j in the candidate for each token j in the reference template if d @xmath0. finally, the action label d π for policy is inferred, with w for d l @xmath1. we may use the extracted tags to do supervised learning. the loss to be minimized is as follows however, there are following issues when attempting to utilize the labels produced by distant supervision for training. firstly, the importance of every token in candidate is different. for example, noun phrase (colored in blue) is critical and should be copied. function words (colored in red) is of little relevance and can be generated by irn itself. however, distant supervision treats them the same. secondly, rule-based matching may cause semantic ambiguity (dashed line colored in black). lastly, the training criterion of cross entropy is not directly relevant to the evaluation metric using slot error rate. to address these issues, we use reinforcement learning to obtain the optimal actions. ## training with policy-based reinforcement learning in this section, we describe another method to train irn. we apply policy gradient @xcite to optimize models with discrete rewards. ## experiments 6.1 experiment setup we assess the model performances on four nlg datasets of different domains. the sf hotel and sf restaurant benchmarks are collected in @xcite while laptop and tv benchmarks are released by @xcite . each dataset is evaluated with five strong baseline methods, including hlstm @xcite , sc-lstm @xcite , tgen @xcite , aroa @xcite and ralstm @xcite . following these prior works, the evaluation metrics consist of bleu and slot error rate (err), which is computed as we follow all baseline performances reported in @xcite and use open source toolkits, rnnlg 1 and tgen 2 to build nlg systems, hlstm, sclstm and tgen. we reimplement the baselines aroa and ralstm since their source codes are not available. ## related work conventional approaches for solving nlg task are mostly pipeline-based, dividing it into sentence planning and surface realisation @xcite @xcite . @xcite introduce a class-based ngram language model and a rule-based reranker. @xcite address the limitations of n-gram language models by using more sophisticated syntactic dependency trees. mairesse and young (2014) employ a phrase-based generator that learn from a semantically aligned corpus. despite their robustness, these models are costly to create and maintain as they heavily rely on handcrafted rules. recent works @xcite @xcite build data-driven models based on end-to-end learning. @xcite combine two recurrent neural network (rnn) based models with a cnn reranker to generate required utterances. @xcite introduce a novel sc-lstm with an additional reading cell to jointly learn gating mechanism and language model. @xcite present an attentive neural generator to apply attention mechanism over input da. @xcite employ a refiner component to select and aggregate the semantic elements produced by the encoder. more recently, domain adaptation @xcite and unsupervised learning @xcite for nlg also receive much attention. we are also inspired by the post-edit paradigm @xcite , which uses a second-pass decoder to improve the translation quality. a recent method in @xcite defines an auxiliary loss that checks if the object words exist in the expected system response of a task-oriented dialogue system. it would be interesting to apply this auxiliary loss in the proposed method. on the other hand, the reinforce @xcite algorithm applied in this paper is more general than @xcite to incorporate other metrics, such as bleu. nevertheless, end-to-end neural-based generators suffer from hallucination problem and are hard to avoid generating slot-inconsistent utterance @xcite . @xcite attempts to alleviate this issue by employing a treestructured meaning representation and constrained decoding technique. however, the tree-shaped structure requires additional human annotation. ## conclusion we have proposed iterative rectification network (irn) to improve slot consistency of general nlg systems. in this method, a retrieval-based bootstrapping is introduced to sample pseudo mistaken cases from training corpus to enrich the original training data. we also employ policy-based reinforcement learning to enable training the models with discrete rewards that are consistent to evaluation metrics. extensive experiments show that the proposed model significantly outperforms previous methods. these improvements include both of correctness measured with slot error rates and naturalness measured with bleu scores. human evaluation and case study also confirm the effectiveness of the proposed method.