Boosting Naturalness of Language in Task-oriented Dialogues via Adversarial Training (2020)

Abstract: The natural language generation (NLG) module in a task-oriented dialogue system produces user-facing utterances conveying required information. Thus, it is critical for the generated response to be natural and fluent. We propose to integrate adversarial training to produce more human-like responses. The model uses Straight-Through Gumbel-Softmax estimator for gradient computation. We also propose a two-stage training scheme to boost performance. Empirical results show that the adversarial training can effectively improve the quality of language generation in both automatic and human evaluations. For example, in the RNN-LG Restaurant dataset, our model AdvNLG outperforms the previous state-of-the-art result by 3.6% in BLEU.

## introduction in task-oriented dialogues, the computer system communicates with the user in the form of a conversation and accomplishes various tasks such as hotel booking, flight reservation and retailing. in this process, the system needs to accurately convert the desired information, a.k.a. meaning representation, to a natural utterance and convey it to the users ( however, one issue yet to be solved is that the system responses often lack the fluency and naturalness of human dialogs. in many cases, the system responses are not natural, violating inherent human language usage patterns. for instance, input name @xcite , eattype @xcite , food @xcite , area @xcite , familyfriendly @xcite , near[raja indian cuisine] with adv. wildwood is an indian restaurant in the riverside area near raja indian cuisine. it is not family friendly. wildwood is a restaurant providing indian food. it is located in the riverside. it is near raja indian cuisine. in the last row of to solve this problem, several methods use reinforcement learning (rl) to boost the naturalness of generated responses @xcite . however, the monte-carlo sampling process in rl is known to have high variance which can make the training process unstable. @xcite proposes to use maximum mutual information (mmi) to boost the diversity of language, but this criterion makes exact decoding intractable. on the other hand, the adversarial training for natural language generation has shown to be promising as the system needs to produce responses indiscernible from human utterances @xcite @xcite . apart from the generator, there is a dis-criminator network which aims to classify system responses from human results. the generator is trained to fool the discriminator, resulting in a min-max game between the two components which boosts the quality of generated utterances @xcite . due to the discreteness of language, most previous work on adversarial training in nlg apply reinforcement learning, suffering from high-variance problem @xcite @xcite . in this work, we apply adversarial training to utterance generation in task-oriented dialogues and propose the model advnlg. instead of using rl, we follow @xcite to leverage the straight-through gumbel-softmax estimator @xcite for gradient computation. in the forward pass, the generator uses the argmax operation on vocabulary distribution to select an utterance and sends it to the discriminator. but during backpropagation, the gumbel-softmax distribution is used to let gradients flow back to the generator. we also find that pretraining the generator for a warm start is very helpful for improving the performance. to evaluate our model, we conduct experiments on public datasets e2enlg @xcite and rnn-lg @xcite . our model achieves strong performance and obtains new state-of-the-art results on four datasets. for example, in restaurant dataset, it improves the best result by 3.6% in bleu. human evaluation corroborates the effectiveness of our model, showing that the adversarial training against human responses can make the generated language more accurate and natural. ## problem formulation the goal of natural language generation module in task-oriented dialogues is to produce system utterances directly issued to the end users @xcite . the generated utterances need to carry necessary information determined by upstream dialogue modules, including the dialogue act (da) and meaning representation (mr). the dialogue act specifies the type of system response (e.g. inform, request and confirm), while the meaning representation contains rich information that the system needs to convey to or request from the user in the form of slot-value pairs. each slot indicates the information category and each value represents the information content. therefore, the training data for the supervised nlg task is {x @xmath0, r i ), y i } n @xmath1, where d i is the dialogue act, r @xmath2, v 1 ), ..., (s t , v t )} is the set of mr slot-value pairs, and y i is the humanlabeled response. nlg models typically use delexicalization during training and inference, replacing slots and values in the utterance with a special token slot name . in this way, the system does not need to generate the proper nouns. finally, the model substitutes these special tokens with corresponding values when delivering to users. ## experiments we conduct empirical tests on a number of benchmarks for task-oriented dialogues over a variety of domains such as restaurant booking, hotel booking and retail. the datasets include the e2e-nlg task @xcite with 51.4k samples, and the tv, laptop, hotel and restaurant datasets from rnn-lg @xcite , with 14.1k, 26.5k, 8.7k and 8.5k samples respectively. we use bleu-4 @xcite for the automatic metric, computed by the official evaluation scripts from e2e-nlg and rnn-lg. ## conclusion in this paper, we propose adversarial training using the straight-through gumbel-softmax estimator in nlg for task-oriented dialogues. we also propose a two-stage training scheme to further boost the gain in performance. experimental results show that our model, advnlg, consistently outperforms state-of-the-art models in both automatic and human evaluations. in the future, we plan to apply this method to other conditional generation tasks, e.g. produce a natural utterance containing a given list of keywords. located in the riverside area near the raja indian cuisine, wildwood offers indian food and a restaurant. it is not family friendly. advnlg wildwood is an indian restaurant in the riverside area near raja indian cuisine. it is not family friendly. -adv. wildwood is a restaurant providing indian food. it is located in the riverside. it is near raja indian cuisine. nlg-lm wildwood is a restaurant providing indian food. it is located in the riverside. it is near raja indian cuisine. comment only advnlg places the two pieces of location information "riverside" and "near raja indian cuisine" together, which is aligned with human language patterns. mr name[the cricketers], eattype the cricketers, an english restaurant located near café rouge in the city centre, offers food at high price range. although it has a customer rating of 1 out of 5, it also is children friendly. advnlg the cricketers is a child friendly english restaurant in the city centre near café rouge. it has a high price range and a customer rating of 1 out of 5. -adv. the cricketers is a restaurant located in the city centre near café rouge. it is a high priced restaurant that serves english food. it is rated 1 out of 5 and is children friendly. nlg-lm the cricketers is a high priced english restaurant located in the city centre near café rouge. it has a customer rating of 1 out of 5 and is child friendly. comment advnlg model naturally put the negative aspects like "high price" and "rating 1 out of 5" together with conjunction "and", whereas both -adv. and nlg-lm juxtapose negative aspect (low customer rating) and positive aspect (kid-friendly) in one sentence, which appears contradictory. the plough is a cheap chinese restaurant located riverside by raja indian cuisine. it is a family friendly establishment. advnlg the plough is a cheap chinese restaurant in the riverside area near raja indian cuisine. it is family friendly. -adv. the plough is a cheap family friendly restaurant that serves chinese food. it is located in the riverside area near raja indian cuisine. nlg-lm the plough is a restaurant providing chinese food in the cheap price range. it is located in the riverside. it is near raja indian cuisine. comment advnlg places "chinese" immediately before "restaurant", and this is in line with the human reference. and nlg-lm model has two less connected sentences at the end.