Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training (2021)

Abstract: In this paper, we propose Inverse Adversarial Training (IAT) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better. In contrast to standard adversarial training algorithms, IAT encourages the model to be sensitive to the perturbation in the dialogue history and therefore learning from perturbations. By giving higher rewards for responses whose output probability reduces more significantly when dialogue history is perturbed, the model is encouraged to generate more diverse and consistent responses. By penalizing the model when generating the same response given perturbed dialogue history, the model is forced to better capture dialogue history and generate more informative responses. Experimental results on two benchmark datasets show that our approach can better model dialogue history and generate more diverse and consistent responses. In addition, we point out a problem of the widely used maximum mutual information (MMI) based methods for improving the diversity of dialogue response generation models and demonstrate it empirically.

## introduction in recent years, neural end-to-end dialogue response generation models @xcite @xcite has gained increasing popularity with the recent advancements of neural sequence-to-sequence (seq2seq) learning models @xcite . while neural dialogue models can generate seemingly fluent responses, due to the over-simplified maximum likelihood estimation (mle) training objective and the high frequency of generic responses in training corpora, they tend to produce dull and generic responses such as "i don't know" much more often than that humans generally do @xcite , which makes dialogue agents less engaging and ineffective. in addition, recent research on whether neural dialogue systems use dialogue history effectively @xcite shows that most neural dialogue agents fail to take the dialogue history into account when generating responses. this problem makes neural dialogue systems tend to generate responses irrelevant to the current topic of the conversation and are not consistent with the dialogue history. this problem may also intensify the generic response problem, as dull responses are generally off-topic and irrelevant to the dialogue history. to address the above issues, in this paper, we propose inverse adversarial training (iat) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better, thus generating diverse and informative responses. conventional adversarial training methods generally generate label-preserving adversarial inputs with carefully designed methods and train the model to generate the same output to enhance the model's robustness. in contrast, our approach perturbs in input dialogue history such that a good dialogue model should not generate the same output if the output is non-generic and relevant to the dialogue history. we name our proposed method as inverse adversarial training because it is related to conventional adversarial training methods which aim to improve the model's adversarial robustness but our proposed objective is motivated in the opposite direction. note that our work is not directly related to textgans as well as their applications on dialogue response generation. specifically, the proposed inverse adversarial training assigns higher rewards to generated responses or ground-truth responses if their likeli-hood decreases more when the dialogue history is perturbed, and penalize the model when it generates responses whose likelihood is almost unchanged given either original or perturbed dialogue history as input. this encourages the model to generate more relevant and informative responses and capture dialogue history better. the proposed iat algorithm can be used in both supervised and self-supervised fashion (with/without reference response), which can be viewed as a form of reward-augmented maximum likelihood (raml) method @xcite that improves the original mle objective or a rewarding scheme for rl-based text generation algorithms. the inverse adversarial learning framework is also conceptually related to self-adversarial learning @xcite where the the comparison is made between different checkpoints of the same model to provide reward for rl training of the nlg model. in addition, we identify a limitation of the widely-used maximum mutual information (mmi) based methods for improving the diversity of dialogue response generation models. this will be discussed in detail in section 2.1 and empirically demonstrated in section 4.2. we conduct experiments on two dialogue datasets, opensubtitiles and dailydialog, to demonstrate the effectiveness of the proposed approach. experimental results show iat helps neural dialogue systems model dialogue history better and generate more diverse and informative responses. ## inverse adversarial training in this section, we describe the proposed inverse adversarial training algorithm in detail. we first describe how we perturb the dialogue history and then formally introduce the inverse adversarial training algorithm. ## experiments to validate the effectiveness of the proposed inverse adversarial training algorithm, we conduct experiments in order to answer the following two research questions: ## conclusion in this work, we introduce inverse adversarial training (iat) algorithm that is able to simultaneously reduce the dull response problem and help neural dialogue systems model dialogue history better. iat measures the relevance and consistency of responses by the difference of their likelihood conditioning on either original and perturbed dialogue history. in this way, it is able to prevent the dialogue system from preferring generic responses, even they are often of high frequency in the training corpora. our method also encourages the dialogue agent to model dialogue history better by penalizing the model when generating the same responses given perturbed dialogue history. experimental results on two benchmark datasets show that the proposed inverse adversarial training algorithm helps dialogue models capture dialogue history better and generate more diverse and consistent responses. we also identify a limitation of the widely-used mmi based methods for improving the diversity of dialogue response generation models and empirically demonstrate the existence of this problem through our experimetns.