Counterfactual Off-Policy Training for Neural Dialogue Generation (2020)

Abstract: Open-domain dialogue generation suffers from the data insufficiency problem due to the vast size of potential responses. In this paper, we propose to explore potential responses by counterfactual reasoning. Given an observed response, the counterfactual reasoning model automatically infers the outcome of an alternative policy that could have been taken. The resulting counterfactual response synthesized in hindsight is of higher quality than the response synthesized from scratch. Training on the counterfactual responses under the adversarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model as well as the conventional adversarial learning approaches.

## introduction open-domain dialogue generation @xcite @xcite intends to produce coherent responses given dialogue history. nevertheless, it suffers from data insufficiency problem as there may exist many potential responses for a given dialogue history @xcite . an ideal way of exploring the potential responses is to train the model by chatting with real users, which is usually time-consuming and labor-intensive in practice. although replacing a real user with a user simulator could address the issue, the simulator only roughly approximates real user statistics, and its development process is costly @xcite . in contrast, humans could independently reason potential responses based on past experiences from the true environment. having observed a response, one might naturally ask himself or herself: "what would happen if i respond differently, while everything else in the environment remains the same." answering the question will result in a potential response (as an example in figure 1 ), and it is beneficial for improving future decision making @xcite . the potential response inferred in hindsight is called a counterfactual response, where the concept "counterfactual" describes the posterior process of reasoning the outcome of alternative actions (i.e., a different responding policy) that could have been taken while keeping everything else unchanged @xcite . motivated by this, we propose a counterfactual off-policy training (copt) approach to explore potential responses. building upon the adversarial learning framework, copt casts a dialogue generator as a structural causal model (scm), which describes a generation process with two ingredients: scenarios and causal mechanisms @xcite . the scenario is a random noise variable that captures all unobserved yet relevant aspects of the environment, i.e., user profiles. the causal mechanism is a deterministic function that takes a scenario and dialogue history as input and outputs a response. in this way, reasoning a counterfactual response in an observed response's environment can be achieved by feeding the scenario of the observed response into the causal mechanism. after generating the counterfactual response, the generator will receive a reward from a discriminator and optimize itself accordingly. intuitively, a counterfactual response is synthesized by grounding the model in the scenario where an observed response occurs, rather than the scenario sampled from scratch as standard adversarial learning-based approaches. this improves the quality of the synthesized responses and subsequently benefits the model that learns from the synthesis. to verify the effectiveness of our approach, we conduct experiments on the public available dailydialog dataset @xcite . experimental results show that our approach significantly outperforms previous adversarial learning-based approaches in both automatic and human evaluations. the contributions of this paper are summarized as follows: responses of retrieval-based methods come from a fixed candidate response set and thus are incapable of being customized. the generation-based methods can create new responses, but the vanilla sequence to sequence model tends to produce generic responses @xcite . one way to address the generic response problem is by introducing external knowledge, such as keywords @xcite , topics @xcite , persona information @xcite , and retrieved candidate responses @xcite @xcite . another way is to optimize the architecture of networks. there are two architectures widely employed in this research line: the variational auto-encoder @xcite and the generative adversarial network @xcite @xcite @xcite . our approach falls into the latter category. the differences between our approach and other adversarial learning-based approaches are as follows. first, we cast the dialogue generation model as an scm to explore potential responses in the environment where observed responses occur. second, we learn on counterfactual responses that inferred from the scm. third, a pre-trained behavior policy is involved during the generation process, making our approach an off-policy algorithm and benefits the exploration of potential responses. counterfactual reasoning the counterfactual reasoning is a concept derived from psychology. it describes the human capacity to learn from experience by reasoning the outcome of an alternative action that could have been taken @xcite . combined with the scm, counterfactual reasoning improves the performance of policy evaluation in reinforcement learning @xcite . in the area of nlp, counterfactual reasoning in previous work is mainly used for data augmentation @xcite @xcite , which rewrites the original data given a counterfactual label or condition. in this paper, we connect the concept of counterfactual reasoning with the dialogue generation and are the first to cast a generation model as an scm under the adversarial learning framework. ## method we cast a dialogue generation model as an scm to explore potential responses by counterfactual reasoning during the training process. we will first review the concept of the scm (sec. 3.2), and then introduce our copt approach (sec. 3.3). ## conclusion we propose a model-agnostic approach, copt, that can be applied to any adversarial learning-based dialogue generation models. in contrast to existing approaches, it learns on counterfactual responses inferred from the structural causal model, taking advantage of observed responses. this helps the model to explore the high-reward area of the potential response space. experiments show that the copt significantly improves the quality of the generated responses, which demonstrates the effectiveness of this approach.