In this paper , drawing intuition from the Turing test , we propose using adversarial training for open-domain dialogue generation : the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances . We cast the task as a reinforcement learning ( RL ) problem where we jointly train two systems , a generative model to produce response sequences , and a discriminator-analagous to the human evaluator in the Turing test-to distinguish between the human-generated dialogues and the machine-generated ones . The outputs from the discriminator are then used as rewards for the generative model , pushing the system to generate dialogues that mostly resemble human dialogues . In addition to adversarial training we describe a model for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric , while avoiding a number of potential pitfalls . Experimental results on several metrics , including adversarial evaluation , demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines . Open domain dialogue generation ( Ritter et al . , 2011 ; Sordoni et al . , 2015 ; Xu et al . , 2016 ; Wen et al . , 2016 ; Li et al . , 2016b ; Serban et al . , 2016c Serban et al . , , 2017 ) ) aims at generating meaningful and coherent dialogue responses given the dialogue history . Prior systems , e.g. , phrase-based machine translation systems ( Ritter et al . , 2011 ; Sordoni et al . , 2015 ) or end-to-end neural systems ( Shang et al . , 2015 ; Vinyals and Le , 2015 ; Li et al . , 2016a ; Yao et al . , 2015 ; Luan et al . , 2016 ) approximate such a goal by predicting the next dialogue utterance given the dialogue history using the maximum likelihood estimation ( MLE ) objective . Despite its success , this over-simplified training objective leads to problems : responses are dull , generic ( Sordoni et al . , 2015 ; Serban et al . , 2016a ; Li et al . , 2016a ) , repetitive , and short-sighted ( Li et al . , 2016d ) . Solutions to these problems require answering a few fundamental questions : what are the crucial aspects that characterize an ideal conversation , how can we quantitatively measure them , and how can we incorporate them into a machine learning system ? For example , Li et al . ( 2016d ) manually define three ideal dialogue properties ( ease of answering , informativeness and coherence ) and use a reinforcement-learning framework to train the model to generate highly rewarded responses . Yu et al . ( 2016b ) use keyword retrieval confidence as a reward . However , it is widely acknowledged that manually defined reward functions ca n't possibly cover all crucial aspects and can lead to suboptimal generated utterances . A good dialogue model should generate utterances indistinguishable from human dialogues . Such a goal suggests a training objective resembling the idea of the Turing test ( Turing , 1950 ) . We borrow the idea of adversarial training ( Goodfellow et al . , 2014 ; Denton et al . , 2015 ) in computer vision , in which we jointly train two models , a generator ( a neural SEQ2SEQ model ) that defines the probability of generating a dialogue sequence , and a discriminator that labels dialogues as human-generated or machine-generated . This discriminator is analogous to the evaluator in the Turing test . We cast the task as a reinforcement learning problem , in which the quality of machinegenerated utterances is measured by its ability to fool the discriminator into believing that it is a human-generated one . The output from the discriminator is used as a reward to the generator , pushing it to generate utterances indistinguishable from human-generated dialogues . The idea of a Turing test-employing an evaluator to distinguish machine-generated texts from human-generated ones-can be applied not only to training but also testing , where it goes by the name of adversarial evaluation . Adversarial evaluation was first employed in Bowman et al . ( 2016 ) to evaluate sentence generation quality , and preliminarily studied for dialogue generation by Kannan and Vinyals ( 2016 ) . In this paper , we discuss potential pitfalls of adversarial evaluations and necessary steps to avoid them and make evaluation reliable . Experimental results demonstrate that our approach produces more interactive , interesting , and non-repetitive responses than standard SEQ2SEQ models trained using the MLE objective function . In this paper , drawing intuitions from the Turing test , we propose using an adversarial training approach for response generation . We cast the model in the framework of reinforcement learning and train a generator based on the signal from a discriminator to generate response sequences indistinguishable from human-generated dialogues . We observe clear performance improvements on multiple metrics from the adversarial training strategy . The adversarial training model should theoretically benefit a variety of generation tasks in NLP . Unfortunately , in preliminary experiments applying the same training paradigm to machine translation , we did not observe a clear performance boost . We conjecture that this is because the adversarial training strategy is more beneficial to tasks in which there is a big discrepancy between the distributions of the generated sequences and the reference target sequences . In other words , the adversarial approach is more beneficial on tasks in which entropy of the targets is high . Exploring this relationship further is a focus of our future work . Zhou Yu , Ziyu Xu , Alan W Black , and Alex I Rudnicky . 2016b . Strategy and policy learning for nontask-oriented conversational systems . In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue . page 404 . Yuan Zhang , Regina Barzilay , and Tommi Jaakkola . 2017 . Aspect-augmented adversarial networks for domain adaptation . arXiv preprint arXiv:1701.00188 .

Challenge: Existing approaches to replace the over-simplified objective for open domain dialogue generation use manually defined reward functions which cannot cover all crucial aspects.
Approach: They propose the Turing test-inspired adversarial training scheme where the generative model is trained to fool the discriminator as a reinforcement learning problem.
Outcome: The proposed model can generate more interactive, interesting, and non-repetitive responses than previous baselines on several metrics including adversarial evaluation.