Refine and Imitate: Reducing Repetition and Inconsistency in Persuasion Dialogues via Reinforcement Learning and Human Demonstration (2021)

Abstract: Persuasion dialogue system reflects the machine’s ability to make strategic moves beyond verbal communication, and therefore differentiates itself from task-oriented or open-domain dialogues and has its own unique values. However, the repetition and inconsistency problems still persist in dialogue response generation and could substantially impact user experience and impede the persuasion outcome. Besides, although reinforcement learning (RL) approaches have achieved big success in strategic tasks such as games, it requires a sophisticated user simulator to provide real-time feedback to the dialogue system, which limits the application of RL on persuasion dialogues. To address these issues towards a better persuasion dialogue system, we apply RL to refine a language model baseline without user simulators, and distill sentence-level information about repetition, inconsistency, and task relevance through rewards. Moreover, to better accomplish the persuasion task, the model learns from human demonstration to imitate human persuasion behavior and selects the most persuasive responses. Experiments show that our model outperforms previous state-of-the-art dialogue models on both automatic metrics and human evaluation results on a donation persuasion task, and generates more diverse, consistent and persuasive conversations according to the user feedback. We will make the code and model publicly available.

## introduction persuasion dialogue systems have become an increasingly important subject in both social science and computational linguistics @xcite @xcite . such systems aim to employ conversational strategies to change the audience's attitude or behaviour, and therefore, are inherently difficult to build with multiple challenges. the first one is that users often expect highly smooth conversation experience from persuasion systems in order to be persuaded @xcite . so the long-standing problems of dialogue repetition and inconsistency can be especially salient in persuasion dialogue systems. secondly, different from traditional dialogue tasks, the persuasion task is non-collaborative where the user and the system have different goals @xcite , and hence highly intellectual and strategic. previous studies have attempted to address the first challenge, the dialogue repetition and inconsistency problems, by changing the object function in supervised learning @xcite or applying reinforcement learning (rl) @xcite . but these methods either may lead to uninterpretable behaviors, or rely on handcrafted user simulators that are hard to design for persuasion dialogues. to tackle these challenges, we propose to extract a policy directly from the data and let the models learn from its own mistakes without the use of simulators. leveraging decoding methods such as nucleus sampling @xcite , the finetuned language model can generate lexically diverse response candidates given the same context. some candidates are appropriate, while others are repetitive or inconsistent. these good and bad examples are used as positive and negative feedback to the model through meaningful rewards in rl, and help refine the language model. besides being diverse and consistent, a good response in persuasion dialogues also needs to accomplish the task: to persuade people. existing work simply relied on the language models to generate persuasive responses @xcite , which could result in uncontrollable task-oblivious replies. to quantify intellectual persuasion activities, we employ imitation learning, and ask human experts to demonstrate the persuasion process. we build a response imitator to imitate these human demonstrations and select the most persuasive responses in our framework. we evaluate our models on a donation persuasion task @xcite , and deploy the persuasion systems on amazon mechanical turk to interact with real users. the results on both automatic and human evaluations show that our systems achieve better persuasion outcomes (higher donation amount and donation ratio), and generates more diverse, consistent and persuasive responses compared to the baselines. this work makes multiple contributions. firstly, we propose the first rl-based persuasive dialogue system framework that achieves state-of-the-art performance on a complex donation persuasion task. secondly, we design dialgail, an rl-based generative algorithm to refine a baseline language model for dialogue generation without the use of user simulators. additionally, we introduce a human persuasion demonstration dataset that can be used for future research. previous dialogue research has mostly focused on pure task-oriented dialogues and pure social conversations; but looking forward, it becomes more and more important to pay attention to strategic dialogues that involves both task and social components. we sincerely hope this work could inspire more research and discussions on strategic dialogues in the community. ## related work strategic dialogue tasks such as persuasion and negotiation have emerged and attracted more attention recently, given its wide applications in industry and daily life @xcite @xcite @xcite . these tasks are close to human-human conversations and often contain both a specific task goal and social components to build rapport for better task completion. previously, @xcite proposed an agenda-based user-adapted persuasion system to build relationship with users and change their eating habit. @xcite developed a dialogue system for educational debate with strategic heuristics. more recently, @xcite utilized large-scale language models to build a donation persuasion system by generating multiple responses and selecting appropriate candidates with human-defined rules. we take a similar approach to generate candidates but eliminate the manual work for rule design, and teach the model to select task-relevant candidates through human demonstration. although large-scale language models have achieved great success in many nlp tasks, these models still suffer from repetition and inconsistency when applied to dialogue tasks. many previous studies have worked on these issues @xcite @xcite . for example, @xcite proposed to detect the inconsistency with natural language inference data, and penalize it with unlikelihood loss to achieve more consistent personality in open-domain dialogues. @xcite detected and rewrote the contradicting responses to achieve a more consistent personality. our work tackles these problems with rl to reduce exposure bias in supervised learning and improve the interpretability. previous work has also explored rl-based methods in dialogue system building @xcite @xcite . for instance, @xcite integrated the goal of coherent into the reward design towards more diverse dialogue generation. @xcite presented a hybrid reinforcement and imitation learning approach to enable the agent to learn from interactions with users in task-oriented dialogues. however, such methods not only rely on hand-crafted user simulators that are inherently hard to build @xcite for persuasion systems, but also require meaningful rewards that are difficult to design. in this work, we propose to let the model learn from its own mistakes by generating multiple responses without the use of simulators. our work is also closely related to response selection, which focuses on obtaining good context representations to match the context and retrieve the best response from a large collection of humanhuman conversations. however, such response selection models are highly dependent on the quality and availability of the underlying datasets. to address the data scarcity issue, @xcite pretrained a response selection model with large conversational corpora, and finetuned it on new domains in task-oriented settings for a better context representation. instead of retrieving candidates from human dialogues, we adopt the imitation learning approach, and leverage language models' ability to generate coherent responses, and build a selector to imitate human selection process and choose among the generated candidates. ## methods: persrfi our framework is shown in figure 1 . the language model is p θ and there are two steps in the framework, 1) the reinforcement learning (rl) process to refine a baseline language model q for better response generation (i.e., p θ 0 = q), and 2) the imitation process to learn from human demonstration and select the best response. during rl training, for each user utterance, p θ generates n response candidates, shown in the response candidates box. then the response detector annotates these candidates with corresponding status such as "repetition" and "inconsistency". these labels along with the golden human response provide feedback through the reward function to guide p θ to generate nonrepetitive and consistent responses. during test time, we use the refined language model p θ * to generate n candidates again, and apply the response filter to remove the repetitive and inconsistent candidates to further ensure the candidate quality. finally, the response imitator takes in the remaining candidates, and imitates the human demonstration to select one persuasive candidate as the final response. to detect repetition and inconsistency, we build usr profile and sys profile shown in the top right table in figure 1 , where task-relevant information is extracted from the dialogue and stored @xmath0: @xmath1, such as "want_to_donate: no". we describe each module below. ## conclusions persuasion dialogue system is an important topic in as it measures the machine's ability to take strategic actions in conversations towards a persuasion goal. but the current conversational systems still suffer from repetition, inconsistency and task-oblivious responses, which will hinder the persuasion success. to address these issues, we propose dialgail to refine a baseline language model and extract a policy directly from the data without user simulators by learning from its own mistakes. moreover, to better accomplish the persuasion task, we provide human demonstration for the model to imitate human persuasion activity. experiments show that our persrfi framework achieves state-ofthe-art performance in a donation persuasion task, and produces more diverse, consistent, and persuasive conversations with small amount of human efforts. looking into the future, strategic dialogues with both task and social contents will become more and more important, and it is our sincere hope that this work could inspire more research and discussion in strategic dialogue tasks such as persuasion and negotiation in the community.