Seamlessly Integrating Factual Information and Social Content with Persuasive Dialogue (2022)

Abstract: Complex conversation settings such as persuasion involve communicating changes in attitude or behavior, so users’ perspectives need to be addressed, even when not directly related to the topic. In this work, we contribute a novel modular dialogue system framework that seamlessly integrates factual information and social content into persuasive dialogue. Our framework is generalizable to any dialogue tasks that have mixed social and task contents. We conducted a study that compared user evaluations of our framework versus a baseline end-to-end generation model. We found our model was evaluated to be more favorable in all dimensions including competence and friendliness compared to the baseline model which does not explicitly handle social content or factual questions.

## introduction persuasive dialogue systems are designed for chatbots to communicate with and to influence users with specific goals. such systems are often designed to benefit individual users (e.g., promoting healthy behaviors) or society at large (e.g., persuading people to make donations). @xcite introduced this idea with the persuasion-forgood dataset, which contains 1,017 humanhuman conversations where one participant persuaded the other to donate to the charitable organization save the children 1 , with 300 conversations having sentence-level dialogue act annotations. the social and communicative dynamics behind persuasive conversation contexts are complex. a persuasive conversation by definition involves one party, the persuader, intending to change the attitude or behavior of the other party, the persuadee @xcite . changing persuadees' attitude has several dimensions including establishing mutual trust and credibility, strategically presenting persuasive appeals, 1 https://www.savethechildren.net/ human: yeah, a lot of developing countries don't have systems or infrastructure to protect children from these causes. that's why organizations like save the children are so important. baseline: i am going to donate a bit of my money to save the children. rap: i agree. there is a lack of support for children in developing countries, especially in war zones. it's so hard to imagine what it's like for a child to grow up facing the daily threat of violence. figure 1 : chatbot running on the baseline bart model and chatbot running on rap responding to the same user utterance. the baseline model does not appropriately acknowledge the user's statement, whereas rap is able to show acknowledgement and respond appropriately. and eliciting emotional reactions from the persuadee ( @xcite . moreover, grice's maxims of conversation define conversations as a cooperative and collaborative process @xcite @xcite . thus, effective and successful persuasive conversations do not mechanically relay taskrelated information to the persuadee. there has to be a significant exchange of social and emotional content to empathetically address persuadees, e.g. by answering specific questions and developing positive relationships throughout the conversation. for this reason, persuasive conversations are not strictly task-oriented, but are built around tasks with additional social conversational strategies. in essence, persuasive conversations have two goals: one that is task-oriented to elicit behavioral changes, and another that is social-oriented to build trust and empathy and develop positive relationships in order to better navigate the persuasive context. in this work, we propose the response-agenda pushing framework (rap) for persuasive dialogue systems, which can explicitly handle these two goals. rap jointly addresses social response and task-oriented dialogue generation. in a given turn, rap first focuses ond appropriately triggering modules to generate answers to factual questions and social responses to address users' comments. rap then pushes the persuasive agenda of a conversation using a language model that conditions on individual persuasive appeals. compared to state-of-the-art end-toend conditional generation models, rap is more semantically coherent and persuasive, while being generalizable to any dataset annotated with dialogue acts. in addition, we tackle the challenge of multiple-sentence conditional generation in a single turn given specific pragmatic argumentative strategies (e.g., "emotional appeal"). concretely, our contributions are threefold. contrary to recent work which attempts to transition from social to task-oriented dialogue @xcite , we blend social and task-oriented dialogue in an approach grounded in social science theory postulating the need for social acknowledgement in the midst of advancing conversational goals @xcite @xcite @xcite . additionally, we present an account of conditional generation on fine-grained pragmatic persuasive strategies, unlike earlier attempts using looser semantic controls @xcite @xcite . finally, we present a qualitative account of rap, including individual anecdotes of its strengths and weaknesses. overall, we present a novel perspective on persuasive dialogue, marking important progress towards intelligent persuasive agents. ## related work much earlier work in persuasion-like social conversations has been towards building dialogue systems for negotiation tasks, e.g., using the craigslist bargaining @xcite and deal or no deal datasets @xcite . however, in negotiation tasks, the goal is to come to a consensus, whereas in persuasion tasks, the target result is a one-sided change or a "win" for the persuader, as in a debate. recently, there has been increasing interest in persuasive dialogues because of the rise in online-mediated persuasion scenarios (e.g. online sales, health promotion, political debates); much work focuses on understanding the social dynamics behind online persuasive conversations on social media platforms like reddit (e.g. atkinson et al. (2019); musi (2018); srinivasan et al. (2019); tan et al. (2016) ). in addition, a burgeoning line of work has been invested in developing chatbots to deliver healthcare remotely and to persuade people to adopt healthier lifestyles @xcite . such efforts have inspired a growing body of work towards building persuasive dialogue systems that are conditional, strategic and factual to benefit individuals and society at large. many early iterations of persuasive dialogue systems have used template-based @xcite or retrieval-based @xcite utterance generation methods. @xcite introduced persuasionforgood and proposed designing a personalized persuasive dialogue system. @xcite used two pretrained language models to separately models both speakers in a conversation, finding success in creating human-like utterances without supervision (from human annotations). other studies propose end-to-end neural generation models @xcite . however, in approaches solely performing language modeling, there is less semantic control over generated utterances; they are not guaranteed to follow a particular persuasive strategy or dialogue act. beyond persuasion, conditional text generation has emerged as a popular method of controllable generation for more coherent and "harmonious" human-dialogue system interactions @xcite . much earlier work in sentence-level conditional text generation has facilitated control by conditioning on entire topic statements @xcite or simple semantic codes @xcite @xcite . while such approaches work well in chit-chat, they do not guarantee strategy execution for complex tasks. we propose using conditional generation conditioned on pragmatic dialogue acts to specifically control the strategic flow of a persuasive conversation. much existing work in persuasion tasks has focused on strategy/policy planning @xcite @xcite @xcite domain social dialogue systems should form a consistent personality to develop users' trust, satisfy the human need for affection and social belonging, and generate interpersonal responses @xcite @xcite suitable for any input @xcite . consistent with this need for affection and acknowledgement, zhang and danescu-niculescu-mizil (2020) find that in crisis counseling, it is necessary to balance the goals of both "empathetically addressing the crisis situation" and "advancing the conversation towards a resolution." additionally, @xcite improved engagement with taskoriented dialogues by adding "chit-chat." this suggests that balancing the need for human acknowledgement with advancing towards conversational goals may improve persuasion outcomes. very recent work has made progress by transitioning from chit-chat to task-oriented dialogue @xcite . however, to truly achieve this balance, we propose interweaving social content with pushing a conversational agenda in order to improve coherence, friendliness, and persuasiveness. retrieval-based dialogue systems have long been considered one of the core classes of conversational systems @xcite , often being used for question answering systems @xcite due to their ability to return "fluent and informative responses" @xcite . but, recent work has been able to directly improve their opendomain dialogue systems by ensembling both retrieval methods (e.g., database queries) with neural generation methods @xcite @xcite . thus, we propose retrieving factual information to improve a persuasive dialogue system's ability to consistently and coherently address user questions, which may lead to improved perceptions of intelligence, coherence, and trustworthiness. ## dataset we use the 300 annotated anonymous english conversations in the persuasionforgood dataset. in each conversation, one person, the "persuader," tries to convince their conversational partner, the "persuadee," to donate to save the children. the conversations last for 10 turns, and a user's utterance during a turn contains at least one sentence. each sentence is annotated with one of several dialogue acts, including inquiries (e.g. "have you donated to a charity before?") and various persuasive appeals (e.g. "i'll match your donation, and together we can double the amount!"). in this work, we build a system that acts as a persuader. the full list of persuader dialogue acts used is provided along with examples in ## evaluation we evaluate rap against an end-to-end fine-tuned bart model as described in section 4.3.2. this allows us to directly evaluate the impact of integrating factual information and social content and persuasive strategies in contrast to a conversation only driven by persuasive strategies. we evaluate the performance of the conditional generation model by calculating the dialogue act accuracy on a withheld test set consisting of 10% of all conversations. as language generation is nondeterministic, we average the dialogue act accuracy across ten passes. we chose bart over blenderbot in the persuasive agenda-pushing module because we additionally compare utterance-based proxies for user engagement in ## experimental setup we deployed our chatbot using the legoeval platform @xcite . the chatbot is given a genderneutral name, jaime. the task consists of a pre-task survey, a conversation where each participant responds to the chatbot with a minimum of seven and maximum of ten conversational lines, and a posttask survey. the pre-task survey consists of questions about demographic information (e.g., age, gender, income) and a test of the big five personality traits @xcite . the post-task survey asks participants about their conversation experiences. it includes an attention validation question ("what charity was the chatbot talking about?") then asks about the users' intention to donate to save the children and their perception of the chatbot, including evaluations on various traits such as perceived competence and warmth. the full lists of questions is outlined in ## results in this section, we discuss the results of comparing rap and baseline only using bart, the impact of individual components of rap , and qualitatively examine participant case studies. ## discussion overall, we find that rap and each of its individual modules is able to outperform state-of-the-art conditional generation models on persuasion-forgood . one of the core advantages of endto-end conditional generation models is that they are easily transferrable to different datasets. but, rap is also easily transferrable -the only requirement is that the dataset contains a set of dialogue acts with sufficient data to train a classifier, as the biggest bottleneck is being able to use a dialogue classifier for l p and in building the dispatcher. on smaller datasets, it may even be possible to perform transfer learning using a classifier pre-trained on the persuasionforgood dataset. the social response module is directly transferrable, as we are able to achieve high quality results using it zero-shot, and the factual answer module uses sentence-bert to group together training data. limitations due to the cost of human evaluation, our sample size is relatively small, 51 and 57 people for the two conditions. this limitation restricted us from performing a full ablation in which we evaluated chatbots which used each module individually. we hope to obtain larger samples in the future to better evaluate the efficacy of our system. additionally, considering the sample consists of students enrolled in natural language processing, they possess a more technical background with higher standards for chatbots than the average user on mechanical turk. moreover, because the sample did not enter as participants out of personal interest in save the children, they are less likely to be interested in childrens' charities than an individual on the internet who goes out of their way to interact with such a chatbot, which may be reflected in evaluation scoring. anecdotally, we see in section 7.3 that individuals who do have some sort of inclination towards charitable organizations are actually quite positive and receptive towards the chatbot. in this regard, we are likely limited by the funds necessary to acquire a sample whose interests better align with persuasionforgood. our work faces several challenges to ultimately evaluate the hypothesis that persuasive conversations should be handled as jointly social and task-oriented. while the dialogue act accuracy of the agenda-pushing module is only 62.3%, this metric is bottlenecked by f dc in equation 3; the f1-score of the classifier is only 0.66 (the state-of-the-art on the persuasionforgood dataset), implicitly limiting the upper bound of any generation model that is reliant on it. we find from users' conversation experiences that the chatbot more than sufficiently presents persuasive strategies. if one has a dialogue act classifier with stronger performance, they would be able to improve the ability of their agenda-pushing model to learn persuasive strategies even further. we additionally find that without a dialogue act classifier (i.e., without l p ), bart is unable to achieve a dialogue act accuracy higher than 30% on the persuasionforgood dataset. ## conclusion overall, we find perceptual improvements by specifically integrating social content and factual information into persuasive dialogues with rap compared to a strong end-to-end conditional generation model like bart. while existing methods like @xcite ; @xcite achieve strong performance on automatic metrics like perplexity, rap directly emphasizes upon users' conversational experience with a modular design rooted in social science theory. rap is generalizeable and may even be applied towards persuasive contexts outside of charitable conversations, e.g., in the case of therapy and crisis counseling @xcite where there are also split goals (ensuring users feel heard and pushing a conversational agenda). future work on persuasive dialogue systems could consider implementing a strategy planner using supervised learning. additionally, researchers could consider looking for relationships between personality data, persuasive strategies, and persuasion outcomes.