Zero-shot Persuasive Chatbots with LLM -Generated Strategies and Information Retrieval (2024)

Abstract: Persuasion plays a pivotal role in a wide range of applications from health intervention to the promotion of social good. Persuasive chatbots employed responsibly for social good can be an enabler of positive individual and social change. Existing methods rely on fine-tuning persuasive chatbots with task-specific training data which is costly, if not infeasible, to collect. Furthermore, they employ only a handful of pre-defined persuasion strategies. We propose PersuaBot, a zero-shot chatbot based on Large Language Models (LLMs) that is factual and more persuasive by leveraging many more nuanced strategies. PersuaBot uses an LLM to first generate a natural responses, from which the strategies used are extracted. To combat hallucination of LLMs, Persuabot replace any unsubstantiated claims in the response with retrieved facts supporting the extracted strategies. We applied our chatbot, PersuaBot, to three significantly different domains needing persuasion skills: donation solicitation, recommendations, and health intervention. Our experiments on simulated and human conversations show that our zero-shot approach is more persuasive than prior work, while achieving factual accuracy surpassing state-of-the-art knowledge-oriented chatbots.

## introduction persuasive technologies are designed to reinforce, change or mold user's attitude or behavior @xcite . numerous investigations have shown that these technologies may be used effectively for a range of purposes, such as social good promotion @xcite , health intervention @xcite , marketing @xcite , and so on @xcite @xcite @xcite ). yet, automated persuasive chatbots still struggle to achieve their goal, which is to "deliver the right message, at the right time, and in the right way" @xcite , especially when there is no training data for the persuasion task. this paper proposes persuabot, a zero-shot persuasive chatbot, which is: (1) factual, (2) adaptive to the user, and (3) adaptive to the domain. ## method: persuabot as shown in figure 1 , our proposed method (per-suabot) generates a response to the user utterance by running two modules in parallel: strategy maintenance module (smm) for factual correctness while maintaining strategies and question handling module (qhm) based on user requests. no modules require domain-specific data. ## experimental setup we compare our system persuabot with a state-ofthe-art knowledge-oriented chatbot @xcite , and a manually designed persuasion chatbot for donations to save the children @xcite . all experiments are conducted in english. we use gpt-3.5-turbo-instruct and llama 3 as the base llm. ## simulated conversations experiments we conduct cost-effective experiments to evaluate our proposed method by using simulated conversations. recently, there have been experiments using powerful llms to generate training conversation data @xcite @xcite . we generate a simulated conversation by instructing the llm to converse with the chatbot with a created personality. to conduct experiments across diverse scenarios, we design "soft user" and "tough user" personalities with various challenges. tables 5 and 6 show some of the user personalities we use in our experiments. to the best of our knowledge, experiments with "tough user" scenarios have not been conducted, making them effective not only in revealing the limits of chatbots but also as a frontier perspective for the societal implementation of chatbots. we prepare 20 personalities of soft users and 20 personalities of tough users for each task. by engaging each chatbot in conversations with these users, a total of 400 simulated conversations are generated for each base llm. the quality of chatbots in these simulated conversations is evaluated by three crowdsourced workers using the evaluation metrics outlined in section 4.3. the number of claims extracted from conversations for factuality evaluation is 3,538 in the experiments with gpt-3.5 and 1,985 in the experiments with llama 3. ## conclusion we demonstrate a method for creating a factual persuasive chatbot without compromising the persuasive capabilities of llms in a zero-shot manner. the primary contribution lies in extracting strategies from responses generated by llms, searching for data to maintain those strategies, and generating factually accurate responses without compromising its strategic functionality. through experiments with both real users and simulated users, we validate the performance of this approach. persuabot demonstrates factuality surpassing state-of-the-art factual chatbots and a higher level of persuasiveness compared to existing persuasion-driven chatbots by using a much wider variety of nuanced strategies.