Communication Makes Perfect: Persuasion Dataset Construction via Multi- LLM Communication (2025)

Abstract: Large Language Models (LLMs) have shown proficiency in generating persuasive dialogue, yet concerns about the fluency and sophistication of their outputs persist. This paper presents a multi-LLM communication framework designed to enhance the generation of persuasive data automatically. This framework facilitates the efficient production of high-quality, diverse linguistic content with minimal human oversight. Through extensive evaluations, we demonstrate that the generated data excels in naturalness, linguistic diversity, and the strategic use of persuasion, even in complex scenarios involving social taboos. The framework also proves adept at generalizing across novel contexts. Our results highlight the framework’s potential to significantly advance research in both computational and social science domains concerning persuasive communication.

## introduction persuasion techniques play a critical role in shaping societal behaviors and public opinion @xcite , which has led to sustained interest across a range of disciplines. social science research has established detailed taxonomies of persuasion strategies @xcite , while datasets have been developed to cover various domains, including charitable donations @xcite , argument ranking in debates @xcite , detecting mental manipulation @xcite , and understanding advertising strategies @xcite . despite these advances, ambiguities persist in defining persuasion @xcite , and applying persuasion strategies across different contexts remains complex @xcite @xcite . additionally, the high cost of manually annotating quality data poses a significant challenge @xcite . the advent of large language models (llms) has unlocked new possibilities for enhancing various forms of communication, including online political discourse @xcite , personalized advertising @xcite , public health messaging @xcite , and opinion shaping on social media @xcite . recent research, such as that by @xcite , has begun exploring llm-generated persuasive dialogues. however, their approach is limited to simple, two-party dialogues where a persuader seeks to change the persuadee's viewpoint. these dialogues often lack depth, presenting brief exchanges with simplistic logic and unnatural flow, restricting their usefulness for studying persuasion in more complex settings. in response to these limitations, we propose a multi-agent framework for generating persuasion data. in this framework, multiple agents are assigned distinct roles, ensuring that each aspect of the dialogue generation process is handled efficiently. this structure minimizes the risk of an agent missing important details due to task abstraction or prompt complexity, a common issue in llm prompting @xcite . additionally, auxiliary agents manage dialogue flow to ensure that the resulting exchanges are coherent, logically consistent, and incorporate diverse persuasive strategies, simulating natural human conversation. our approach imposes no preconditions regarding speakers, language styles, domains, or persuasion strategies, allowing it to generate a wide range of dialogues. for instance, our framework can support adversarial dialogues, where both participants attempt to persuade one another while maintaining their original positions. moreover, we employ a continuous labeling scheme to measure the degree of perspective change throughout the dialogue, avoiding the limitations of binary utterance labels. this framework also integrates ethical considerations, incorporating cultural norms and taboos from normbank @xcite to explore ethically challenging persuasive scenarios, such as dialogues involving manipulation or unethical persuasion. careful analyses conducted by experts from both nlp and social sciences confirm the quality of our generated dialogues, particularly in terms of their naturalness, logical structure, and diversity of persuasion strategies. our sentence-level persuasiveness labels align closely with human judgments (see @xcite . we further demonstrate the flexibility of our framework through experiments controlling for specific persuasion strategies and in more complex scenarios, such as multi-party conversations. across all tested configurations, our framework consistently produced high-quality dialogues, showcasing its adaptability and generalizability. these findings indicate that our framework offers a robust platform for studying persuasion techniques, particularly in high-stakes contexts where ethical concerns, such as the spread of misinformation and propaganda, are paramount @xcite . ## multi-agent data generation & annotation framework our framework incorporates 6 groups of language agents as shown in figure 1 . in our experiments, all agents utilize a gpt-3.5 backbone, except for the utterance quality monitor and global regulation agents which are based on gpt-4 due to their need for advanced reasoning capabilities and enhanced memory retention. note that this choice of llms aims to balance data generation costs with quality, and using more powerful models could further improve the effectiveness of our approach. our preliminary experiments on model selection are outlined in appendix a. ## data quality assessment to evaluate our data generation framework, we constructed a small dataset of 200 dialogues using randomly selected norms from normbank for human validation. these norms consist of 98 taboos, 76 normal behaviors, and 26 expected behaviors. we intentionally placed greater emphasis on taboos because these behaviors often conflict with widely accepted moral standards, causing llms to refuse to generate persuasive dialogues (figure c1 ). as such, they present a unique challenge in persuasion scenarios for both humans and llms. our data assessment plan focuses on three key aspects, progressing from more specific to broader levels of analysis: (1) the language fluency of indi- ## discussion this section presents generations of our framework in strategy-controlled and multi-party dialogues to show its flexibility and generalizability. ## conclusions this paper introduces a fully automated framework for generating persuasive dialogues, designed to address the lack of data in persuasion-related research. leveraging this framework, we generated 200 sample dialogues based on scenarios from normbank and validated them for language fluency, logical coherence, and the diversity of persuasion strategies. the results highlight our framework's ability to produce high-quality dialogues that follow human instructions. additionally, we demonstrated its flexibility in handling controlled persuasion strategies and its adaptability to more complex, multi-party conversations. this framework offers significant potential for advancing persuasion research in both computer science and social sciences domains.