A uto P ersuade: A Framework for Evaluating and Explaining Persuasive Arguments (2024)

Abstract: We introduce a three-part framework for constructing persuasive messages, AutoPersuade. First, we curate a large collection of arguments and gather human evaluations of their persuasiveness. Next, we introduce a novel topic model to identify the features of these arguments that influence persuasion. Finally, we use the model to predict the persuasiveness of new arguments and to assess the causal effects of argument components, offering an explanation of the results. We demonstrate the effectiveness of AutoPersuade in an experimental study on arguments for veganism, validating our findings through human studies and out-of-sample predictions.

## introduction persuasion is a common task in politics, business, government, and our daily lives. modern tools-like a/b experiments, surveys, and focus groups-are well-equipped to identify which of a pre-existing set of messages is most persuasive, but provide little insight into what about them is compelling. large language models (llms) can help to generate new plausibly persuasive messages, but they do not offer causal evidence on whether or how they have succeeded @xcite @xcite @xcite . in this paper, we introduce a new workflow for identifying the topical components of an argument that are persuasive, autopersuade. our framework assists with each step of the persuasion task. our three-step workflow is shown in figure 1 . we demonstrate this workflow in a novel study of proveganism persuasion. first, we gather persuasive arguments and responses to those arguments. these arguments can be from various sources, like social media, llms, or manual generation. with such a collection of arguments, we discuss our exemplary case study to evaluate arguments for veganism. using a forcedchoice design with arguments randomly assigned to respondents, this case study allows us to assess which arguments survey respondents report as more persuasive. while self-reported assessments of persuasion may not correspond to actual changes in behavior @xcite , our new framework is well suited for alternative settings that use behavioral measures of persuasion, which might be explored in future studies. second, with arguments and responses in hand, we introduce a new model to extract the latent features that cause arguments to be more or less persuasive. we call this model the supervised semi-non-negative (sun) topic model. the sun topic model uses an embedding representation of the arguments and builds upon matrix factorization methods to extract latent features that characterize arguments and affect responses @xcite @xcite . because our model provides interpretable output, we show how it can be used to provide easy-tounderstand and actionable insights into why particular messages are (or are not) persuasive. third, using the output from the sun topic model, we compute causal effects from varying the content of our arguments and assess the persuasiveness of future arguments. we target causal effects that determine how the average persuasiveness of an argument changes as the prevalence of a latent feature changes. this allows us to predict how persuasive a new argument would be if deployed to the same population. however, as we demonstrate through our experiments, our current method is better at answering questions about what is (and is not) persuasive than finding the optimal figure 1 : the autopersuade workflow. after generating a collection of arguments, we collect reactions from respondents. using these reactions and arguments, we fit the sun topic model that discovers latent topics that simultaneously describe the documents and their persuasiveness. we then use the output from the sun model to estimate causal effects from the existing sample and to predict the persuasiveness of new arguments. while our validation studies confirm the causal estimates of our case study, the current approach using average marginal component effects of topics appears less well-suited to finding the best argument. improving the identification of optimal arguments in the last step might be the subject of future studies. our quantity of interest is in the content of arguments that are persuasive in particular settings rather than the rhetorical strategies that are effective across settings. in this sense we are pursuing a different goal than, e.g., the pioneering work of @xcite which considers linguistic features that are persuasive in online discourse. ## the autopersuade workflow in this section, we describe the three-step autopersuade workflow (summarized in figure 1 ) in detail. ## case study: pro-veganism arguments we use a survey experiment to collect respondents' reactions to pro-veganism arguments. we ask each respondent to compare two messages and to choose the one they find more persuasive. we then summarize these pairwise contests using a bradley-terry model and use this as our response variable @xcite . finally, we use the arguments and the summarized performances to fit the sun topic model and evaluate the persuasiveness of new potential arguments. in particular, we start out with an original collection of arguments and responses followed by three validation studies to validate our estimates and explore the predictability of argument performances. validation studies 1 & 2 focused on generating and comparing synthesized and modified arguments with high persuasiveness scores to test whether we can improve on the best-performing arguments of the original argument collection. however, we find that such filtering based on the average marginal component effect of latent topics does not reliably improve arguments within the tail of best-performing arguments. validation study 3, on the other hand, validates our estimate of the average marginal component effect on argument persuasiveness by intervening on arguments across the entire distribution of topic loadings, not just a tail. ## related work the autopersuade workflow builds on a burgeoning literature that examines the causal effects of texts on outcomes @xcite . @xcite and @xcite introduce a procedure for identifying the features of texts that drive responses, but their framework relies upon a more constrained topic model and was unable to infer the persuasiveness of new texts. @xcite provide a general guide for causal inference with texts and outline a series of identification issues. @xcite experimentally demonstrate that llms can perform nearly as well as humans at producing persuasive arguments. other work has analyzed the linguistic characteristics of persuasive messages. feng and hirst (2011) categorize arguments into common schemes, while @xcite , @xcite , and @xcite examine successful arguments in online discussions, debate forums, and wikipedia, investigating the predictive power of structural features. @xcite explore personalized persuasion processes and @xcite propose a systematic taxonomy for argument quality. @xcite explore the causal effects of conversational tendencies. however, their work does not allow for domain-specific feature discovery and causal inference with these features. @xcite model the relatedness among controversial topics using embedding-based methods based on individuals' stances, integrating topic semantics from arguments and persuasion factors. currently, controllable argument generation relies upon previously identified features or domain expertise @xcite . these procedures can augment steps 1 and 3 of the workflow. ## conclusion this paper introduces autopersuade-a new workflow for persuasion. our autopersuade approach curates arguments and collects responses to those messages, identifies the latent features that cause them to be more or less persuasive, infers the causal effects of those topics, and enables the selection of more persuasive messages from a new collection of candidate messages. each step of the workflow is modular and can be improved as new technologies become available. for example, better initial curation of arguments will make data collection more efficient; other interpretable models can be used to assess why some arguments are persuasive; and we can explore targeting of messages to particular people. new techniques could use the result of data and models to automatically generate more persuasive messages.