Measuring and Benchmarking Large Language Models’ Capabilities to Generate Persuasive Language (2025)

Abstract: We are exposed to much information trying to influence us, such as teaser messages, debates, politically framed news, and propaganda — all of which use persuasive language. With the recent interest in Large Language Models (LLMs), we study the ability of LLMs to produce persuasive text. As opposed to prior work which focuses on particular domains or types of persuasion, we conduct a general study across various domains to measure and benchmark to what degree LLMs produce persuasive language - both when explicitly instructed to rewrite text to be more or less persuasive and when only instructed to paraphrase. We construct the new dataset Persuasive-Pairs of pairs of a short text and its rewrite by an LLM to amplify or diminish persuasive language. We multi-annotate the pairs on a relative scale for persuasive language: a valuable resource in itself, and for training a regression model to score and benchmark persuasive language, including for new LLMs across domains. In our analysis, we find that different ‘personas’ in LLaMA3’s system prompt change persuasive language substantially, even when only instructed to paraphrase.

## introduction we live in a time characterised by a large stream of information; including content with an inherent agenda to convince, persuade and influence readers. examples are headlines for clicks, news with political framing, political campaigns for votes or even information operations as an element of warfare @xcite . in general, we encounter a lot of text with persuasive language, which is a style of writing using rhetorical techniques and devices to influence a reader @xcite . at the same time, llms are used in various aspects of writing and communication -and the models can also be used to generate persuasive text @xcite text a i've been looking into their work and i'm impressed by their commitment to protecting children's fundamental rights, including access to healthcare, education, and a safe environment. it's a truly noble cause, don't you think? i was just doing some research on them. they help to ensure children's rights to health, education, and safety. that sounds like a good mission, don't you agree? figure 1: annotations by three workers. text a from persuasionforgood @xcite . llama3, instructed to be more persuasive, produces text b. @xcite . several studies call on the need to study and safeguard against persuasive ai @xcite , but little is known quantitatively about the capabilities of llms to generate persuasive language. we address this by measuring and benchmarking to what degree llms can amplify or diminish persuasive language when instructed to rewrite various texts to sound more or less persuasive, or when instructed to merely paraphrase text. to the best of our knowledge, we are first to measure to which degree persuasive language is diminished or amplified when llms rewrite text across different domains. we envision our results to be useful for choosing models and settings in different applications and in the mitigation of unwanted persuasive language. measuring persuasive language is challenging because the boundaries of when text is persuasive are hard to define. existing work on detecting persuasive language is domain-specific, e.g. for news and propaganda, clickbait, or persuasion for social good @xcite @xcite . instead, we propose to employ a broad definition of persuasive language across various domains, as we posit that there are commonalities in persuasive language regardless of the domain. as shown in fig. 2 , we approach our research question by constructing the dataset persuasive-pairs: we start with short texts previously tagged as exhibiting phenomena related to persuasion, such as clickbait, and paraphrase the texts using different llms to contain more or less persuasive languages. we instruct llms to change style or semantics @xcite . the pairs are then multi-annotated on an ordinal scale (marginally, moderately or heavily more persuasive); cf. fig. 1 . using this data, we train a regression model to score the relative difference in persuasive language of text pairs. the model allows us to score and benchmark new llms in different settings, e.g. varying the prompt and system prompt, and on various texts and domains, on the llm's ability to generate persuasive language. in sum, our contributions are: detecting persuasive language existing works on detecting persuasion 1) view persuasion as either problematic or beneficial @xcite , or are concerned with different 2) types of influence on either actions or beliefs, and focus on 3) specific text genres like news, debates, social media, arguments, etc. some works measure persuasion using different classification schemes of rhetorical strategies/persuasion techniques; examples are propaganda techniques in news @xcite , propaganda in social media @xcite , logical fallacies in political debates @xcite , rhetorical strategy in persuading to donate @xcite . other works measure persuasiveness based on the change in actions or behaviours; examples are outcomes regarding course selection @xcite , changing opinions @xcite or donations @xcite . yet, other research streams look at rhetorical devices as style units with figures such as rhythm, repetitions or exaggerations @xcite @xcite . closer to our paper on measuring persuasive language on a scale is the study by @xcite on measuring clickbait in social media, annotated with human perception on a 4-point scale. in argument mining, different works have measured the quality of arguments in text pairs @xcite @xcite . our research differs because we are not restricted to the structure of arguments. in general, the different lines of research discussed above are tailored to measure some form of persuasive language in specific domains or for specific aspects of persuasiveness. our paper aims to measure a broad definition of persuasive language based on human intuition, applicable to diverse domains including headlines and utterances, and independent of its intentionality, e.g. for social good or propaganda, as we posit that they have linguistic commonalities. ## benchmarking llms we benchmark different llms using different prompts and 'personas' in system prompts. we find that all models can amplify and reduce persuasive language when instructed to do so. when models are merely asked to paraphrase text, they tend to reduce the degree of persuasive language in text which is already persuasive. however, different 'personas' in the system prompt affect the degree of persuasive language in the rewritten text. setup we select 200 new text samples similar to section 3.3, paraphrasing the text using different llms and instructions. using our regression model (section 3.4) we predict a persuasion score on the pair, and compare the distributions of the scores from the different llms/setting. to statistically examine differences in the distributions, we apply the mann-whitney u rank test @xcite of whether the underlying distributions of two observation rows from pairwise settings are equal with a significant level of 0.05. further details on benchmark setup are in app. f. ## conclusion in this paper, we study the capabilities of llms to generate persuasive language by measuring the differences in persuasiveness in pairs of paraphrased short texts. we obtain annotations of the relative degree of persuasive language between text pairs and train a regression model to predict the persuasiveness score for new pairs, enabling a way to benchmark new llms in different domains and settings. we find that when prompting models to paraphrase (with no instruction on persuasiveness) as a 'default' helpful assistant, the models tend to reduce the degree of persuasive language. moreover, using different personas in the system prompts significantly affects the degree of persuasive language generated with llama3. for instance, we observe significant differences in persuasive language use depending on whether the system prompt was set as a 'right-wing' or 'centre-right' politician. our findings underline the importance of being aware of persuasive language capabilities in llms even with default system prompts.