Do Emotions Really Affect Argument Convincingness? A Dynamic Approach with LLM -based Manipulation Checks (2025)

Abstract: Emotions have been shown to play a role in argument convincingness, yet this aspect is underexplored in the natural language processing (NLP) community. Unlike prior studies that use static analyses, focus on a single text domain or language, or treat emotion as just one of many factors, we introduce a dynamic framework inspired by manipulation checks commonly used in psychology and social science; leveraging LLM-based manipulation checks, this framework examines the extent to which perceived emotional intensity influences perceived convincingness. Through human evaluation of arguments across different languages, text domains, and topics, we find that in over half of cases, human judgments of convincingness remain unchanged despite variations in perceived emotional intensity; when emotions do have an impact, they more often enhance rather than weaken convincingness.We further analyze whether 11 LLMs behave like humans in the same scenario, finding that while LLMs generally mirror human patterns,they struggle to capture nuanced emotional effects in individual judgments.

## introduction emotional appeals have long been recognized as a core component of persuasion @xcite . aristotle's triad of logos, ethos, and pathos (aristotle and kennedy @xcite , 1991) emphasizes the multifaceted nature of effective rhetoric. while logical reasoning (logos) and the speaker's credibility (ethos) are essential, the ability to evoke emotions in the audience (pathos) may also be crucial in order to make the audience more receptive to the arguments @xcite . despite active research on argumentation and argument quality in the nlp community (e.g. @xcite @xcite @xcite common examples that we have heard include elephant rides and swimming with dolphins experiences. for these actions to take place, there are cruel training techniques, coercive control, conditioned unnatural behaviours, stressful interactions and so on. n: [...] the government will have the ability, through supporting the bill, to bring forward future targeted bans on lowwelfare activities involving animals. [...] figure 1 : an example test case. e is an argument with emotions and n is an argument without emotions, both addressing the same topic with the same stance. g -(e) is a counterpart of e with reduced emotion. we compare the convincingness ranking of the pair (e, n) to that of the pair (g -(e), n) to observe the effect of emotions on argument convincingness in a dynamic way. 2017; @xcite , the pathos dimension has received undeservedly little attention @xcite ; emotional appeal is often discussed as a logical fallacy in arguments (e.g., @xcite @xcite . existing nlp studies exploring the interplay between emotions and argument convincingness often lack a specific focus on the emotional dimension and fail to control for confounding factors (e.g. @xcite @xcite . a confounder refers to a variable that influences both the independent variable (the factor being manipulated: emotions) and the dependent variable (the outcome being measured: convincingness), potentially distorting the observed relationship between them. to address this gap, we propose a dynamic approach that systematically varies emotional intensity to observe its effect on argument convincingness, following the logic of psychological manipulation checks @xcite . here, emotional intensity is the manipulated variable and convincingness the dependent one. we call it dynamic because it captures the effect of varying emotional intensity on convincingness, moving beyond static comparisons between fixed argument pairs. to achieve this, we leverage llms to rephrase an argument to generate a counterpart that evokes stronger/weaker emotions, and then compare its convincingness to the original argument, thereby minimizing the effect of confounders. the judgments are evaluated relative to an anchor argument ( §5.1), as illustrated in figure 1 , to obtain more reliable subjective human evaluation @xcite @xcite . this framework enables us to examine how variations in perceived emotional intensity influence judgments of convincingness for a given argument in a controlled manner. to test the robustness of our findings across annotation setups, we further use likert-scale ratings of argument convincingness as a simpler alternative to the relative, anchorbased evaluation ( §5.2). although absolute ratings are generally considered less reliable for such subjective evaluation, our observations remain consistent, demonstrating the robustness of both our dynamic approach and findings. besides, we move beyond prior studies that focus predominantly on english arguments or singledomain datasets @xcite @xcite . we expand the scope to explore both english and german arguments across diverse text domains, including political debates, online portals, and curated human-written arguments. our multilingual and cross-domain analysis provides a comprehensive view of how perceived emotional intensity affects convincingness across different contexts. finally, inspired by recent studies exploring cognitive biases in llms @xcite @xcite , we further investigate whether llms behave like humans when judging argument convincingness under the influence of emotional 'bias'. although emotion is not always considered a fallacy or bias in argumentation @xcite @xcite , understanding its impact on argument evaluation is crucial for developing models intended for automated argument evaluation (e.g., @xcite @xcite . our contributions are: foot_0 • we propose a novel framework to analyze how emotions influence perceived convincingness in a controlled manner. our findings show that in over half of cases, human judgments remain unaffected by emotional intensity, while emotions more often enhance rather than weaken convincingness. ## related work this work primarily connects to (1) the interplay between emotions and argument convincingness, while also relating to ( 2 ) human-like biases in llms. emotion vs. convincingness emotions have been shown to play a role in argument convincingness in both fields of computational argumentation (e.g. @xcite @xcite and philosophy/psychology (e.g. aristotle and kennedy @xcite @xcite . in nlp, emotional appeal is primarily studied within the context of logical fallacy in arguments @xcite or as a secondary focus in relation to argument convincingness @xcite . the most relevant works include: @xcite find that human annotators identify emotional aspects as positively contributing to argument convincingness. @xcite introduce an emotional appeal layer in a modified toulmin argumentation model, showing that 6% of arguments are purely emotional. @xcite analyze arguments across 15 dimensions, finding a weak positive correlation between emotional appeal and convincingness. lukin et al. (2017) demonstrate that audience-specific factors improve belief change prediction, particularly for emotional arguments. @xcite examine specific emotions, showing that arguments expressing joy and pride are rated as more convincing, while those expressing anger are rated as less convincing. previous studies rely on fixed analyses that do not control for confounders. in contrast, we adopt a dynamic approach, controlling for confounding factors and examining how perceived convincingness changes with varying emotional intensity. our methodology aligns with psychological manipulation checks @xcite , treating emotional intensity as the manipulated variable and convincingness as the dependent variable. additionally, prior work has largely focused on english, except for greschner and klinger (2024), who examine german arguments. since emotional effects may vary across cultures, we study both english and german arguments. we also expand the scope by incorporating diverse text domains, including political debates, online portals, and curated human-written arguments, unlike previous studies limited to a single domain. human-like biases in llms an array of studies has demonstrated human-like biases in llms (e.g., @xcite @xcite . social biases, such as sentiment, stereotype, and gender biases, have been extensively investigated (e.g., @xcite @xcite . beyond social biases, llms also mimic human cognitive biases in reasoning and decision-making @xcite @xcite @xcite @xcite . for instance, lampinen et al. @xcite show that llms, like humans, perform better when task semantics align with logical inference ('content effect'). similarly, echterhoff et al. (2024) find llms exhibit decision-making biases such as anchoring bias @xcite , status quo bias @xcite , and framing bias @xcite . meanwhile, @xcite analyze llms' irrationality across 12 cognitive tasks @xcite , revealing both human-like errors and distinct deviations. although emotional appeal is not inherently a bias or fallacy but a persuasion strategy, it is crucial to examine whether llms' preferences align with human judgments, especially given their growing role in argument evaluation (e.g., @xcite @xcite . inspired by studies on cognitive biases in llms, we investigate whether llms exhibit human-like behavior in how emotional intensity influences argument convincingness. ## evaluation setup we employ a dynamic framework to explore how the intensity of emotions evoked in readers impacts their judgments of convincingness. in this work, we treat emotional intensity as the overall strength of emotions felt by readers, without considering specific emotions. we follow previous works @xcite to leverage pairwise comparisons for evaluation because it yields more reliable annotations compared to the absolute ratings, especially for such subjective evaluation tasks @xcite @xcite . our setting is as figure 1 shows: among one pair of arguments that share the same stance on a given topic but differ in their content, e is (set up to be) emotion-evoking, while n does not (typically) evoke emotions. we then use llms to generate a counterpart argument for e, g -(e), which retains the same meaning as e but evokes less emotion. to inspect how perceived argument convincingness is affected by emotions, we compare the convincingness ranking of (g -(e),n) to that of the original pair (e,n). the reason to not compare the arguments with a similar content, i.e., e vs. g -(e) is that we want to minimize the effect of human's prior belief about whether emotions should contribute to argument convincingness. analogously, we generate a counterpart for n, g + (n), with in- the goal of g + /g -is to maintain the core meaning of the argument while modifying its emotional appeal. although humans could be used to create such counterparts (e.g. @xcite , this approach is largely impractical at scale because it is costly. instead, we use llms to efficiently generate required variations and assess their ability to perform this task through human evaluation. thus, for each original argument pair (e, n ), we create three counterpart pairs with varying levels of emotional intensity, resulting in a total of four argument pairs per test instance. the original argument pair serves as the anchor, from which we see how the convincingness rankings of the other argument pairs change. we list all possible change scenarios in the first row in the table presents all possible convincingness rankings of the original argument pair (e,n). the subsequent rows show the convincingness rankings of the counterpart argument pairs where the emotional intensity of the argument on the left has been reduced (g -(e),n), that of the argument on the right has been increased (e,g + (n)), or both (g -(e),g + (n)). cells highlighted in green indicate cases where the convincingness of the left argument decreases as its emotional intensity de-creases relative to the right argument, suggesting a positive impact of emotions on convincingness. this occurs when the convincingness ranking shifts from the left @xmath0, or from @xmath1. conversely, cells highlighted in red indicate cases where the convincingness of the left argument increases as its emotional intensity decreases relative to the right argument, reflecting a negative impact of emotions on convincingness. finally, cases where the convincingness rankings remain consistent retain a white background. metrics for each instance (e, n ), we calculate the percentages of consistent, positive, and negative cases. we then average the percentages of each category across all test instances to derive three metrics that indicate the overall frequencies of the three categories in humans. we call the metrics: consistency rate, positivity rate, and negativity rate. their formulas are as follows: ## dataset construction we source 50 anchor argument pairs from each of five datasets ( §4.1) and utilize gpt4o foot_1 to generate their counterparts with variations in emotional intensity ( §4.2). ## human annotation we randomly divide the 50 instances (200 argument pairs) from each dataset into 10 batches, each with 5 instances (20 argument pairs). every batch is annotated by 5 individuals. one annotator evaluates at least one batch, allowing us to calculate inter-annotator agreements and base observations on individual annotators. although our primary focus is on how convincingness rankings change, we also include comparisons of emotional intensity to evaluate whether gpt4o adjusts the emotional appeal of arguments as intended. annotators compare emotions and convincingness of one argument pair by answering two subjective questions: (i) convincingness: which argumentative text do you find more convincing? (ii) emotion: which argumentative text evokes stronger emotions in you? equivocal judgments are allowed, i.e., annotators can judge both arguments as equally convincing or evoking an equal level of emotion. during annotation, argument pairs are shown with their topics. see appendix d for screenshots of the annotation interface. annotators we hire annotators from two sources: university students and the crowdsourcing platform prolific: 9 • student: 4 students are hired for this task. all annotators possess fluent to native-level proficiency in the languages of the evaluated arguments and are all based in germany. one of them is a phd student, and the others are master's students. three of them are involved in the pre-annotation phase to select out the needed argument pairs. we summarize the number of student and crowdsourcing annotators for each dataset in ## conclusion in this work, we examined how emotional intensity influences perceived convincingness. using gpt4o to rephrase arguments with varying emotional impact, we developed a dynamic framework inspired by manipulation checks in psychology and social sciences. our results show that gpt4o reliably generates counterpart arguments, preserving meaning while altering emotional tone. for both humans and llms, convincingness is largely unaffected by emotions. however, when emotions do play a role, they more often enhance rather than weaken convincingness, particularly in political debates, where emotional appeal is frequently used as a persuasive strategy. additionally, while llms broadly mirror human patterns, they struggle to capture emotional nuances. future research could explore when and how emotions influence convincingness across argument types. investigating specific emotions @xcite or justified vs. unjustified emotions and their persuasive effects may provide deeper insights. enhancing llms' ability to capture emotional nuances through improved prompts or fine-tuning could further strengthen their reliability in evaluating emotional arguments.