Analyzing Persuasion Strategies of Debaters on Social Media (2022)

Abstract: Existing studies on the analysis of persuasion in online discussions focus on investigating the effectiveness of comments in discussions and ignore the analysis of the effectiveness of debaters over multiple discussions. In this paper, we propose to quantify debaters effectiveness in the online discussion platform: “ChangeMyView” in order to explore diverse insights into their persuasion strategies. In particular, targeting debaters with different levels of effectiveness (e.g., good vs. bad), various behavioral characteristics (e..g, engagement) and text stylistic features (e.g., used frames) of debaters are carefully examined, leading to several outcomes that can be the backbone of writing assistants and persuasive text generation.

## introduction persuasion, a primary goal of argumentation, is the ability to convince people to do a certain action or form a particular belief (o' @xcite . persuasion has always influenced the dynamics of communication and social interactions, either positively by raising awareness on critical issues like climate change or negatively by influencing the behavior of voters in elections or disseminating propaganda and fake news. social media, through its growing role in the formation of beliefs, has gained notable interest as a means to gather a deeper understanding of persuasion @xcite . the changemyview (cmv) subreddit in particular has been used in various studies that model text persuasiveness using a variety of linguistic, argumentative, and behavioral features (e.g., @xcite , @xcite , and @xcite ). however, scholarly work on online persuasion centers around studying persuasive comments in individual discussions without considering the importance of analyzing persuasive debaters (luu * equal contribution @xcite . hence, debater strategies and their effectiveness have not been adequately studied. understanding effective debating strategies and debater persuasiveness can be highly beneficial for media analysis, rhetorical review, and for learning debating skills. besides, it can advance the development of several applications, where effective strategies can be recommended in writing assistants and dialog management systems or encoded in the backbone of text generation tools. this paper focuses on analyzing the debaters' persuasion strategies, seeking to uncover the behavior, language style, and argumentative techniques that distinguish good from poor debaters. to do so, we categorize cmv debaters based on their effectiveness in persuasion and examine key differences in their behaviors and skills (i.e., engagement and experience), as well as their argument's style (at the semantic, syntactical, lexical, and pragmatic levels). we propose the task of identifying effective debaters and present an approach to tackle it. our analysis of debater strategies yields several insights. for example, we find that the effectiveness of persuasion improves over time for average debaters, the distribution of 'frames' in the debaters' arguments can play a major role in persuasion, and argumentative features based on the presence of certain types of arguments in the debaters' text do not seem sufficient to indicate the effectiveness of persuasion. the contribution of this paper is threefold: ## background and related work in this section, we introduce cmv's core concepts as required for our study. afterward, we overview the primary studies on modeling persuasiveness on cmv and related platforms. ## data acquisition and preparation to conduct our study on debater persuasion strategies, we created a dataset with 3, 801 cmv debaters, equally sampled for good, average, and poor debater persuasiveness. here, we detail our data collection method, quantification of debater persuasiveness, and sampling method to balance the dataset by debater persuasiveness. quantifying debater persuasiveness we define the persuasiveness of a debater d with comments c 1 , . . . , c i,∆1 , . . . , c j,∆k , . . . , c n in cmv as ratio of delta comments c ∆ to all comments: the persuasiveness is, hence, the number of debater's delta comments normalized by her total comment count. as based on the persuasiveness score, we categorize debaters into three groups as follows: the separation of debaters with a non-zero persuasiveness is based on the observation that obtaining any c ∆ is already challenging. hence, the highly persuasive tail should be studied as a separate population. the 5%-threshold used in categorization separates the non-poor debaters into two groups of approximately equal size. ## debater engagement and experience our first analysis concerns the relationship between the debaters' persuasiveness and their engagement with and experience on the cmv subreddit. we presume that engagement on cmv may correspond to rebuttals in live debates. our findings suggest that a high engagement is indicative of persuasive debaters. we further inspect the relationship between experience and persuasiveness in both absolute measures such as comment count and active period and relative measures such as changes in style and persuasiveness with experience gain. our findings suggest that debaters become more persuasive with increased experience, especially average debaters. however, the experience effect is not reflected in absolute experience measures, and hence it is hard to operationalize for classification. ## debater style analysis stylistic features are frequently used to determine the characteristics of authors. since stylistic features are indicative of persuasive comments, we consider stylistic features to also be indicative of persuasive debaters. in particular, we study the re-lationship between a debater's persuasiveness and the lexical, syntactic, semantic, and pragmatic dimensions. we found notable differences in persuasiveness in each dimension. the most substantial feature is again comment length. additionally, we found that better debaters tend to have lower lexical diversity and syntactic complexity, but a higher semantic diversity. we also found correlations between certain word class patterns and certain patterns of elementary argumentative units, particularly rhetorical statements. lastly, found that persuasive debaters use political and cultural identity frames more often. ## debater persuasiveness prediction in addition to the analytical scrutiny of debater persuasiveness, we conduct an experimental validation of our findings by classifying debaters by persuasiveness. we define the general task of debaterlevel persuasiveness prediction as: given a debater d with comments c 1 , . . . , c n , classify this debater as persuasive (good) or non-persuasive (average or poor). to conclusively supplement our analysis, we individually inspect the classification performance of the introduced features (cf. section 5). we encoded the syntactic, semantic, and pragmatic features of our analysis for each of the 3,801 debaters in our cmv debaters' corpus. each encoding was chosen to obfuscate comment length as far as reasonable. we encoded the word class and all argumentative unit n-grams tf-idf vectors of the aggregated comments. we encoded the numerical features, outdegree centrality, closeness centrality, and the number of dependents for text complexity and comment-op distance and within-comment distance for wmd, by averaging comment-level counts per debater. we encoded each of the 15 frames with the absolute and relative number of comments that utilize a frame. as baselines, we selected feature sets previously used for comment persuasiveness prediction: bagof-words, vocabulary interplay after @xcite , which covers op and commenters' vocabularies' absolute and relative overlap and jaccard similarity, and common stylometrics, which cover counts of words, selected word classes, links, word lists, symbols, type-token ratio, and readability scores. the baseline feature sets were implemented trivially following the related work. we consider two binary classification settings for our experimental validation: (1) good vs. average and (2) good vs. poor. we maintained a balanced distribution of the classes (1,267 each). the the classification results reveal several findings: first, most features distinguish good from poor debaters better than good from average ones. syntactic features are the only exception to this trend, which can not be explained by our analysis. second, bag-of-words is a strong feature for the two settings as it outperforms most of the other features. besides, the weak effectiveness of the argumentative features is similar to the observations of @xcite ; the mere distribution of argumentative units in the text is insufficient to identify its persuasiveness. third, the distribution of the frames in the debaters' comments results in the best scores across the two experimental settings. the most discriminating frames are 'quality of life', 'morality', and 'health and safety', all with negative weights towards the 'good debater' class. ## conclusion and discussion the persuasion skills of debaters can vary depending on different cultural and social aspects, among others. understanding how people argue and what makes some debaters more successful than others are interesting research questions that have been neglected in the literature. this paper has contributed in this regard by modeling debater effectiveness in cmv and analyzing their behavior and argumentative stylistic choices, demonstrating several inter-esting insights that can be utilized for improving the persuasion skills of new debaters and assessing the development of advanced text generation and writing assistant tools. still, we think there is room for further analysis. first, we quantified the persuasiveness of cmv debaters based on awarded ∆s only; although it appears to be a standard method in previous work, we believe that a more comprehensive quantification, possibly using human judgments and a more finegrained scale, would account for a degree of subjectivity to consider the evaluating user's idiosyncrasies. @xcite touches on this briefly, finding that despite general agreement about what is persuasive, there are differences in the assessment of persuasion based on the positions of the evaluating party. second, while argumentative features based on the distribution of argumentative units did not perform well in our prediction task, possible improvements can be achieved through modeling features that can capture the effective use of argumentative units. a possible direction is to identify the interdependencies between the different argumentative units in the text @xcite as well as their relative arrangement @xcite . third, other, more in-depth features can disclose useful insights into debater persuasiveness. conceivable are features that better model behavior like experience and the dynamic of debater interaction or the velocity of experience gain. fourth, using more sophisticated models in the prediction task may lead to better results, although our logistic regression is ideal to compare class separability by feature. @xcite proposed using conditional random fields (crf) to model the cumulative effect of persuasion in cmv discussions, and li et al. (2020) used bi-lstm and bert to model their persuasiveness task.