A nalyzing the P ersuasive E ffect of S tyle in N ews E ditorial A rgumentation (2020)

Abstract: News editorials argue about political issues in order to challenge or reinforce the stance of readers with different ideologies. Previous research has investigated such persuasive effects for argumentative content. In contrast, this paper studies how important the style of news editorials is to achieve persuasion. To this end, we first compare content- and style-oriented classifiers on editorials from the liberal NYTimes with ideology-specific effect annotations. We find that conservative readers are resistant to NYTimes style, but on liberals, style even has more impact than content. Focusing on liberals, we then cluster the leads, bodies, and endings of editorials, in order to learn about writing style patterns of effective argumentation.

## introduction the interaction between the author and the intended reader of an argumentative text is encoded in the linguistic choices of the author and their persuasive effect on the reader @xcite . news editorials, in particular, aim to challenge or to reinforce the stance of readers towards controversial political issues, depending on the readers' ideology @xcite . to affect readers, they often start with an enticing lead paragraph and end their argument with a "punch" @xcite . existing research has studied the persuasive effect of argumentative content and structure @xcite or combinations of content and style @xcite . in addition, some works indicate that different types of content affect readers with different personalities @xcite and beliefs @xcite . however, it remains unexplored so far what stylistic choices in argumentation actually affect which readers. we expect such choices to be key to generating effective argumentation @xcite . this paper analyzes the persuasive effect of style in news editorial argumentation on readers with different political ideologies (conservative vs. liberal). we model style with widely-used features capturing argumentativeness @xcite , psychological meaning @xcite , and similar (section 3). based on the ny-times editorial corpus of el @xcite with ideology-specific effect annotations (section 4), we compare style-oriented with content-oriented classifiers for persuasive effect (section 5). 1 while the general performance of effect prediction seems somewhat limited on the corpus, our experiments yield important results: conservative readers seem largely unaffected by the style of the (liberal) nytimes, matching the intuition that content is what dominates opposing ideologies. on the other hand, the style features predict the persuasive effect on liberal readers even better than the content features -while being complementary. that is, style matters as soon as ideology matches. knowing about the specific structure of news editorials, we finally obtain common stylistic choices in their leads, bodies, and endings through clustering. from these, we derive writing style patterns that challenge or reinforce the stance of (liberal) readers of (liberal) news editorials, giving insights into what makes argumentation effective. ## related work compared to other argumentative genres @xcite , news editorials use many rhetorical means to achieve a persuasive effect on readers @xcite . computational research has dealt with news editorials for retrieving opinions @xcite , mining arguments @xcite , and count of subjective and objective sentences @xcite in related work, @xcite found that emotional and rational arguments affect people with different personalities, and durmus and cardie (2018) take into account the religious and political ideology of debate portal participants. in followup work, @xcite observed that style is more important for decided listeners. unlike them, we focus on the stylistic choices made in well-planned argumentative texts. the lead paragraphs and the ending of an editorial have special importance @xcite . @xcite analyzes how leads and endings changed over time, whereas @xcite examined the readability of an editorial based on them. to our knowledge, however, no one investigated their importance computationally so far. in this paper, we close this gap by analyzing what style of leads and endings is particularly effective compared to the editorial's body. ## style features to model style, we need to abstract from the content of a news editorial. this section outlines the feature types that we employ for this purpose. most of them have been widely used in the literature. liwc psychological word usage is reflected in the linguistic inquiry and word count @xcite . liwc is a lexicon-based text analysis that assigns words to psychologically meaningful categories @xcite . we use the liwc version of @xcite , which contains 15 dimensions listed in the following with examples. webis adus to identify argumentative units in editorials that present evidence, we use the pre-trained evidence classifier of @xcite . for each editorial, we identify the number of sentences that manifest anecdotal, statistical, and testimonial evidence respectively. @xcite constructed a lexicon that includes various patterns of arguing such as assessments, doubt, authority, emphasis. for each lexicon, we have one feature that represents the count of the respective pattern in an editorial. ## data as the basis of our analysis, we use the webis-editorial-quality-18 corpus @xcite . the corpus includes persuasive effect annotations of 1000 english news editorials from the liberal new york times (nytimes). 2 the annotations capture whether a given editorial challenges the prior stance of readers (i.e., making them rethink it, but not necessarily change it), reinforces their stance (i.e., helping them argue better about the discussed topic), or is ineffective for them. each editorial has been annotated by six annotators: three with liberal and three with conservative ideology. to evaluate an editorial's persuasive effect on liberals, we computed the majority vote of their annotations for the editorial (and, similarly, for conservatives). we ended up with 979 editorials with effect labels for liberals and conservatives, because we found 21 duplicate editorials with the same content but different ids (for these, we use the majority vote across all duplicates). the corpus does not have predefined evaluation datasets. to mimic real-life scenarios, we chronologically split it into a training set (oldest 80%) and a test set (newest 20%). ## prediction of persuasive effects to assess the impact of news editorial style on readers, we employ our style-based features on the task of predicting an editorial's persuasive effect: given either of the two ideologies (liberal or conservative), predict for each editorial whether it is challenging, reinforcing, or ineffective. we developed separate prediction models for the effect on liberals and conservatives, respectively. for each style feature type and for their combinations, we trained one svm model with a linear kernel on the training set using scikit-learn @xcite . given the dataset split mentioned above (training set 80%, test set 20%), we tuned the svm's cost hyperparameter using grid search with 5-fold cross-validation on the training set. since the distribution of effect labels is highly skewed, we set the hyperparameter class_weight to "balanced". we then trained the best model on the whole training set and evaluated it on the test set. for comparison, we also built models for standard content features (lemma 1-to 3-grams), and we consider the random baseline that picks an effect class by chance. for both ideologies, in general, the results indicate that the persuasive effect seems hard to predict on the given corpus. still, we observe that the style features play a notable role in predicting the effect of editorials on liberals. they achieve a significantly better macro f 1 -score of 0.43 when combined with content compared to 0.36 when using content alone, at @xmath1.05. on the other hand, the f 1 -scores of content (macro 0.37, micro 0.38) and style (both 0.36) in predicting the effect on conservatives, are insignificantly different even from the baseline (0.33, 0.34). these results suggest that style is important as soon as the ideology of a reader matches the one of the news portal (at least, this holds for liberal ideology), but not if it mismatches (here, conservative). ## identification of style patterns observing that the style of nytimes editorials affects liberal readers, we seek to learn what patterns of writing style makes their argumentation effective. to this end, we (1) abstract each discourse part of an editorial (lead, body, ending) into a style label using cluster analysis and (2) identify sequential patterns of style labels that are specific to challenging, ineffective, and reinforcing editorials. clustering styles of discourse parts given the importance of specific discourse parts of editorials @xcite , we split each editorial into lead, body, and ending. for each part, we separately perform three steps on the training set of the given corpus: foot_4 ## conclusion this paper analyzes the importance of news editorials style in achieving persuasive effects on readers with different political ideologies. we find evidence that style has a significant influence on how a (liberal) editorial affects a (liberal) reader. inspired by the theory of the high importance of the lead and ending in writing editorials @xcite , we also reveal common effective and ineffective style sequences (lead-body-ending) statistically. our findings help to understand how effective argumentation works in the political sphere of editorial argumentation -and how to generate such argumentation. in related work, el @xcite revealed the impact of style features on generating pathos-and logos-oriented short argumentative texts based on the rhetorical strategies discussed by @xcite . with the findings of this paper, we go beyond, defining the basis of a styledependent generation model for more sophisticated argumentation, as found in news editorials.