Persuasiveness of News Editorials depending on Ideology and Personality (2020)

Abstract: News editorials aim to shape the opinions of their readership and the general public on timely controversial issues. The impact of an editorial on the reader’s opinion does not only depend on its content and style, but also on the reader’s profile. Previous work has studied the effect of editorial style depending on general political ideologies (liberals vs.conservatives). In our work, we dig deeper into the persuasiveness of both content and style, exploring the role of the intensity of an ideology (lean vs.extreme) and the reader’s personality traits (agreeableness, conscientiousness, extraversion, neuroticism, and openness). Concretely, we train content- and style-based models on New York Times editorials for different ideology- and personality-specific groups. Our results suggest that particularly readers with extreme ideology and non “role model” personalities are impacted by style. We further analyze the importance of various text features with respect to the editorials’ impact, the readers’ profile, and the editorials’ geographical scope.

## introduction news editorials are considered the backbone of a community in which they tackle timely controversial issues, aiming to sway readers towards certain opinions. nowadays, editorials do not only focus on issues affecting their close entourage (e.g., within a state or a country), but rather tackle issues relevant across continents to shape the views of those living there and worldwide. for example, the new york times and der spiegel 1 lately invested resources to write news editorials about the august 4th beirut blast. as such, news editorials represent an important source for research on computational social science. to be persuasive, a news editorial should comply with the communication-persuasion paradigm defined by @xcite consisting of five factors: source, message, target, impact, and channel: (1) the source represents an editorial's author who tries to persuade the readers. usually, authors reflect the ideology of their newspaper. for example, the new york times is considered a liberal news portal and its editorials reflect this ideology. (2) the message represents an editorial's content and the linguistic choices it made, e.g., in terms of style. (3) the target represents the readers, their prior beliefs (e.g., their political ideology or its intensity) and their characteristics (e.g., personality traits or gender). (4) the impact represents the actual effect of an editorial on a reader. @xcite states that persuasive text aims at changing or affecting the behavior of others or at strengthening the existing beliefs of those who already agree. and (5) the channel represents the mean used to read the editorial, e.g., an online news portal. we leave an analysis of the impact of the medium used on the editorial's effectiveness to future research. previous research tackled how people are affected by arguments depending on their personality traits, interests, and beliefs. however, most studies conducted their analysis on dialogical text from debate portals and similar @xcite @xcite . for news editorials, we recently revealed that liberal readers, unlike conservatives, are affected by the linguistic style @xcite . still, it remains unexplored to what extent also the intensity of a political ideology plays a role, let alone a reader's personality traits. in our work here, we fill this gap, and we consider both the content and the style of an editorial. in particular, this paper analyzes the persuasive effect (the impact) of linguistic content and style choices of news editorials (the message) on readers (the target) with two profile varieties, the intensity of their political ideology and their personality traits. we distinguish lean and extreme intensity of ideology, and we resort to the "big five" personality traits @xcite : agreeableness, conscientiousness, extraversion, neuroticism, and openness. for our analysis, we employ a corpus with 1000 new york times news editorials @xcite . each editorial is annotated for a notion of impact that defines the editorial to either challenge its readers' stance, by making them rethink their current opinion towards a topic, to reinforce their stance, by helping them argue better about a certain issue, or neither. the annotations were added by 24 readers with different political ideologies (liberals and conservatives). for each reader, also the ideology intensity and the big five personality traits are reported, but this information has not been used so far to our knowledge. for each intensity and personality group in the corpus, we train one model to predict the persuasive effectiveness of an editorial, using various content and style features. our results show that people with extreme ideology are somewhat impacted by style, and the same holds for readers whose personality is relatively high in neuroticism and low in extraversion. we further investigate the role of editorials' geographical scope; whether it tackles a global, national, or state topics. ## related work news editorials reflect argumentation related to political issues and, therefore, comprise hidden rhetorical means @xcite , which makes them a challenging genre to study. some works dealt with news editorials for information retrieval purposes @xcite or for analyzing arguments @xcite @xcite . al @xcite represent editorial argumentation explicitly by annotating 300 news editorials with argumentative discourse units on the sub-sentence level. we employ their model to extract features from news editorials for predicting the persuasive effectiveness of news editorials, as detailed in section 4. @xcite argued that a persuasive effect is best achieved by providing showing a good character (ethos), evoking the right emotions (pathos), and providing logically reasoned arguments (logos) in a wellarranged and well-phrased way. this view was modeled by @xcite for argumentation synthesis. instead, we here follow the communication-persuasion paradigm of @xcite , stating that an argumentative text, and hence a news editorial, should comply with five factors to be persuasive, as already indicated in section 1. each of them is tackled in some way in related work: in previous work, we annotated news editorials at the document level, covering an editorial's persuasive effectiveness by reflecting to what extent a writer persuades a reader @xcite ; the effectiveness concept is based on the argumentation quality taxonomy defined by @xcite . in our annotation setup, we considered the beliefs of readers by profiling annotators based on political ideology (liberals or conservatives). we also provides additional information about the annotators' ideology ## data the analysis is conducted using the webis-editorial-quality-18 corpus @xcite . 1000 new york times editorials were annotated regarding their persuasive effects by three liberals and three conservatives each. the persuasive effect of each editorial was determined based on whether the editorial challenged their stance by making them rethink it, reinforced their stance by helping them argue better, or was ineffective. we previously utilized a corpus to investigate the role of editorial's style on readers with different political ideologies @xcite . in order to ease the comparison to el @xcite , we use the same dataset (with similar training/test split) in all our experiments. in particular, we chronologically split the dataset into 80% for training and 20% for testing ( ## features in this section, we describe the set of features that we select to explore the linguistic choices in editorials. these features encode semantic and pragmatic properties that may manifest the author's means of persuasion, implicitly or explicitly, and follow those in previous work @xcite : psychological meaningfulness, eight basic emotions, editorial evidence types, argumentativeness, and subjectivity. as those features essentially target the modeling of text style, we also consider standard text features to model text content. in the following, we describe all features in detail (an overview is given in ## analysis of the persuasive effect in this section, we assess the impact of the style and content of news editorials on their persuasive effectiveness for readers with different ideology intensities (extreme or lean), personality traits, and personality groups (role models or others). similar to el baff et al. (2020), we perform the analysis by approaching the following task: given a news editorial and a reader's profile characteristic (ideology intensity, personality trait, or personality group), predict the effect of the editorial. this task is tackled by developing a separate effect prediction model for each ideology intensity (extreme or lean), each personality trait (e.g., low agreeableness), and each personality group (role models or others). the prediction models use svm classifiers (with a linear kernel), in which each classifier is trained using its corresponding profile training set and evaluated on the test set (see section 3). the classifiers employ the features described in section 4, considering both the style and the content of the editorials. the svm cost was tuned using grid search with 5-fold cross-validation on the training set. we set the class weight to "balance" because of the skewed distribution of the data, as shown in tables 1 and 2 . the prediction results are reported using the macro-f 1 scores for each style feature alone, for the best combination of style features (top style), for the best combination of style and content (top content+style), and the random baseline. we measure significance using a t-test (wilcoxon's test if normal distribution is missing) to quantify the differences between each two feature-based models among random baseline', content, top style, and top style+content. in the following presentation of the results, we see readers as impacted by style and/or content if at least one model based on the respective feature manages to outperform the random baseline significantly. ## analysis of the impact of geographical scopes given the importance of the topic and its role in persuasive text @xcite , we conduct an analysis study considering both the readers' profiles and the topic of the editorials. in particular, we cluster the topics of the editorials and group them into three geographical scopes: (1) global discusses global issues, such as the iraq war. (2) national discusses national issues such as election, and (3) state discusses state (e.g. new york) related issues such as new york governor. we conduct our analysis on the training sets as in section 5, following two settings: (i) using the whole training sets, and (ii) using the editorials that belong to each geographical scope in the training sets overall, our approach is divided into two steps: (1) cluster the editorials into their three geographical scopes: global, national and state. and (2) extract feature importance for each setting (i, ii), and profile (ideology intensity, extreme and personality group, other). ## conclusion in this paper, we analyzed how linguistic choices, in news editorials, affect readers with different ideology intensities, personality traits and groups, filling the gap for el @xcite analysis. argumentative text, especially editorials tend to be very challenging to study due to the strategic maneuver used by the authors who are considered (usually) expert writers. therefore, the performance of predicting effectiveness is limited. in our work, we used one news editorial portal (the new york times) with an obvious ideology (liberal). the picture will be more complete if this analysis is conducted on news editorials with different ideologies. however, the purpose of this paper was to shed light on which linguistic choices affect which profile and on the importance of topical information when studying persuasiveness. our findings can be employed in augmented writing tools, to help editorials writer improve their message, based on their target's profile, to have a higher impact.