A Closer Look at Multidimensional Online Political Incivility (2024)

Abstract: Toxic online political discourse has become prevalent, where scholars debate about its impact to Democratic processes. This work presents a large-scale study of political incivility on Twitter. In line with theories of political communication, we differentiate between harsh ‘impolite’ style and intolerant substance. We present a dataset of 13K political tweets in the U.S. context, which we collected and labeled by those categories using crowd sourcing. Our dataset and results shed light on hostile political discourse focused on partisan conflicts in the U.S. The evaluation of state-of-the-art classifiers illustrates the challenges involved in political incivility detection, which often requires high-level semantic and social understanding. Nevertheless, performing incivility detection at scale, we are able to characterise its distribution across individual users and geopolitical regions, where our findings align and extend existing theories of political communication. In particular, we find that roughly 80% of the uncivil tweets are authored by 20% of the users, where users who are politically engaged are more inclined to use uncivil language. We further find that political incivility exhibits network homophily, and that incivility is more prominent in highly competitive geopolitical regions. Our results apply to both uncivil style and substance.

## introduction an active discourse between political groups and citizens is considered a fundamental condition for a healthy democracy @xcite . the recent rise of social media has been argued however to intensify disrespectful and hostile online political discourse @xcite . the term political incivility is commonly used in the political communication literature that studies the prominence, determinants and consequences of using harsh style and discriminatory discourse in online platforms. according to researchers, negative consequences of this trend are that it fosters polarization between rival political groups, and may disengage some citizens from being politically involved @xcite @xcite . conversely, others argue that incivility is not inherently negative, considering certain flavors of incivility as a legitimate rhetoric in heated political discussions @xcite . aiming to study the consequences and contextual factors that underlie this general phenomenon, several previous research works have attempted to empirically detect, quantify and characterise political incivility on discussion groups and social media platforms @xcite @xcite @xcite . in this research, we take a closer look at the challenges involved in the automatic detection of political incivility online, considering it as a multidimensional concept. we then present the results of a large-scale study, where we examine incivility as detected within a very large sample of political tweets posted on the social media platform of twitter. 1 in particular, we gauage and quantify user-level and geopolitical factors that correlate with political incivility online. we operationalize political incivility as a twodimensional concept in accordance with recent theories of political communication @xcite . the first dimension is personallevel incivility (impoliteness). this flavor of incivility pertains to a harsh tone that violates interpersonal norms, including foul language, namecalling, vulgarity, and aspersion towards other discussion partners or their ideas (e.g., "are you really so stupid that you would defund this program?"). the second dimension of public-level incivility (intolerance) rather pertains to exclusionary speech, impolite: "all hell has broken loose under the leadership of the senile old man. i don't believe a damn word from this dumb son of a bitches."; "that's what they are protesting, you rank imbecile. people like you need a damn good kicking." intolerant: "hillary and the dems are enemies, foreign and domestic"; "if you agree with democrats in congress, you are an anti-american commie" neutral: "how long do republicans believe you can keep pushing this line? you never intended to secure the border"; "there are 400,000,000 guns in the united states, you're going to have to stop the criminals not the guns" there are several motivations for identifying political incivility at this multidimensional resolution. in general, scholars of political communication have shown that the exposure to either impolite style or intolerant content online leads to increased polarization and intergroup tensions @xcite . yet, recent studies argue that heated political talk should not be dismissed due to interpersonal incivility, whereas expressions of intolerance on digital platforms have a more detrimental effect on democratic processes @xcite . it is therefore desired to distinguish between the different dimensions of political incivility in studying this phenomenon. in this work, we further show that interpersonal incivility and intolerance differ in their language characteristics. while impolite speech often contains unequivocally negative lexical expressions, the interpretation of intolerance is generally a more challenging task, in that it requires contextual, political and social, understanding. a main contribution of our work is the construction of a large dataset of 13k political tweets. we carefully retrieved and sampled these tweets using diverse strategies, aiming to capture both incivility types, while avoiding lexical and topical biases @xcite . the dataset was labeled by multidimensional incivility via crowd sourcing, having the annotation process supervised by a domain expert. using our dataset, we adapt and evaluate a variety of state-of-the-art language models on the task of multi-label incivility detection. our results indicate that political incivility detection is a challenging task, where we obtain best f1 scores of 0.70 and 0.59 on impoliteness and intolerance detection, respectively. in the second part of this work, we report the results of a large scale study, in which we performed multidimensional incivility detection and examined the prevalence of incivility among the political posts by more than 200k users. we find that both types of political incivility are prevalent on social media, identifying 17.6% of the political tweets as impolite, 13.3% as intolerant and 2.5% as both, with an overall political incivility rate of 28.4%. a user-level analysis shows that a minority of the users, who are politically engaged (as measured by the proportion of their tweets that concern political topics), are more inclined to use uncivil language, generating the majority of the uncivil tweets. our analysis further establishes that social patterns of political incivility involve network homophilly. considering the large scope of our study, we were also able to assess differences in the prevalence of incivility across geopolitical regions, specifically, states. we find that state-level incivility on social media is significantly correlated with partisan competition per state, observing higher incivility levels in 'battleground states', where the two camps are on par. we interpret our findings in light of existing theories of political communication, and discuss the challenges and potential of political incivility detection for future research. ## related work as noted in a recent survey, the concepts of uncivil, offensive, and toxic speech often overlap, where incivility is most frequently used by social scientists @xcite . in the political communication literature, some researchers frame incivility in terms of impolite speech @xcite , whereas others define it as either impoliteness, intolerance or hate speech @xcite . accordingly, most relevant empirical studies address incivility detection as a binary classification problem, differentiating between neutral and uncivil discourse @xcite @xcite . following recent theories of political communication @xcite , we consider political incivility as a multidimensional concept, defining uncivil language as either impolite or intolerant, or both. in a closely related work, @xcite introduced a dataset of tweets annotated with fine grained labels, distinguishing between our high-level categories of rude or offensive tone (profanities, insults, outrage, or character assassination) and intolerant expressions (discrimination, hostility). overall, they report f1 performance of roughly 0.7 on all categories. while offering valuable insights into multidimensional incivility detection, their dataset is focused on the topic of immigration, which receives limited attention in online political discourse @xcite . crucially, we refrained from sampling tweets based on topical keywords, while targeting political tweets by u.s. residents. consequently, our dataset captures incivility mainly in the u.s. partisan context, which is prevalent in twitter, across various topics (only 1.8% of the sampled tweets mention immigration). aiming at lexical as well as topical diversity, we also minimized the use of pre-trained tools as means for sampling texts that were likely to be toxic. possibly for these reasons, we observe substantially lower performance on intolerance detection in comparison to bianchi et al. (f1 of ∼0.6 vs. ∼0.7). 2 our analysis indicates that in the lack of clear lexical cues, contextual social understanding is required in order to improve on the task of intolerance detection. in this respect, our work relates to a recent line of works than concern the detection of implicit hate speech, where the underlying toxic intention is encoded using indirect semantics rather than by foul language @xcite . finally, this work makes the contribution of applying multidimensional political incivility detection at large-scale, studying its prevalence while considering various contextual factors, including user-level characteristics and geopolitical conditions. ## multidimensional incivility detection next, we evaluate the extent to which neural models can detect political incivility as perceived by humans. we perform multi-label classification, detecting impoliteness and intolerance as orthogonal dimensions, as well as experiment with binary prediction of political incivility. ## from tweets to users: a large-scale evaluation automatic incivility detection may be used to identify and quantify political incivility at scale, addressing research questions of interest. here, we introduce and examine the following questions: (i) are certain users more inclined to post uncivil political content online? (ii) do incivility levels vary by geopolitical region, specifically, across u.s. states? in both cases, we explore contextual factors that correlate with increased political incivility levels with respect to either impoliteness or intolerance. to investigate these questions, we collected a corpus comprised of the twitting history of a large number of user accounts. concretely, we randomly sampled users who authored tweets between july-nov. 2022, whom we verified to be residents of the u.s. based on the location attribute of their profiles. for each user account, we retrieved the most recent (up to 200) tweets posted by them, discarding retweets and non-textual tweets, as well as tweets posted by overly active accounts suspected as bots. 15 this resulted in a corpus of 16.3m tweets authored by 373k users. our of those, 2.6m tweets by 230k users were classified as political, henceforth, the corpus. finally, 17.6% of the political tweets were identified as impolite, 13.3% as intolerant, and 2.5% as both categories, accounting for overall incivility ratio of 28.4%. these proportions ## conclusion we presented mupid, a dataset of political incivility annotated via crowd sourcing, distinguishing between dimensions related to style (impoliteness) and substance (intolerance). as discussed in detail, we refrained from term matching and from using available toxicity detection tools so as to diminish topical and lexical bias. our experiments using finetuned language models and fewshot learners reached best f1 performances of 0.70 and 0.59 in identifying impolite and intolerance language, respectively. our results and analyses suggest that finer semantic and social understanding is required for more accurately decoding incivility as perceived in political contexts, where this particularly holds for intolerant expressions. a large-scale study demonstrates the utility of our models for studying various aspects of political incivility. we find that users who are politically engaged, in that they post political content more often, are more inclined to use uncivil language, where as few as 20% of the users authored 80% of the uncivil tweets. we also track network homophily, showing that 'uncivil users' tend to follow other accounts with increased incivility. analysing incivility at the aggregate level, we find that increased incivility is more prominent in battleground states. our dataset and models of multidimensional political incivility detection may support future research about the relationship between incivility and other contextual factors, e.g., user sociodemographics, as user traits such as age, gender, and education level, may be elicited given popular accounts that are followed by them @xcite . a temporal analysis may highlight the impact of political events on incivility levels. we believe that political incivility detection would benefit from the modeling of relevant social context, such as conversation history @xcite and the political events that the text refers to @xcite . incorporating information about the user alongside the text authored by them may also help decode the text meaning @xcite . initial experiments, in which we conjoined user network embeddings with the text encoding showed improved prediction performance. we hope that researchers will benefit from our dataset in exploring similar directions.