Developing a New Classifier for Automated Identification of Incivility in Social Media (2020)

Abstract: Incivility is not only prevalent on online social media platforms, but also has concrete effects on individual users, online groups, and the platforms themselves. Given the prevalence and effects of online incivility, and the challenges involved in human-based incivility detection, it is urgent to develop validated and versatile automatic approaches to identifying uncivil posts and comments. This project advances both a neural, BERT-based classifier as well as a logistic regression classifier to identify uncivil comments. The classifier is trained on a dataset of Reddit posts, which are annotated for incivility, and further expanded using a combination of labeled data from Reddit and Twitter. Our best performing model achieves an F1 of 0.802 on our Reddit test set. The final model is not only applicable across social media platforms and their distinct data structures, but also computationally versatile, and - as such - ready to be used on vast volumes of online data. All trained models and annotated data are made available to the research community.

## introduction given the growing polarization in the united states, the increasing popularity of partisan media, and the widespread use of social media for information and discussion (see @xcite for a review), many scholars and observers worry about the accelerated use and spread of incivility in the online environment. incivility, defined as "features of discussion that convey disrespectful tone toward the discussion forum, its participants, or its topics" @xcite is a common aspect of many online communities, especially anonymous forums @xcite such as reddit. estimates suggest that more than 84% of americans have experienced incivility online, and among those who have ever experienced it, the number of average weekly en-counters with incivility was as high as 10.6 times @xcite . in addition to lowering the standards of public discourse, incivility has concrete effects on users, online discussions, and social media platforms. the use of and exposure to incivility generates negative emotions, such as anger, anxiety, or mental distress, and is related to aggression @xcite and hostile communication @xcite . incivility also turns users away from online discussions altogether @xcite @xcite . given these reasons for the public and industry to be concerned with online incivility, many companies seek to automatically detect incivility in order to understand its scope, identify the online communities in which incivility is particularly prevalent, and -ultimately -address the problem. this project offers a step in this direction. we present machine learning models for detecting incivility in social media, models that are not only computationally efficient but also applicable across platforms. we propose both a bert-based neural classifier as well as a logistic regression based classifier trained on manually annotated and artificially labeled data. our results suggest that the proposed models perform well across distinct data/communication structures of different platforms, and, as such, can be easily applied to detect incivility. ## previous work there is considerable conceptual and operational ambiguity in the literature on incivility and related concepts under the umbrella of offensive or intolerant speech (see @xcite for a review). some studies use incivility interchangeably with hate speech, which refers to speech that aims to discriminate against a certain identity group, or aggressive or toxic language, which includes personal attacks @xcite . however, incivility is a broader concept, which focuses on content that goes against acceptable social norms in terms of vulgarity, name-calling, or offensive language @xcite , whereas hate speech or aggressive language captures more specifically discourse that offends, derogates, or silences others and may promote harm @xcite . increasingly, incivility is conceptually and operationally distinguished from such intolerant discourse, and evidence suggests that the effects of these two forms of expressions also differ @xcite . definitions of incivility vary, ranging from "a normdefying behavior" @xcite , "an explicit attack" @xcite , to the violation of interpersonal politeness norms @xcite , yet most include a lack of respect toward discussion participants or arguments @xcite , and a impolite tone of discourse @xcite . the often used definition, which we adopt for the purpose of our machine learning model, sees incivility as features of discussion that convey disrespectful tone toward the discussion participants or its topics, including name-calling, mean-spirited or disparaging words directed at a person or group of people, an idea, plan, policy, or behavior, vulgarity, using profanity or language that would not be considered proper in professional discourse, and pejorative remarks about the way in which a person communicates @xcite . as such, our approach encompasses both the less societally detrimental foul language or harsh tone as well as the more intolerant discourse. from a technical perspective, previous research using machine learning models to detect incivility and other offensive or intolerant language online has focused primarily on the use of logistic regression @xcite @xcite , support vector machines @xcite , and various neural classification models @xcite . bert @xcite and related transfomer language models have been used in related tasks, such as identifying abusive language on twitter @xcite , including many entrants in the offenseval task at semeval-2020 @xcite . to our knowledge, this paper is the first to utilize a fine-tuned bert model to identify incivility on social media platforms, and one of few projects that train the classifier on data from more than one platform. also, past work on identifying incivility over time has mostly analyzed twitter data during certain political events, such as the 2016 presidential election in the us @xcite , and/or looked at political incivility in specific contexts (e.g., among politicians, e.g., @xcite . these rather narrow, single-platform foci limit the applicability of the developed classifications, a limitation we address in this project. in addition to these contributions of our work, our primary contribution may lie in our data augmentation method. specifically, we extend recent approaches to automatically label additional training data to improve the performance of a logistic regression classifier. previous work in detection of offensive language has used back-translation @xcite and data transformation techniques @xcite to augment limited training data. while some work @xcite utilizes the google perspectives api to label additional training data, which introduces noise to the operationalization of incivility, we take advantage of our well-performing bert classification model to generate artificial training data for a logistic regression classifier. the resulting classifier can be efficiently run on cpu and is far less computationally expensive than our comparably performing bert model. this extension makes our classifier easily applicable to vast amounts of data and readily implemented on social media platforms or the comments sections of websites of news media organizations. ## dataset we present a corpus of reddit data annotated for incivility at the post level, and further annotated for political/non-political content at the subreddit level. we chose to use reddit as the source of our dataset because reddit is the sixth most popular website in the us and the third most visited social media platform following @xcite . there are more than 430 million active users worldwide on reddit @xcite . also, the anonymous nature of reddit makes it popular for sharing information and engaging in long and complex discussions, making it ideal for observing online discourse. furthermore, the public nature of reddit allowed us to gather a large number of posts from across various user communities, known as subreddits. each subreddit has a general topic, behavioral norms, and community standards, allowing for a creation of a diverse dataset, which further increases the applicability of the resulting machine learning model. to tackle the detection problem, we identified the most popular subreddits from 2006 to 2019 that contained 95% of the total comments by (1) the number of comments in the subreddit each year, and (2) the number of followers that commented in the subreddit each year, which resulted in 9355 subreddits across the years. we then collected 5000 comments from these subreddits using stratified random sampling technique, such that the random sampling from each year is based on each year's proportion in the total number of comments. these 5000 posts were the manually labeled. ## classifier training to demonstrate the efficacy of our collected dataset, we use supervised machine learning to automatically identify uncivil reddit posts. however, annotating a dataset large enough to train a state-ofthe-art neural classifier from scratch is a costly and time-consuming undertaking. we experimented with several neural binary classifiers, with our bestperforming models built on top of transformerbased language models, namely bert @xcite and its relative, distilbert @xcite . past work has demonstrated that finetuning large, pre-trained language models, such as bert and distilbert, is an effective method for creating a high-quality neural classifier with limited supervised training data. as described in @xcite , we conduct additional pretraining of the bert-base and distilbert-base models on a large collection of reddit posts as in-domain data. once pretrained, we fine-tune our models for classification on our annotated dataset of reddit comments, which trained annotators classified with binary labels for incivility. finally, in an effort to extend past work by creating a more flexible, platform agnostic classifier, we train a logistic regression classifier for incivility prediction in social media by combining the data presented in @xcite with our annotated and artificially labeled datasets. our reddit dataset (including annotation disagreements), test predictions, scripts and models are available on the project github repository foot_0 . ## experiments and results our bert and distilbert models begin with the respective base pretrained language models, as implemented in huggingface's transformer's package. we then further pretrain these models on dataset of 3 million reddit posts, for 100,000 training steps (as suggested by @xcite ) using the masked word prediction task @xcite . we then utilize these pretrained models in a classification setup, utilizing a softmax layer to predict binary class probability based on the [cls] token in bert's final hidden layer. for classification fine-tuning, all inputs to the models are limited to 256 tokens in length, with a training batch size of 16. we use the adamw optimizer (gugger and howard, 2018) with default learning rate and epsilon values. we fine-tune our model for classification for four epochs on our dataset of 5,000 reddit posts which are coded for incivility, with 10% of the data set aside for training validation, and 1000 annotated posts set aside for model testing. classification results using bert and distilbert are shown in ## analysis and discussion our encouraging results in classifying incivility in reddit posts demonstrate the efficacy of our dataset due to the scale of the data to be ultimately classified, we were concerned as much with computational efficiency as with prediction accuracy when building our incivility classifier. when we use our trained bert model to generate a large quantity of synthetically labeled training data, the performance of our log regression model is comparable to that of the our fine-tuned bert models. concern with computational efficiency also informed our choice of features in our logistic regression model. while alternate features could be used, such as doc2vec or word2vec embeddings, we chose to use tfidf due to the simplicity of calculating these features. additionally, the choice of tfidf was informed by the work of @xcite , who demonstrate the utility of tfidf for the task for incivility classification. finally, the fact that our tfidf-based logistic regression model performs similarly well to the bert model is evidence of the effectiveness of the choice of tfidf features. that said, the use of alternate features may improve model performance, and we leave this to future work. the similarity between the predictions made by our bert model and our logistic regression model indicates that the logistic regression model retains much of the predictive power of the bert model. in fact, across 996 test comments, the two models disagreed on only 27 comments, for a rate of 2.7%. from reviewing the disagreements we can identify several classes of comment on which the two mod-els often disagree. the first, and most obvious, is very long comments. bert is designed to truncate long input text (our implementation truncates inputs longer than 256 tokenized word pieces). thus, our bert model may mislabel longer comments in which the incivility occurs later in the comment. another source of disagreement comes from the fact that our tfidf-based classifier tends to be more sensitive to individual lexical items, which is to be expected as bert is known to condense far more semantic information than do count-based vectorization techniques such as tfidf @xcite . for example, our regression model mislabels the comment "this is dope! does anyone know where i can purchase one for myself?" as an uncivil comment, presumably due to the presence of the word "dope", while our bert model labels the comment correctly. in future work, we plan to conduct a more rigorous analysis of labelling disagreements between the two models to better understand the role of lexicon and compositional semantics in the incivility classification task. finally, we demonstrate the flexibility of our model training strategy by creating a combined incivility prediction model using our automatically labeled reddit data with the synthetic data provided by @xcite . the resulting model has shown promise as a platform agnostic incivility classifier model for social media. ## conclusion and future work in this paper, we present a new dataset of reddit posts annotated at the comment level for incivility, as well as at the subreddit level for political content. further, we demonstrate the efficacy of this dataset to train machine learning models for incivility detection, both alone and in combination with previously available datasets, to create a platform agnostic classifier for incivility on social media. using our trained classifier, our future goal is to provide a systematic overview of trends in incivility on social media, across time and variety of discussion topics. the project aims to capture the fluctuations in the prevalence of incivility in political and non-political online spaces, politically homogeneous and heterogeneous discussions, liberal and conservative ones, and also among different non-political topics. the anticipated study will add our understanding of the development of online incivility and shed light on incivility interventions.