“It’s Not Just Hate”: A Multi-Dimensional Perspective on Detecting Harmful Speech Online (2022)

Abstract: Well-annotated data is a prerequisite for good Natural Language Processing models. Too often, though, annotation decisions are governed by optimizing time or annotator agreement. We make a case for nuanced efforts in an interdisciplinary setting for annotating offensive online speech. Detecting offensive content is rapidly becoming one of the most important real-world NLP tasks. However, most datasets use a single binary label, e.g., for hate or incivility, even though each concept is multi-faceted. This modeling choice severely limits nuanced insights, but also performance.We show that a more fine-grained multi-label approach to predicting incivility and hateful or intolerant content addresses both conceptual and performance issues.We release a novel dataset of over 40,000 tweets about immigration from the US and UK, annotated with six labels for different aspects of incivility and intolerance.Our dataset not only allows for a more nuanced understanding of harmful speech online, models trained on it also outperform or match performance on benchmark datasets

## introduction though once considered a problem driven primarily by reduced inhibitions in anonymous online spaces @xcite , offensive content has grown exponentially-to the point that many users no longer feel restricted by traditional conversational norms of tolerance and politeness, even when posting in their own names @xcite . the pervasiveness of toxic discourse on social media in particular has helped sow the seeds of discord and hatred that harm the health and well-being of its targets and pose significant threats to the fundamental rights of individuals and social groups on the margins @xcite . concerned about such outcomes, over the last two decades scholars and practitioners from a variety of fields have scrutinized online incivility and hateful speech. those working in natural language processing, for example, have developed techniques to detect different types of offensive discourse, ranging from incivility to hate speech, while social scientists have focused extensively on the larger substantive effects of these phenomena. most computational approaches for detecting online toxicity are based on classifiers that predict the presence of a single main binary label @xcite @xcite , with some notable exceptions @xcite @xcite . 1 however, while singlelabel binary approaches to harmful speech detection are conceptually tidy and tend to yield good predictive performance, they have major limitations. most notably, such approaches are unable to distinguish discourse that threatens democratic norms, values, and rights from expressions that are merely rude or impolite. prior work detecting incivility, for instance, has combined relatively harmless expressions that break traditional norms of polite speech-for instance, profanities and swearingwith discourse that is potentially more harmful, such as personal insults, stereotyping, or hateful speech @xcite @xcite . binary approaches to toxic and offensive content detection oversimplify these complex concepts, and ultimately undermine researchers' and practitioners' ability to understand potential harms and evaluate what content should receive most focus and intervention, including for the purposes of content moderation. to address these open issues, we show that our multi-label approach rooted in insights drawn from social science is not only potentially more insightful, but also improves performance of detection models. in contrast to most previous work, we build upon a conceptual model that disentangles uncivil from intolerant online discourse @xcite . the resulting labels can meaningfully distinguish discourse that is simply rude or offensive (incivility) from expressions that threaten democratic norms and values, such as equality, diversity, and freedom (intolerance). we collect a dataset of more than 40,000 us-and uk-based tweets related to the topic of immigration, and annotate these tweets for four sub-types of incivility (profanities, insults, outrage, character assassination) and two sub-types of intolerance (discrimination, hostility). we refer to this dataset as not just hate (njh). we then fine-tune large pre-trained language models and show that these labels can be predicted with consistently good performance. we compare our results to other benchmark hate speech datasets to produce more insights about the dataset we introduce. models trained on our data match or outperform state-of-the-art performance on those datasets. our approach, annotation methodology, and dataset can help the future development of automated harmful online speech detection, and foster a more nuanced understanding of the distinctive types of discourse that constitute online toxicity and abuse. data and additional details on the annotation are available on osf. foot_1 details are also available on the github repository. 3 contributions we describe a novel perspective on harmful online speech detection. we describe in detal our annotation pipeline and we release a dataset, njh, of just over 40,000 tweet ids annotated with four sub-types of incivility (profanities, insults, outrage, character assassination) and two sub-types of intolerance (discrimination, hostility). we show that our data set generalizes to various types of offensive language and reaches state-of-2 data collection we collected our dataset via the twitter enterprise api, downloading over 150 million tweets over the course of 2020-2021. we selected the keywords used to collect these tweets in a multistage process. beginning with a list of 30 keywords and phrases commonly associated with immigration in the us and/or uk (e.g., immigration, immigrant, refugee, illegals), we drew a random sample of tweets containing those words from the public streaming api, produced a list of words commonly co-occurring with the seeded terms, and qualitatively analyzed that list to identify the appropriateness of each additional keyword or phrase. we performed the same process with a series of subreddits related to immigration, carefully curating the list of subreddits to represent both pro-and anti-immigration sentiment, as well as a variety of immigration subtopics (e.g., subreddits dedicated to asian or muslim immigrants/immigration, refugees, etc.). please see the appendix for the final set of keywords used to collect tweets. annotation our annotation approach was based on quantitative content analysis @xcite , a social scientific method used by communication scholars to interpret meaning in textual data at scale. the annotation guidelines were broadly inspired by those of @xcite , which we augmented and adapted with examples that were context-and country-specific to capture the nuances of immigration debates across the us and the uk. the annotators were ten undergraduate researchers from the university of liverpool (uk) and syracuse university (us). we trained the students on the annotation guidelines until they achieved a satisfactory inter-annotator reliability score for two consecutive weeks (krippendorff's α of 0.68 or above) and gwet's ac1 (of .6 or above). we used these to correct for expected issues in the data quality-krippendorff's α penalizes data scarcity, which is a problem in some of our labels, while gwet's ac1 corrects for the probability that the annotators agree by chance (this is more likely for difficult annotation tasks such as this one). we continuously monitored the quality of the annotation by measuring inter-annotator reliability on a monthly basis. how to aggregate annotation scores is a central and open problem in machine learning @xcite . we divided the trained annotators into teams to individually annotate tweets from their respective country. we compared their individual annotations, and they met to discuss and adjudicate any disagreements-i.e., all annotators had to agree on a best label(s). we opted for this rigorous annotation process instead of a simple majority rule due to the complexity of the phenomena we investigate. while prior efforts on incivility and hate speech detection have relied on approaches such as 'majority rule' to determine labels based on crowdsourced annotations, leading to biases in interpretation that may undermine the quality of annotation, we leverage social scientific content analysis techniques to develop a rigorous annotation pipeline to ensure the quality of our dataset. we randomly selected samples for annotation, stratifying by location. we restricted the data to the us and uk via geolocated tweets and user location, and use simple trigram tf-idf based classifier to distinguish between the uk, us, and other (macro @xmath0.82, albeit we have updated our algorithm several times during the course of the research project) trained on 3 million tweets. given the relative scarcity of intolerant discourse in particular, we used jigsaw's perspective api to over-sample potentially toxic tweets and increase the presence of some labels on approximately 60% of the annotated sample. ## conclusions we suggest that a more fine-grained approach improves offensive and potentially harmful speech detection online, and, crucially, can allow for a better understanding of the spectrum of online toxicity. our approach can successfully disentangle incivility from likely more harmful cases of intolerant content, allowing scholars and practitioners to better understand and detect discourse that undermines broader democratic norms, values, and rights @xcite . we introduce a dataset of just over 40,000 tweets, annotated with six labels. models trained on our dataset can predict labels with good confidence and perform well on other benchmark datasets. we acknowledge financial support for this research in the form of a gift from twitter, inc.