Ruddit: N orms of Offensiveness for E nglish R eddit Comments (2021)

Abstract: On social media platforms, hateful and offensive language negatively impact the mental well-being of users and the participation of people from diverse backgrounds. Automatic methods to detect offensive language have largely relied on datasets with categorical labels. However, comments can vary in their degree of offensiveness. We create the first dataset of English language Reddit comments that has fine-grained, real-valued scores between -1 (maximally supportive) and 1 (maximally offensive). The dataset was annotated using Best–Worst Scaling, a form of comparative annotation that has been shown to alleviate known biases of using rating scales. We show that the method produces highly reliable offensiveness scores. Finally, we evaluate the ability of widely-used neural models to predict offensiveness scores on this new dataset.

## introduction social media platforms serve as a medium for exchange of ideas on a range of topics, from the personal to the political. this exchange can, however, be disrupted by offensive or hateful language. such language is pervasive online @xcite , and exposure to it may have numerous negative consequences for the victim's mental health @xcite . automated offensive language detection has thus been gaining interest in the nlp community, as a promising direction to better understand the nature and spread of such content. there are several challenges in the automatic detection of offensive language @xcite . the nlp community has adopted various definitions for offensive language, classifying it into specific categories. for example, @xcite classified comments as racist, sexist, neither; @xcite as hate-speech, offensive but not hate-speech, neither offensive nor hate-speech @xcite as abusive, hateful, normal, spam. @xcite ; @xcite ; @xcite ; @xcite summarize the different definitions. however, these categories have significant overlaps with each other, creating ill-defined boundaries, thus introducing ambiguity and annotation inconsistency @xcite . a further challenge is that after encountering several highly offensive comments, an annotator might find subsequent moderately offensive comments to not be offensive (de-sensitization) @xcite . at the same time, existing approaches do not take into account that comments can be offensive to a different degree. knowing the degree of offensiveness of a comment has practical implications, when taking action against inappropriate behaviour online, as it allows for a more fine-grained analysis and prioritization in moderation. the representation of the offensive class in a dataset is often boosted using different strategies. the most common strategy used is key-word based sampling. this results in datasets that are rich in explicit offensive language (language that is unambiguous in its potential to be offensive, such as those using slurs or swear words @xcite ) but lack cases of implicit offensive language (language with its true offensive nature obscured due to lack of unambiguous swear words, usage of sarcasm or offensive analogies, and others @xcite ) @xcite . key-word based sampling often results in spurious correlations (e.g., sports-related expressions such as announcer and sport occur very frequently in offensive tweets). lastly, existing datasets consider of-fensive comments in isolation from the wider conversation of which they are a part. offensive language is, however, inherently a social phenomenon and its analysis has much to gain from taking the conversational context into account @xcite . in this paper, we present the first dataset of 6000 english language reddit comments that has finegrained, real-valued scores between -1 (maximally supportive) and 1 (maximally offensive) -normative offensiveness ratings for the comments. for the first time, we use comparative annotations to detect offensive language. in its simplest form, comparative annotations involve giving the annotators two instances at a time, and asking which exhibits the property of interest to a greater extent. this alleviates several annotation biases present in standard rating scales, such as scale-region bias @xcite , and improves annotation consistency @xcite . however, instead of needing to annotate n instances, one now needs to annotate n 2 instance pairs-which can be prohibitive. thus, we annotate our dataset using an efficient form of comparative annotation called best-worst scaling (bws) @xcite @xcite . by eliminating different offensiveness categories, treating offensiveness as a continuous dimension, and eliciting comparative judgments from the annotators (based on their understanding of what is offensive), we alleviate the issues regarding category definitions and arbitrary category boundaries discussed earlier. by obtaining real-valued offensiveness scores, different thresholds can be used in downstream applications to handle varying degrees of offensiveness appropriately. by framing the task as a comparative annotation task, we obtain consistent and reliable annotations. we also greatly mitigate issues of annotator de-sensitization as one will still be able to recognize if one comment is more offensive than another, even if they think both comments are not that offensive. in contrast to existing resources, which provide annotations for individual comments, our dataset includes conversational context for each comment (i.e. the reddit thread in which the comment occurred). we conduct quantitative and qualitative analyses of the dataset to obtain insights into how emotions, identity terms, swear words, are related to offensiveness. finally, we benchmark several widely-used neural models in their ability to predict offensiveness scores on this new dataset. foot_0 2 related work ## data collection and sampling we extracted reddit data from the pushshift repository @xcite using google bigquery. reddit is a social news aggregation, web content rating, and discussion website. it contains forums called subreddits dedicated to specific topics. users can make a post on the subreddit to start a discussion. users can comment on existing posts or comments to participate in the discussion. as users can also reply to a comment, the entire discussion has a hierarchical structure called the comment thread. we divided the extracted comments into 3 categories based on their subreddit source: 1. topics (50%): contains comments from topic-focused subreddits: askmen, askreddit, twoxchromosomes, vaxxhappened, worldnews, worldpolitics. these subreddits were chosen to cover a diverse range of topics. askreddit, vaxxhappened, worldnews, worldpolitics discuss generic themes. twoxchromosomes contains women's perspectives on various topics and askmen contains men's perspectives. we selected 808 posts from the subreddits based on criteria such as date, thread length, and post length. studies have shown that the primary dimensions of emotion are valence, arousal, and dominance (vad) @xcite @xcite . valence is the positive -negative or pleasuredispleasure dimension. arousal is the excitedcalm or active-passive dimension. dominance is powerful-weak or 'have full control'-'have no control' dimension @xcite . to boost the representation of offensive and emotional comments in our dataset, we up-sampled comments that included low-valence (highly negative) words and those that included high-arousal words (as per the nrc vad lexicon @xcite ). 2 the manually constructed nrc vad lexicon includes 20,000 english words, each with a real-valued score between 0 and 1 in the v, a, d dimensions. in order to do this upsampling, we first defined the valence score of each comment as the average valence score of the negative words within the comment (a negative word is defined as a word with a valence @xmath0.25 in the vad lexicon.) similarly, we defined the arousal score for a comment as the average arousal score of high-arousal words in each comment. (a high-arousal word is defined as a word with an arousal @xmath1.75.) we selected comments from the comment pool such that 50% were from the topics category, 25% from the cmv category, and 25% from the random category. within each category, 33% of the comments were those that had the lowest valence scores, 33% of the comments were those that had the highest arousal scores, and the remaining were chosen at random. ## annotation the perception of 'offensiveness' of a comment can vary from person to person. therefore, we used crowdsourcing to annotate our data. crowdsourcing helps us get an aggregation of varied perspectives rather than expert opinions which can leave out offensiveness in a comment that lies outside the 'typical' offensiveness norms @xcite . we carried out all the annotation tasks on amazon mechanical turk (amt). due to the strong language, an adult content warning was issued for the task. reddit is most popular in the us, which accounts for 50% of its desktop traffic @xcite . therefore, we restricted annotators to those residing in the us. to maintain the quality of annotations, only annotators with high approval rate were allowed to participate. ## data analysis in this section, we analyze various aspects of the data, including: the distribution of scores, the association with identity terms, the relationship with emotion dimensions, the relationship with data source, and the role of swear words. to analyze the data, we placed the comments in 5 equi-spaced score bins of size 0.4 (bin 1: -1.0 to -0.6, bin 2: -0.6 to -0.2, and so on). ## (in appendix a.3). to explore whether specific bins capture specific topics or key-words, we calculated pointwise mutual information (pmi) scores of all the unique words in the comments (excluding stop words) with the five score bins. identity terms a common criticism of the existing offensive language datasets is that in those datasets, certain identity terms (particularly those referring to minority groups) occur mainly in texts that are offensive @xcite @xcite @xcite . this leads to high association of targeted minority groups (such as muslims, females, black people and others) with the offensive class(es). this bias, in turn, is captured by the computational models trained on such datasets. as mentioned earlier, in ruddit, certain words such as gay, trans, male, female, black, white were found to exhibit a relatively higher association with the offensive bins than with the supportive bins. in order to probe the effect of this on the computational models, we created a variant of ruddit by replacing all the identity terms (from the list given in appendix a.4) in the comments with the [group] token and observed the effect on the models' per-formance. we refer to this variant of the dataset as the identity-agnostic dataset. we analyse the models' performance in the next section. offensiveness vs. emotion as discussed earlier, our emotions impact the words we use in text. we examined this relationship quantitatively using ruddit and the nrc vad lexicon (which has intensity scores along the valence, arousal, and dominance dimensions). for each comment in ruddit, we calculated three scores that captured the intensities of the v, a, d words (the averages of the intensities of the v/a/d words in the comment), using the entire lexicon. we then determined the correlation between each of the three scores and the degree of offensiveness. only comments containing at least 4 words from the vad lexicon were considered for the score and correlation calculation. a total of 4831 comments qualified the criteria. see offensiveness vs. data source as mentioned earlier, comments in our dataset come from three different sources -topics, cmv, and random. figure 3 shows the distribution of comments from each source over the score bins. we observed that comments from topics have near equal representation on both sides of the scale, while for the other emotion pearson's r valence -0.301 arousal 0.256 dominance -0.086 ## computational modeling in this section, we present benchmark experiments on ruddit and its variants by implementing some commonly used model architectures. the task of the models was to predict the offensiveness score of a given comment. we performed 5-fold crossvalidation for each of the models. foot_5 bert we fine-tuned bert base @xcite . we added a regression head containing a linear layer to the pre-trained model. we used mse loss as the objective function, batch size of 16, and learning rate of 2e -5 (other hyperparameters same as @xcite ). we used the adamw optimizer with a linear learning rate scheduler with no warm up steps. the model was trained for 3 epochs. (more details in appendix a.5.) hatebert hatebert @xcite is a version of bert pretrained for abusive language detection in english. hatebert was trained on ral-e, a large dataset of english language reddit comments from communities banned for being offensive or hateful. hatebert has been shown to outperform the general purpose bert model on the offensive language detection task when finetuned on popular datasets such as offenseval 2019 @xcite , abuseval @xcite . we fine-tuned hatebert on ruddit and its variants. the experimental setup for this model is the same as that described for the bert model. ## conclusion we presented the first dataset of online comments annotated for their degree of offensiveness. we used a comparative annotation technique called best-worst scaling, which addresses the limitations of traditional rating scales. we showed that the ratings obtained are highly reliable (shr pearson r ≈ 0.88). we performed data analysis to gain insight into the relation of emotions, data sources, identity terms, and swear words with the offensiveness scores. we showed that valence is inversely correlated with offensiveness and arousal is directly correlated with offensiveness. finally, we presented benchmark experiments to predict the offensiveness score of a comment, on our dataset. we found that computational models are not benefiting from the association of identity terms with specific range of scores on the offensiveness scale. in future work, it would be interesting to explore the use of conversational context in computational modeling of offensiveness, as well as studying the interaction between offensiveness and emotions in more depth. we make our dataset freely available to the research community.