Capturing Covertly Toxic Speech via Crowdsourcing (2021)

Abstract: We study the task of labeling covert or veiled toxicity in online conversations. Prior research has highlighted the difficulty in creating language models that recognize nuanced toxicity such as microaggressions. Our investigations further underscore the difficulty in parsing such labels reliably from raters via crowdsourcing. We introduce an initial dataset, COVERTTOXICITY, which aims to identify and categorize such comments from a refined rater template. Finally, we fine-tune a comment-domain BERT model to classify covertly offensive comments and compare against existing baselines.

## introduction warning: this paper contains content that may be offensive or upsetting. tools identifying toxic content are essential for moderation of problematic online conversations @xcite . generally, "toxic" is an umbrella label for various types of disrespectful comments, including but not limited to identity attacks, profanity and threats, that could encourage a user to leave a conversation. besides, "toxicity", other terms like "hate-speech", and "online violence" have also been used to refer to similar problems @xcite ) -yet no single conclusive definition exists. in this work, we chose to use "toxicity" because our research suggests that "toxicity" is an easy concept for annotators to understand, meaning we can gather opinions from a diverse range of people, allowing us to capture the inherent subjectivity of the concept. in other words, it was easier for more people to agree on what constituted "toxic" speech-comments likely to make someone leave a conversation-than it was for people to agree on other terms to describe problematic comments. 1 1 https://jigsaw.google.com/the-current/toxicity/ previous works have published baseline toxicity detection datasets @xcite along with models for detecting such content @xcite @xcite @xcite . however, @xcite recently established that covert or veiled toxicity is overwhelmingly misclassified by leading models. perspective, a free api that uses machine learning to score comments for toxicity, is widely used to aid content moderators on online platforms @xcite . as example, the deeply offensive comment masked as humor "what's the difference between boy scouts and jews? boy scouts come back from camp." is assigned a toxicity probability score of 0.31 by perspective. such comments can be as harmful to participants as more overt toxicity @xcite . covert toxicity is an umbrella term which includes types of toxicity that may not be immediately obvious. covertly toxic comments may use obfuscation, code words, suggestive emojis, dark humor, or sarcasm. it also includes subtle aggression, such as the user comment "'slurp slurp", says chang', which contains a stereotype regarding asian individuals. microaggressions may be unintentional but can have profound psychological repercussions in aggregate @xcite @xcite . in this work, we seek to close the gap in toxicity models. our two main contributions include: ogbonnaya-ogburu et al. (2020) provides a call for race-conscious efforts within the human-computer interaction community. for example, @xcite argues that asian american pacific islander online community members need support. further @xcite points out that the perceived toxicity of a comment is influenced by a variety of factors not limited to cultural background, rater bias, context and cognitive bias. in particular, some of these efforts include explorations of microaggressions in social media posts @xcite with disadvantaged groups, surfacing contextual hate speech or words @xcite that have alternate meanings or veiled toxicity like codewords, adversarially generated or novel forms of offense @xcite , and exploring humor used as a tool for propagating hate @xcite . small datasets exist @xcite ) specifically for exploring microaggressions in online conversations. the social bias inference corpus (sbic), a dataset of social media posts with structured annotations of thousands of demographic groups @xcite is another example. to filter such datasets for veiled toxicity, @xcite proposed a procedure using probing examples. our paper builds on previous work with a strategy for consistently rating covertly toxic content, a shared dataset for training, and a baseline covert toxicity model. ## covert toxicity in order to formulate a covertly toxic dataset, we first extracted a set of candidate comments from the civilcommentsidentities dataset @xcite as described in section 4.1. we manually labeled a random sample of these comments, seeking those which could have a negative impact on online conversations, but were frequently missed by raters. from this sample, we defined broad categories of covert toxicity types. see figure 1 for an example. ## datasets the coverttoxicity foot_0 dataset consists of covertly offensive rater labels using the described template on a subset of civilcommentsidentities data @xcite . ## results comments with higher covertly offensive rating fractions (i.e. rater agreement), were more likely to be categorized as containing obfuscation, and less likely to be categorized as containing humor or emoticons/emoji. additionally, obfuscation and comment length are anti-correlated in our dataset. for example, the new moon with face emoji in 'but that's just "their nature". ## conclusion we iterate on rater feedback to create an initial baseline dataset, coverttoxicity, that encapsulates a variety of often mislabeled online toxicity. while progress is still needed in extracting coherent rater signals and modelling, our initial work demonstrates the possibility of capturing veiled toxic language with machine learning models.