Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive (2023)

Abstract: Offensive speech detection is a key component of content moderation. However, what is offensive can be highly subjective. This paper investigates how machine and human moderators disagree on what is offensive when it comes to real-world social web political discourse. We show that (1) there is extensive disagreement among the moderators (humans and machines); and (2) human and large-language-model classifiers are unable to predict how other human raters will respond, based on their political leanings. For (1), we conduct a ***noise audit*** at an unprecedented scale that combines both machine and human responses. For (2), we introduce a first-of-its-kind dataset of ***vicarious offense***. Our noise audit reveals that moderation outcomes vary wildly across different machine moderators. Our experiments with human moderators suggest that political leanings combined with sensitive issues affect both first-person and vicarious offense. The dataset is available through https://github.com/Homan-Lab/voiced.

## introduction offensive speech on web platforms is a persistent problem with wide-ranging impacts @xcite @xcite . among many reasons why moderation fails to eliminate the problem, is the reality that people often disagree on what is offensive @xcite . in this study, we break new ground by investigating disagreement through a large-scale noise audit and by introducing the notion of vicarious offense. ## discussion and conclusion in this paper, we present two novel perspectives on moderating social media political discourse: disagreement between machine moderators, and disagreement between human moderators. our key contributions are (1) a comprehensive noise audit of machine moderators; (2) voiced, an offensive speech dataset with transparent annotator details; traditional supervised learning paradigm assumes existence of gold standard labels. while recent lines of work have investigated disagreement among annotators that stems from the inherent subjectivity of the task @xcite @xcite , our analyses of political discussions on highly sensitive issues reveal that there could be practically no agreement among annotator groups and depending on who we ask, we can end up with wildly different gold standard labels reminiscent of alternative facts @xcite . our current work is primarily descriptive. we address the elephant in the room and quantify the challenges of offensive content moderation in political discourse both from the machine and human moderators' perspectives. we believe our dataset will open the gates for modeling ideas considering multiple vantage points and yielding more robust systems. our study raises the following points to ponder upon. our dataset was limited to english posts, primarily focusing on one platform (youtube). ideally, a study would consider other platforms and languages to provide a greater degree of external validity. we generally work with hard labels and where bayesian truth serum is about predicting uncertainty, we do not model uncertainty in this study.