H ate M oderate: Testing Hate Speech Detectors against Content Moderation Policies (2024)

Abstract: To protect users from massive hateful content, existing works studied automated hate speech detection. Despite the existing efforts, one question remains: Do automated hate speech detectors conform to social media content policies? A platform’s content policies are a checklist of content moderated by the social media platform. Because content moderation rules are often uniquely defined, existing hate speech datasets cannot directly answer this question. This work seeks to answer this question by creating HateModerate, a dataset for testing the behaviors of automated content moderators against content policies. First, we engage 28 annotators and GPT in a six-step annotation process, resulting in a list of hateful and non-hateful test suites matching each of Facebook’s 41 hate speech policies. Second, we test the performance of state-of-the-art hate speech detectors against HateModerate, revealing substantial failures these models have in their conformity to the policies. Third, using HateModerate, we augment the training data of a top-downloaded hate detector on HuggingFace. We observe significant improvement in the models’ conformity to content policies while having comparable scores on the original test data. Our dataset and code can be found on https://github.com/stevens-textmining/HateModerate.

## introduction social media platforms such as facebook, reddit, and twitter/x have facilitated users to exchange information, but they also expose users to undesirable content, including hateful speech, misinformation, graphic violence, and pornography. to protect users from a massive amount of hateful content, existing work has been vigorously investigating new nlp approaches and providing new resources and open-source tools for studying hate speech ## conclusions in this paper, we propose a dataset hatemoderate, which includes hateful and non-hateful examples matching the 41 community standards guideline policies of facebook. first, we leverage manual annotation with 28 graduate students followed by information retrieval, data augmentation, and verification to construct a dataset containing both hateful and non-hateful examples. second, we use hate-moderate to test the failures of state-of-the-art hate detection models. we find that popular content moderation models frequently make mistakes for both hateful and non-hateful examples. finally, we observe that by augmenting the training data with hatemoderate, the model can better conform to hatemoderate while having a comparable performance to the original test data. our study highlights the importance of investigating hate speech detectors' conformity to content policies.