Model-Dependent Moderation: Inconsistencies in Hate Speech Detection Across LLM -based Systems (2025)

Abstract: Content moderation systems powered by large language models (LLMs) are increasingly deployed to detect hate speech; however, no systematic comparison exists between different systems. If different systems produce different outcomes for the same content, it undermines consistency and predictability, leading to moderation decisions that appear arbitrary or unfair. Analyzing seven leading models—dedicated Moderation Endpoints (OpenAI, Mistral), frontier LLMs (Claude 3.5 Sonnet, GPT-4o, Mistral Large, DeepSeek V3), and specialized content moderation APIs (Google Perspective API)—we demonstrate that moderation system choice fundamentally determines hate speech classification outcomes. Using a novel synthetic dataset of 1.3+ million sentences from a factorial design, we find identical content receives markedly different classification values across systems, with variations especially pronounced for specific demographic groups. Analysis across 125 distinct groups reveals these divergences reflect systematic differences in how models establish decision boundaries around harmful content, highlighting significant implications for automated content moderation.

## introduction content warning: this paper analyzes hate speech and contains examples of offensive language in a research context. research has shown that online hate speech foot_0 is on the rise, polarizes public opinion, hurts political discourse, and may even have offline impacts on mental and physical health @xcite . further, social media networks have emerged as crucial public spaces for political discourse @xcite , but toxic behavior in these spaces threatens democratic engagement by shutting down deliberation, contributing to polarization, and ultimately discouraging citizen participation @xcite . in an effort to curb or moderate online hate speech, leading companies have released models and toolkits-from openai's and mistral's moderation endpoints to frontier models like claude 3.5 sonnet, @xcite deepseek v3-that promise automated content filtering at scale. however, an open question remains regarding the consistency and effectiveness of these automated content moderation and hate speech detection (hsd) systems. to date, no systematic evaluation has compared how these systems analyze and classify content related to different identities, communities, or political contexts. this lack of transparency raises serious concerns about arbitrariness and fairness. if two systems produce different outcomes for the same piece of content-flagging it as hate speech in one case but not in another-it undermines the legitimacy of the moderation process. such inconsistency can erode public trust, create perceptions of bias, and lead to uneven protections, where some groups are disproportionately exposed to harmful speech while others are shielded. inconsistencies across moderation systems can arise from both technical and social factors. at the technical level, models employ different architectures, training data, methodologies, and classification thresholds-choices that can lead to divergent decisions even when analyzing identical content. however, these technical variations are likely intertwined with deeper social complexities in how systems encode cultural assumptions and societal biases @xcite . common nlp practices for hsd often fail to capture the contextual nature of hate speech, risking harm to marginalized communities @xcite . when platforms implement moderation systems, they often remain opaque, unaccountable, and poorly understood, with unverified classification metrics that raise fundamental questions about whether automated content moderation systems should be deployed at all, particularly given their potential to disproportionately impact marginalized communities @xcite @xcite . given these technical and social factors, we hypothesize that different content moderation and hsd systems will show substantial disagreement in their classifications of identical content, with particularly large variations in how they evaluate content targeting different groups. we expect these disparities to extend beyond straightforward hate speech to include systematic differences in false positive rates for benign content and inconsistent handling of implicit hate speech disguised within seemingly positive language. this paper evaluates seven leading models to understand how model selection impacts filtering outcomes. through analysis of over 1.3 million sentences across 125 demographic groups, we find that these systems show substantial variation in their classification of identical content-what one flags as harmful, another might deem acceptable. the disparities reflect fundamental differences in how each model conceptualizes unacceptable speech, going beyond technical variations in architecture or training. these findings have significant implications for online discourse and user protection, as a platform's choice of moderation system fundamentally shapes the nature of permissible speech within its digital spaces. ## related work content moderation and hsd methods have evolved significantly over time, with research examining general hsd @xcite , antisemitic hate speech @xcite , and sexism @xcite . early approaches used rule-based systems with predefined linguistic patterns and keyword matching @xcite , but these proved too rigid and achieved very low recall rates on real-world data @xcite . supervised learning methods subsequently emerged as the next stateof-the-art approach @xcite , with the first iteration of perspective api becoming a widely adopted solution across many platforms. yet these systems, while more sophisticated than their rule-based predecessors, still faced limitations in adapting to novel forms of harmful content @xcite . the field has since shifted toward zero-shot/few-shot learning (zsl) methods, particularly those leveraging large language models, which have demonstrated superior flexibility and contextual understanding @xcite , with researchers often pointing to their remarkable ability to annotate text data @xcite . others, however, have highlighted their numerous limitations in annotation @xcite @xcite . further research has shown these systems may disproportionately penalize language used by certain demographic groups (e.g., speakers of african american english) @xcite , which can reflect and reinforce societal biases @xcite . specific to hsd, researchers have shown models have problems with cross-lingual hsd @xcite . for example, chatgpt was shown to have poor performance on hsd for english, and even worse performance for non-english languages @xcite . further, research has shown that large language models can perpetuate discriminatory biases in toxic speech detection applications, manifesting as specific harms targeting protected groups @xcite . more recently, a new category of content moderation systems has emerged: dedicated moderation endpoints, such as those offered by openai (ope-nai, 2025) and mistral @xcite . these endpoints represent a hybrid approach, combining the advantages of zsl methods with specialized training and optimization for content moderation tasks, marking the latest evolution in automated content moderation technology. along with the development of new content moderation systems, recent work has created datasets for hsd in low-resource languages including, sinhala and tamil @xcite , hindi and nepali @xcite , and hausa @xcite . despite the wealth of research into creating new models and datasets, llm-based content moderation systems lack comparative studies on how they respond to identical content across demographic groups. this research gap raises concerns about potential detection biases, possibly providing inconsistent protection levels for different communi-ties, particularly troubling for protected and vulnerable populations these systems aim to safeguard. without a systematic comparison, similar content may be inconsistently classified, undermining the reliability and fairness of online safety measures. our work addresses this critical gap by providing the first large-scale comparative analysis of how leading content moderation systems respond to potentially harmful content. ## conclusion our systematic evaluation of the seven different models reveals significant inconsistencies in how they moderate hate speech. through an analysis of over 1.3 million sentences generated from a full factorial design, we show wide variation in how hsd systems evaluate hate speech, especially for speech targeting different demographic groups. our primary aim was to establish a solid empirical foundation for understanding inconsistencies in hate speech detection across llm-based systems. this work helps lay the groundwork for more sophisticated theoretical frameworks in automated content moderation, particularly as llms become increasingly central to these systems. the magnitude of these disparities is striking. when evaluating identical instances of hate speech targeting different groups, we observed classification scores that spanned almost the complete measurement scale across different systems. even more concerning, these differences are amplified for certain demographic groups, with some communities receiving markedly different levels of protection depending on which moderation system is deployed. our analysis of decision boundaries reveals that these disparities exist not only for the specific hate value assigned, but also for the binary classifications. specifically, frontier llms demonstrate group-specific probability thresholds for hate speech classification, suggesting embedded biases in their decision-making processes. these biases extend to false positive rates and implicit hate speech detection, where systems showed dramatic disagreements. this work addresses a critical gap in our understanding of automated content moderation by providing the first large-scale comparative analysis of how different systems handle identical content across diverse demographic groups. the substantial variation we observed suggests that current automated moderation systems, despite their sophisticated architectures, may be perpetuating rather than mitigating existing social inequities in online spaces. consequently, addressing these issues requires several key actions: developing standardized benchmarks with consistent decision boundaries across demographics, implementing model ensembles to reduce group-specific variation, forging industry-academic partnerships to establish consistent standards, and enhancing transparency in moderation system documentation.