Is LLM an Overconfident Judge? Unveiling the Capabilities of LLM s in Detecting Offensive Language with Annotation Disagreement (2025)

Abstract: Large Language Models (LLMs) have become essential for offensive language detection, yet their ability to handle annotation disagreement remains underexplored. Disagreement samples, which arise from subjective interpretations, pose a unique challenge due to their ambiguous nature. Understanding how LLMs process these cases, particularly their confidence levels, can offer insight into their alignment with human annotators. This study systematically evaluates the performance of multiple LLMs in detecting offensive language at varying levels of annotation agreement. We analyze binary classification accuracy, examine the relationship between model confidence and human disagreement, and explore how disagreement samples influence model decision-making during few-shot learning and instruction fine-tuning. Our findings reveal that LLMs struggle with low-agreement samples, often exhibiting overconfidence in these ambiguous cases. However, utilizing disagreement samples in training improves both detection accuracy and model alignment with human judgment. These insights provide a foundation for enhancing LLM-based offensive language detection in real-world moderation tasks.

## introduction motivation. a fundamental challenge in offensive language detection is annotation disagreement-cases where human annotators provide conflicting labels for the same text. disagreement arises due to differences in individual perception, cultural context, and linguistic ambiguity, making offensive language detection inherently subjective @xcite @xcite . however, prior research predominantly treats this task as a binary classification problem, * corresponding author. while large language models (llms) have been extensively applied to offensive language detection @xcite , existing studies primarily evaluate their performance on datasets with binary labels, overlooking their ability to handle cases where annotators disagree. this oversimplification limits our understanding of how well llms align with human judgment in ambiguous cases. moreover, models may exhibit overconfidence in cases where human annotators themselves are uncertain, raising concerns about their reliability for real-world moderation. research objectives. to bridge this gap, we systematically evaluate llms' ability to process disagreement samples, analyzing both classification accuracy and model confidence. through this, we seek to determine whether llms can effectively navigate subjective offensive language judgments and align with human reasoning. an ideal model should express high confidence for unanimously labeled cases and lower confidence for ambiguous samples, reflecting their inherent uncertainty. our study provides insights into whether llms capture these nuances or exhibit overconfidence in disagreement cases, which could undermine their trustworthiness in content moderation. this paper systematically investigates how llms handle annotation disagreement in offensive language detection. specifically, we address the following research questions: (rq1) to what extent can llms accurately detect offensive language in cases of human annotation disagreement? (rq2) how do disagreement samples shape llm learning and influence decision-making? to answer rq1, we evaluate multiple opensource and closed-source llms in a zero-shot setting, analyzing both classification accuracy and agreement degree example hard label soft label n ++ stand for something or else fall for anything. #blacklivesmatter hello world! what a great day to be alive #trump2020 #maga 0 0 n + come on man! lock'em up!!! #maga not the first time. you all misspelled #blacklivesmatter. speak up! @user 0 0.2 n 0 set fire to fox news (metaphorically) @user is outing #black-lives-matter as a cult! he is correct! 0 0.4 o 0 #disgusting #democrats terrorize old folks just before #elec-tions2020 i love this shit! #blacklivesmatter 1 0.6 o + @user you're a bumbling fool #elections2020 elections 2020: red rapist v. blue racist 1 0.8 o ++ y'all trending about kpop stans instead of #blacklivesmatter big fack you crazy idiots. this is batshit bullshit. #elections2020 1 1 the relationship between annotation agreement and model confidence. for rq2, we examine the impact of disagreement samples in few-shot learning and instruction fine-tuning, assessing how different agreement levels affect model performance. contributions. we summarize the contributions of this paper as follows: ## rq1: evaluating llms on offensive language with annotation disagreement in this section, we evaluate the ability of llms to detect offensive language in a zero-shot setting. we focus on two key aspects: (1) binary classification accuracy, assessing how effectively models distinguish offensive from non-offensive language across varying annotation agreement levels, and (2) model confidence, analyzing whether llms exhibit appropriate uncertainty in ambiguous cases. these aspects are essential for determining whether llms can reliably perform offensive language detection in real-world scenarios, where human annotators often disagree. ## rq2: impact of disagreement samples on llm learning in this section, we examine how samples with varying annotation agreements influence llm performance during the learning phase. we focus on two key learning paradigms: few-shot learning and instruction fine-tuning. specifically, we explore the impact of both single-category agreement samples and different agreement-level combinations on model performance. ## related work large language model. in recent years, large language models (llms) have rapidly emerged, showcasing extensive world knowledge and strong reasoning capabilities @xcite . many researchers have proposed diverse tasks to deeply analyze the relationship between the model's outputs and human judgments @xcite . in addition, the confidence of llms in their outputs has also attracted attention from researchers, which is often used to assess the reliability and robustness of the generated content @xcite . various methods for estimating confidence have been proposed @xcite @xcite . in this study, we employ the most straightforward approach, selfconsistency @xcite , to estimate the model's confidence. offensive language detection. researchers have developed various methods for detecting offensive language @xcite @xcite . recent studies have argued that treating offensive language detection as a binary classification is an idealized assumption @xcite @xcite , as annotation disagreement is inherent in datasets for such subjective task @xcite . using majority voting for annotation agreement leads to information loss @xcite , as these disagreements arise from the subtlety of the samples, not labeling errors @xcite . @xcite emphasizes that detection models should recognize this disagreement, rather than just improving classification performance. recently, several studies have begun evaluating the potential of llms for detecting offensive language @xcite , and designing detection methods based on them @xcite . some studies @xcite leverage the generative capabilities of llms to provide explanations for offensive language, assisting human annotation. furthermore, @xcite ; @xcite assess the sensitivity of llms to demographic information in the context of offensive language. though great efforts have been made, these studies lack focus on the phenomenon of offensive language with annotation disagreement. in this paper, we aim to fill this research gap. ## conclusion this study examines how llms handle annotation disagreement in offensive language detection, a critical challenge in real-world moderation. we evaluate multiple llms in a zero-shot setting and find that while they perform well on unanimously agreed-upon samples, their accuracy drops significantly for disagreement cases. moreover, their overconfidence leads to rigid predictions, misaligning them with human annotations. to address this, we investigate the impact of disagreement samples in few-shot learning and instruction fine-tuning. our results show that incorporating these samples improves detection accuracy and human alignment, enabling llms to better capture the subjective nature of offensive language. we further find that balancing agreement levels in training data prevents overfitting to ambiguous cases, ensuring model robustness. key findings of this work include: (1) a systematic evaluation of llms on annotation disagreement, (2) insights into how disagreement samples improve learning, and (3) guidelines for leveraging disagreement-aware training strategies. these results emphasize the need for model calibration techniques to mitigate overconfidence and for training strategies that incorporate disagreement to improve generalization. future research should explore dynamic finetuning approaches and confidence-aware moderation systems to bridge the gap between llm decisions and human subjectivity. we hope that our findings and insights will contribute to the further development of offensive language detection, llmas-a-judge, and human-ai collaboration.