Using LLM s and Preference Optimization for Agreement-Aware H ate W i C Classification (2025)

Abstract: Annotator disagreement poses a significant challenge in subjective tasks like hate speech detection. In this paper, we introduce a novel variant of the HateWiC task that explicitly models annotator agreement by estimating the proportion of annotators who classify the meaning of a term as hateful. To tackle this challenge, we explore the use of Llama 3 models fine-tuned through Direct Preference Optimization (DPO). Our experiments show that while LLMs perform well for majority-based hate classification, they struggle with the more complex agreement-aware task. DPO fine-tuning offers improvements, particularly when applied to instruction-tuned models. Yet, our results emphasize the need for improved modeling of subjectivity in hate classification and this study can serve as foundation for future advancements.

## introduction classification tasks involving subjective human judgment often exhibit annotator disagreement. this issue is particularly evident in hate speech detection, where the perception of hatefulness varies depending on context and individual interpretation @xcite . ignoring disagreement in annotations can lead to biased systems that fail to account for minority perspectives @xcite . addressing this variability requires models to go beyond binary classification and account for the degree of disagreement among annotators @xcite . one task that exemplifies this challenge is hateful word in context (hatewic) classification, which determines whether the meaning of a given term is hateful within a specific context @xcite . the initial work introducing hatewic explored several bert-based embedding learning strategies, demonstrating that incorporating additional input information, such as word definitions and annotator-specific data, can enhance performance. in particular, the inclusion of annotator information appears promising given the subjective nature of the task. in this paper, we propose a novel approach to the hatewic task, leveraging large language models (llms) fine-tuned via preference optimization. recent advancements in preference-based learning, particularly reinforcement learning from human feedback (rlhf) @xcite and its more computationally efficient alternatives like direct preference optimization (dpo) @xcite , have driven significant improvements in llm alignment for tasks such as questionanswering and summarization @xcite . however, their potential in subjective hate classification, where human disagreement plays a crucial role, remains unexplored. we focus on two hatewic task variants: 1) hate classification, where the goal is to predict the majority annotator judgment on whether a term is hateful and 2) agreement-aware hate classification, a novel task variant that we introduce to explicitly model annotator disagreement by estimating the proportion of annotators who classify a term as hateful. this task merges classification and annotator disagreement, capturing the subjective nature of hatefulness. contributions we introduce a preference-based fine-tuning approach for hatewic, leveraging direct preference optimization (dpo) to align llms with human judgments of hatefulness. we propose a novel variant of hate classification that integrates annotator disagreement directly into the learning objective, as illustrated in figure 1 . evaluating both task variants, we compare the effect of dpotuning on models that are instruction tuned or not. our results show that models without preference tuning perform effectively for general (majoritybased) hate classification, and additional preference optimization does not yield further improvements. however, for agreement-aware hate classification, dpo fine-tuning enhances performance when applied to instruction-tuned models. despite these improvements, the second task remains challenging for llms. these findings highlight both the potential and limitations of our approach, emphasizing the need for further research to better model annotator disagreement. 1 2 related work ## tasks & data in this section, we briefly describe the hatewic dataset and the tasks addressed in this study, outlining how we create preference pairs for dpo training. ## methods this section details the experimental set-up for our experiments, including the dpo training paradigm, the chosen models and the evaluation pipeline. ## results & discussion this section presents the results of our methods on two variants of the hatewic task. moreover, our sft-models are competitive with the best bert-based approach as reported in the original hatewic paper @xcite (0.78 accuracy). the authors also reported that zero-shot llama 2 performed worse (0.68 accuracy). our results align with their conclusion that, despite their strong performance elsewhere, llama models do not demonstrate superior performance over bertbased methods on this task. dpo enhances agree-aware hate classification, but the task remains challenging. as can be seen in when examining class-wise performance, all models struggle most with the all category, followed by few and most, while the none category yields the best results. this pattern suggests that llms find it easier to align with clear-cut non-hate cases but struggle when the input is hateful or ambiguous, thus prioritizing caution over recall, potentially due to the challenges of handling subjectivity and disagreement between annotators. appendix d provides a more detailed error analysis. instruct-models can benefit from task-specific preference tuning, sft-models not. in task 1 we observe that task-specific dpo fine-tuning has minimal impact on the sft model, but it substantially improves the performance of the instruct model. similarly, in task 2, the effect of dpo fine-tuning varies between the two base models: it improves performance for the instruct model, while it degrades the performance of the sft model. these results suggest that for these tasks, sft models appear less flexible to incorporating task-specific preference signals whereas instruction-tuned models benefit from such additional preference fine-tuning. ## conclusion this paper addresses the challenges of incorporating subjective human judgment, particularly annotator disagreement, in tasks like hatewic classification. we introduce a novel variant of the task, agreement-aware hate classification, which explicitly models the variability in human judgments. to tackle this task, we explore approaches using llms with dpo. our findings show that pre-trained llms perform effectively for majoritybased hate classification. however, these models struggle with the added complexity of agreementaware hate classification. while dpo fine-tuning shows promise in enhancing performance, particularly when applied to instruction-tuned models, our study also emphasizes that further research is needed to better capture the subjective nature of hate speech detection. the novel task we present could serve as valuable foundation for future efforts.