Not Just Who or What: Modeling the Interaction of Linguistic and Annotator Variation in Hateful Word Interpretation (2025)

Abstract: Interpreting whether a word is hateful in context is inherently subjective. While growing research in NLP recognizes the importance of annotation variation and moves beyond treating it as noise, most work focuses primarily on annotator-related factors, often overlooking the role of linguistic context and its interaction with individual interpretation.In this paper, we investigate the factors driving variation in hateful word meaning interpretation by extending the HateWiC dataset with linguistic and annotator-level features. Our empirical analysis shows that variation in annotations is not solely a function of who is interpreting or what is being interpreted, but of the interaction between the two. We evaluate how well models replicate the patterns of human variation. We find that incorporating annotator information can improve alignment with human disagreement but still underestimates it. Our findings further demonstrate that capturing interpretation variation requires modeling the interplay between annotators and linguistic content and that neither surface-level agreement nor predictive accuracy alone is sufficient for truly reflecting human variation.

## introduction words play a central role in hate speech by encoding derogatory meanings. the meaning of such words is rarely fixed but highly dependent on context and interpretation which poses a significant challenge for both theoretical understanding and computational modeling of hate speech @xcite . despite growing interest in hate speech detection, there has been little systematic investigation into the semantic and pragmatic mechanisms that underlie how hateful word meanings are interpreted. recent work by @xcite introduced the hatewic dataset, which identified substantial variation and disagreement in judgments about whether a word is hateful in context. models tend to underperform on those cases where annotators disagree. although incorporating annotator demographic information shows modest improvements in model performance, the underlying sources driving these variations remain poorly understood. this aligns with a broader trend in nlp research, that moves away from aggregated judgments to explicitly modeling inter-annotator variation @xcite . yet, the focus in nlp research on label variation in subjective tasks has largely remained on who is interpreting @xcite , with far less attention given to what is being interpreted. while linguistic content has always been the basis for classification, recent subjectivityfocused approaches tend to sideline the role of the content itself. only a few studies acknowledge the role of linguistic ambiguity in subjective labeling @xcite . word in context term sense definition sense sense context ann. gender hateful label domain person aspect length id ratio miss manvers thrust aside a garnished napoleon another name for a millefeuille food notperson 11 36 female ▲▲▲ ✗ plate and attacked her napoleon. pastry. 69 female ▲▲ ✓ he is the napoleon of crime, watson. he napoleon a person having come to dominate person personality/ 41 36 female ▲▲▲ ✗ is the organizer of half that is evil [...] an area of activity through illegality. behavior 75 male ▲ ✓ my shrink said that he was an enabler, shrink a psychiatrist or psychotherapist. person profession 11 36 female ▲▲▲ ✓ bad for me. 4 female ▲▲▲ ✗ within the ongoing search for meaningful predictors of human variation in subjective language interpretation, relatively little attention has been given to the level of word meaning. moreover, most studies only focus on annotator-related features, neglecting the interplay between semantics, linguistic context, and subjective interpretation that shapes how hateful meanings arise. additionally, existing modeling efforts typically emphasize overall performance metrics without assessing whether models replicate the patterns of human variation. yet understanding and modeling such patterns is crucial for nlp systems to meaningfully reflect the subjective nature of language interpretation in sensitive tasks like hate speech detection. addressing these gaps, we augment the hatewic dataset with linguistic and annotatorlevel features ( §3) and empirically show that variation in hateful meaning interpretation is driven not just by who the annotator is or what is being annotated, but by their interaction ( §4). building on this analysis, we propose an evaluation framework that assesses whether bert-based classification models capture this variation ( §5). the results ( §6) demonstrate that while models incorporating annotator-specific inputs can reproduce superficial variation, they substantially underestimate its magnitude and fail to capture the internal structure of variation found in human annotations. ## related work in what follows, we discuss prior work on hateful word meaning in nlp and subjective variation in hate speech detection (hsd), both of which motivate our study. ## data & features to analyze variation in the interpretation of potentially hateful words, we use the hatewic dataset @xcite , which provides contextualized word usages annotated for perceived hatefulness. we further enrich this dataset with additional linguistic and annotator-related features to facilitate a comprehensive empirical analysis of variation. ## empirical analysis we begin our empirical analysis by assessing the overall degree of annotator agreement in the hatewic dataset. we calculate inter-annotator agreement on the original dataset's annotations, with krippendorff's alpha resulting in 0.452. this value reflects moderate agreement and matches the original hatewic paper's findings @xcite foot_1 . moving beyond surface-level agreement, we statistically test the association between our enriched set of linguistic and annotator features, and the binary outcome of agreement with the majority. ## computational modeling in this section, we investigate to what extent computational models with different inputs can capture human variation in annotations. we address this question in the context of the binary classification task that predicts the individual annotations in the hatewic dataset (12k annotations of words in context, labeled hateful or not hateful based on their meaning in that context). we explicitly model and analyze this variation by conditioning predictions on auxiliary inputs such as annotator identity or demographics. the primary goal is to gain insights into alignment with human interpretation variation rather than optimize benchmark performance. ## results & discussion we present results for all ten model configurations in ## conclusion in this paper we demonstrated that the variation in interpretation of hateful word meaning is not merely a function of who the annotator is or what is being annotated, but of the interaction between the two. through empirical analysis of the hatewic dataset, we showed that both linguistic properties of the target word in context and annotator characteristics shape interpretive variation. our evaluation of model alignment with human variation further reveals that although models that incorporate annotator-specific information introduce humanlike variation at a surface level, they still underestimate the magnitude of variation observed in human annotations and generally fail to represent the internal structure of variation. in conclusion, our findings show that capturing human interpretive variation requires modeling the interaction between annotators and linguistic content, and that surfacelevel agreement or predictive accuracy alone does not ensure true alignment with human variation.