Hateful Word in Context Classification (2024)

Abstract: Hate speech detection is a prevalent research field, yet it remains underexplored at the level of word meaning. This is significant, as terms used to convey hate often involve non-standard or novel usages which might be overlooked by commonly leveraged LMs trained on general language use. In this paper, we introduce the Hateful Word in Context Classification ( HateWiC ) task and present a dataset of ~4000 WiC-instances, each labeled by three annotators. Our analyses and computational exploration focus on the interplay between the subjective nature (context-dependent connotations) and the descriptive nature (as described in dictionary definitions) of hateful word senses. HateWiC annotations confirm that hatefulness of a word in context does not always derive from the sense definition alone. We explore the prediction of both majority and individual annotator labels, and we experiment with modeling context- and sense-based inputs. Our findings indicate that including definitions proves effective overall, yet not in cases where hateful connotations vary. Conversely, including annotator demographics becomes more important for mitigating performance drop in subjective hate prediction.

## introduction this paper introduces the hateful word in context classification (hatewic) task, which aims to determine the hatefulness of a word within a specific context, as illustrated in figure 1 . we argue that hateful word senses are not enough in focus within hate speech detection (hsd) research, and not descriptive only, but highly subjective, asking for another approach than other lexical semantic tasks like word sense disambiguation (wsd). hateful senses are not enough in focus within hsd research. the current focus of hsd research predominantly revolves around the classification of entire utterances, such as social me-figure 1 : illustration of the hatewic classification task and a conceptual semantic space that underlies the targeted phenomenona of hate-heterogeneous word senses, highlighting the distinction between the descriptive aspects (e.g. cookie or person) and hateful connotation. dia posts @xcite . within these utterances, lexical cues frequently play a significant role in the decisionmaking process. yet, the computational modeling of context-specific hateful word meanings remains largely unexplored, with a few exceptions in this direction @xcite . lms commonly employed in hsd systems demonstrate effective word meaning modeling @xcite ), but they tend to lack sensitivity to domain-specific, non-standard or novel word senses @xcite . this insensitivity becomes particularly critical in detecting hateful word meanings, that are used in unconventional or emerging contexts as the evolution of societal events gives rise to the continuous invention of novel expressions of hate @xcite . words within the established lexicon, like oreo, whose primary meaning may not have any negative connotations (a cookie), are repurposed to convey hate towards particular groups or individuals (e.g. based on ethnicity). hateful senses are not descriptive only. following theoretic work by @xcite , hateful terms could be positioned along a meaning continuum from descriptive to expressive, closer to but not at the expressive outer end. the descriptive component comprises the truth-conditional attributes of a term, often recorded in dictionary definitions. the expressive component, i.e. the connotation of a term, concerns speakers' attitudes and emotions, making it highly context-specific and subjective. a word's sense definition could imply a hateful connotation, but this is not always the case, such as when used in a playful or self-identifying way (e.g. the third usage in figure 1 ). thus, a word's hateful connotation is not exclusively tied to its descriptive definition, a phenomena which we term as hate-heterogeneous senses, but depends on various contextual factors like conveyed content or the reader's identity. this aspect is often overlooked in hsd systems, typically developed using data reflecting a single (majority) perspective @xcite . ## the hatewic dataset we introduce the hatewic dataset, which includes hate ratings for words within example usages along with their word sense definitions which may be hate-heterogeneous, as illustrated in figure 1 . we describe the dataset construction below. ## hatewic classification our hatewic dataset enables the development and evaluation of computational methods for predicting whether the meaning of a target term is hateful within a specific context. figure 2 provides an illustration of the primary methodological pipeline we present in this paper. we introduce various classification methods that differ with respect to the sense representations (outlined in 4.1) and incorporation of annotator information (4.2) as input to a classification model (4.3), or that leverage an instruction-tuned llm (4.4). ## evaluation setup we evaluate our proposed methods using various test setups on the hatewic dataset (5.1). additionally, we compare our methods with the work of @xcite , as described in 5.2. ## results this section presents the results of our proposed methods on the hatewic classification, evaluated using the above outlined setups. ## discussion our study offers valuable insights into the detection of hate speech through the lens of lexical semantics, introducing the hatewic dataset and presenting classification experiments. the negligible difference observed in our experimental outcomes between hatebert and general (wsd) models not only questions the efficacy of extensive training on hate speech data for accurately capturing hateful semantics, but also underscores the necessity of a more nuanced approach beyond the existing lexical semantic methods for tasks like hatewic classification. our results demonstrate the impact of incorporating sense definitions and annotator characteristics on model performance, particularly in scenarios involving out-of-vocabulary (oov) terms or high subjectivity. to define or not define? hateful terms, according to lexical semantic theory, primarily contain an expressive component but not exclusively. incorporating sense definitions into our methods, to encompass the descriptive component of hateful terms, yielded mixed results. overall, embedded wiktionary definitions proved highly effective, outperforming word in context (wic) embeddings alone. t5-generated definitions demonstrated the lowest accuracy on their own but performed equally or slightly better than wic-embeddings only when concatenated with wic-embeddings. however, in cases with more variation in the subjective ratings, the performance of all embeddings dropped significantly but most pronounced for wiktionary definition embeddings, though to a lesser extent for t5-generated definitions (with a drop difference of up to 23%). this highlights the usefulness of automatically generating context-specific definitions for subjective lexical semantic tasks like hatewic classification. future research will focus on more advanced definition generation techniques, possibly leveraging larger models or fine-tuning on wiktionary definitions, while avoiding overreliance on dictionary definitions as the ultimate standard. to individualize anyway? the low interannotator agreement in our dataset underscores the importance of considering individual annotator perspectives in hate speech detection. our experiments incorporating annotator information in our computational methods proved beneficial, particularly in cases of annotator disagreement or hateheterogeneous definitions, where including annotator information mitigated accuracy decline by up to 11%-points. this highlights the value of personalizing models to account for subjectivity in annotations. future research could explore additional annotator information and conduct ablation experiments to identify the most effective aspects for hatewic classification. to consider as well? our study paves the way to obtaining deeper insights into the relationship between hateful and non-hateful word senses. for instance, whether certain semantic relations (e.g. metaphorical, metonymical), categories (e.g. food, animals), or attributes (e.g. color, material) are more likely to distinguish between hateful and nonhateful senses. and even next-level, whether these discriminators are language-specific or show cross-language parallels. identifying such consistencies between (non-)hateful senses could enhance the (automatic) discrimination between them. ## conclusion this paper introduces the hateful word in context classification (hatewic) task, addressing the underexplored area of subjective hateful word meanings within specific contexts. we present the hatewic dataset, comprising about 4000 wicinstances, each annotated with three hateful ratings. our study focused on the interplay between descriptive and subjective aspects of hateful word senses. we addressed the prediction of both majority and individual annotator labels. we experimented with different types of inputs to our classification system, including sense definitions and annotator demographics. we demonstrated the impact of these factors on model performance, particularly in cases involving out-of-vocabulary terms or high subjectivity. the incorporation of established sense definitions proved highly effective overall but demonstrating diminished performance in less descriptive scenarios. conversely, including annotator characteristics proved beneficial, particularly in cases of annotator disagreement or hate-heterogeneous definitions. these findings underscore the value of personalizing models to account for subjectivity in annotations. furthermore, our results suggest the potential usefulness of automatically generating definitions for subjective lexical semantic tasks like hatewic classification.