Hatevolution: What Static Benchmarks Don’t Tell Us (2025)

Abstract: Language changes over time, including in the hate speech domain, which evolves quickly following social dynamics and cultural shifts. While NLP research has investigated the impact of language evolution on model training and has proposed several solutions for it, its impact on model benchmarking remains under-explored. Yet, hate speech benchmarks play a crucial role to ensure model safety. In this paper, we empirically evaluate the robustness of 20 language models across two evolving hate speech experiments, and we show the temporal misalignment between static and time-sensitive evaluations. Our findings call for time-sensitive linguistic benchmarks in order to correctly and reliably evaluate language models in the hate speech domain.

## introduction language continuously evolves adapting to social and cultural dynamics @xcite @xcite , e.g., words gain new meanings or lose their existing ones, words shift polarity, and new words emerge. this language evolution challenges nlp models across different domains @xcite , with hate speech being one of the most challenging due to the semantic broadening of harmrelated concepts in the past 50 years @xcite , frequent changes in words' polarity @xcite , and reclaimed language @xcite . indeed, di @xcite recently show that language models' distributional knowledge can be enhanced with temporal linguistic knowledge to effectively detect and explain hateful content. while nlp research has extensively investigated the impact of hate speech evolution in model training paradigms, showing that temporal misalignment between training and test sets leads to decreasing performance over time (i.e., temporal bias) across models and languages @xcite , the implications of evolving hate speech in model benchmarking have not been explored. yet, hate speech benchmarks play a crucial role as they are widely embedded in the safety evaluation of language models (e.g., @xcite @xcite , which are increasingly used in real-world applications and decision making @xcite . although these benchmarks provide a comprehensive comparison of language models that would not be possible with held-out test sets, they face the same issue: they are static. in other words, they are grounded to the specific timestamp in which they were developed, and consequently they cannot account for language change. we argue that evolving hate speech plays a role in the reliability of static model benchmarking over time, potentially leading to an overestimation of language models' safety in light of well-known issues like temporal bias and benchmark saturation @xcite . therefore, we seek to answer "how does static hate speech benchmarking correlate with evolving language?". by providing empirical evidence of this temporal challenge in model benchmarking, we hope our study will raise awareness in the risks associated with static evaluations of language models, and will fuel research towards time-sensitive evaluations of nlp models in a similar way in which studies that investigated the impact of language evolution on model training led to the development of alternative solutions, e.g., temporal attention @xcite or the injection of time-sensitive lexical information @xcite . ## evolving hate speech to answer our research question, we first design two experiments for evolving hate speech detection accounting for different aspects of language evolution, and we propose two time-sensitive metrics to evaluate language models. then, we evaluate the same models on static hate speech benchmarks, and we measure the correlation in models' ranking across time-sensitive and static evaluations. 1 experiment 1: time-sensitive shifts. we investigate contextual evolution of hate speech, focusing on time-sensitive shifts, such as semantic, topical, and polarity changes. for instance, the word 'gammon' has undergone multiple transformations simultaneously @xcite : a semantic change from referring to food (ham) to a political insult; a topic shift towards political discourse; and a polarity shift towards negativity. in contrast, certain terms targeting asian communities predominantly experienced a polarity shift, becoming more offensive during the coronavirus pandemic @xcite . moreover, timesensitive shifts might manifest as changes in the cultural perception of what is considered offensive, e.g., reclaimed slurs. these time-sensitive shifts are notoriously difficult to disentangle @xcite , and we do not attempt to do so in this work. instead, we aim to quantify how their complex interplay affects model performance over time, and in turn how this time-sensitive performance correlates with performance on static benchmarks. to study this, we use the english version of the singapore online attack dataset @xcite as it has the biggest and most recent coverage of annotated texts with timestamp information for hate speech research (i.e., 2011-2022 reddit posts). we evaluate models with time-sensitive macro f1 defined as 1 t t @xmath0, where f 1 t is the macro-averaged f1 specific to year t. this allows to measure how well language models adapt to evolving contexts of hate speech due to yearly time-sensitive shifts. ideally, we want language models to exhibit high and stable time-sensitive f1 scores over time. we limit the analysis to 2017-2022 as there were not enough data before 2017. experiment 2: vocabulary expansion. we examine language expansion, focusing on the emergence of neologisms, i.e., newly coined terms that have entered our vocabulary. to measure model robustness to this type of language evolution, we extend the neobench dataset @xcite to the task of hate speech detection. specifically, neobench contains pairs of sentences (s 1 , s 2 ) where s 2 differs from s 1 by the replacement of a target word with a neolo-gism while ensuring same part of speech and same meaning of s 1 . neologisms are collected between 2020-2023 and account for three types of vocabulary expansion, namely lexical, morphological, and semantic. lexical neologisms include new words, phrases, and acronyms representing new concepts-e.g., 'long covid'. morphological neologisms instead are words that derive from existing words either through blending or splintering-e.g., 'doomscrolling'. semantic neologisms refer to existing words with new meanings-e.g., 'ice' to indicate petrol-or diesel-powered vehicles. we manually annotate the reddit sample of neobench as either hateful or non-hateful, reaching a substantial average inter-annotators' agreement (cohen's @xmath1.67 @xcite ) across three annotators. we take the majority vote as groundtruth. as a result, we have 341 annotated sentences s 1 paired with their 341 counterfactuals s 2 containing the neologisms in place of the target words. we evaluate models using counterfactual invariance, i.e., a formalization of the requirement that changing irrelevant parts of the input (i.e., replacing target words with neologisms) should not change model predictions @xcite . we decompose the counterfactual invariance into label flipping (i.e., rate of how often the model flipped the label when seeing the counterfactual s 2 wrt s 1 ) and hallucination (i.e., rate of how often the model does not follow the instruction when given the counterfactual s 2 but does follow the instruction when given s 1 ). mathematically, we define label @xmath2 models. we zero-shot prompt 20 language models widely used in established hate speech and stateof-the-art research ( clearly, this short-term and long-term volatility of language models in evolving hate speech detection poses real concerns regarding the safety robustness of these models. interestingly, dynamic adversarial training does not make models more robust to time-sensitive shifts: roberta-dyna-r2/3/4 models which have been fine-tuned on more adversarial examples than roberta-dyna-r1 have lower time-sensitive macro f1 than the latter. this corroborates previous research showing that training on adversarially-collected data for qa tasks was detrimental to performance on non-adversarially collected data @xcite . for the other non-adversarially trained models instead, model size improves the overall time-sensitive macro f1 score. the time-sensitive baseline is more robust across years and labels but overall performs similarly to small llms and deepseek llm. gpt-4o reaches the highest time-sensitive performance. language models are sensitive to counterfactuals containing neologisms. high scores in static evaluations do not necessarily translate to time-sensitive evaluations. overall, there is a negative correlation between static evaluations and experiment 1, indicating that models that perform the best in static benchmarks are not the most robust to time-sensitive shifts. similarly, high scores in static evaluations do not necessarily imply high scores in experiment 2, as cor-relation is on average negative or close to zero. on the other hand, static hate speech benchmarks show a positive, non-negligible correlation among each other, with an average correlation coefficient equal to 0.36 (cf. table a11 and app. d). in other words, while performance on a static hate speech benchmark is aligned to the performance on another static benchmark, the same does not hold for time-sensitive evaluations. evolving hate speech introduces variability that static benchmarks fail to capture, making them an unreliable predictor over time. static time-sensitive experiment 1 experiment 2 hatecheck -0.2662 (-0.586, 0.126) -0.0707 (-0.438, 0.317) dynabench -0.1549 (-0.504, 0.238) -0.3053 (-0.613, 0.083) hatexplain -0.2541 (-0.578, 0.138) -0.1865 (-0.528, 0.207) implicit hate -0.2812 (-0.597, 0.110) 0.1909 (-0.203, 0.532) ## conclusions this study is the first to investigate the impact of evolving language on hate speech benchmarking. we design two time-sensitive experiments and metrics to evaluate 20 language models widely adopted in state-of-the-art research. we found that language models are not robust to evolving hate speech as they exhibit short-and long-term volatility to timesensitive shifts in experiment 1 and sensitivity to counterfactuals containing neologisms in experiment 2. interestingly, dynamic adversarial training does not help models generalise in evolving scenarios. finally, we provide empirical evidence of the misalignment between static and time-sensitive evaluations, as we found negative or close to zero correlations between the two, which opens up important concerns about the reliability of current hate speech benchmarks in the future. in light of our findings, we advocate for timesensitive linguistic benchmarks to reliably evaluate models' safety in the hate speech domain. examples might include our proposed time-sensitive metrics or more structured approaches similar to those recently developed for evolving encyclopedic knowledge (e.g., test-of-time @xcite ). future techniques could explore continual learning to enable llms to adapt to evolving hate speech, and context-aware detection to capture subtle shifts in meaning driven by cultural or political events.