Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study (2023)

Abstract: With the ever-growing presence of social media platforms comes the increased spread of harmful content and the need for robust hate speech detection systems. Such systems easily overfit to specific targets and keywords, and evaluating them without considering distribution shifts that might occur between train and test data overestimates their benefit. We challenge hate speech models via new train-test splits of existing datasets that rely on the clustering of models’ hidden representations. We present two split variants (Subset-Sum-Split and Closest-Split) that, when applied to two datasets using four pretrained models, reveal how models catastrophically fail on blind spots in the latent space. This result generalises when developing a split with one model and evaluating it on another. Our analysis suggests that there is no clear surface-level property of the data split that correlates with the decreased performance, which underscores that task difficulty is not always humanly interpretable. We recommend incorporating latent feature-based splits in model development and release two splits via the GenBench benchmark.

## introduction developing generalisable hate speech detection systems is of utmost importance due to the environment in which they are deployed. social media usage is rapidly increasing, and the detection of harmful content is challenged by non-standard language use, implicitly expressed hatred, a lack of consensus on what constitutes hateful content, and the lack of high-quality training data @xcite . when developing hate speech detection models in the lab, it is, therefore, vital to simulate evaluation scenarios requiring models to generalise outside the training context. 'in the wild', nlp models may encounter text from different periods ( @xcite , authors @xcite or dialects @xcite , including unseen words @xcite and words whose spelling changed or was obfuscated @xcite . performing successfully on this data despite such distributional changes is called out-ofdistribution (o.o.d.) generalisation. how can the ability to generalise best be measured? despite recent work illustrating that i.i.d. testing does not adequately reflect models' generalisability (e.g. @xcite , evaluation using randomly sampled test sets is still the status quo @xcite @xcite . potentially, this is because obtaining and annotating new data is expensive, and it is hard to define what o.o.d. data is @xcite . for humans, properties like input length @xcite or spelling mistakes @xcite might determine difficulty. but this need not be the same for models. evaluating models using a notion of modeldependent difficulty is gaining some traction (e.g. @xcite but still remains largely unexplored. contributing to that line of work, we propose a method that reuses existing datasets but splits them in a new way by relying on models' latent features. we cluster hidden representations using k-means and distribute clusters over the train and test set to create a data split. an illustrative example of such a split is shown in fig. 1 . we present two variants (subset-sum-split and closest-split). while this method is in principle applicable to any classification problem, we experiment with four language models and two hate speech datasets (that include reddit, twitter and gab data). the results suggest that these splits approximate worst-case performance. models fail catastrophically on the new test sets, while their performance on independent test data is on par with other systems trained on i.i.d. training sets. the difficulty is relatively stable across different models. we analyse the data splits through correlation analyses, and do not find one clear surface-level property of the data split to be predictive of split difficulty. this underscores that model-based difficulty can be quite elusive. we release two of our data splits for inclusion in the genbench benchmark. the remainder of this work is structured as follows: section 2 elaborates on related work, followed by the introduction of the hate speech datasets (section 3) and the proposed splitting method (section 4). section 5 presents model evaluation results, section 6 analyses the splits in detail, and we conclude in section 7. the genbench eval card can be found in appendix a. ## related work this section discusses related work on o.o.d. generalisation evaluation (section 2.1), followed by a discussion on why generalisation is a persisting challenge in hate speech detection (section 2.2). ## data we develop and evaluate our splitting method using the following two hate speech datasets. ## methodology our proposed splitting strategy, for which we introduce two variants, is detailed in section 4.1. we evaluate our splits through comparisons to a random splitting baseline and on external test sets. we discuss the corresponding experimental setups in section 4.2. ## results we now turn to evaluating models' performance on our newly proposed splits. ## analysis the performance of models deteriorates heavily when using the proposed splits. this section analyses the generated splits; first examining the surfacelevel properties of the resulting train and test sets, and then taking a closer look at two specific splits by visualising the datapoints in the train and test sets. additionally, an analysis of the topics in the train and test sets can be found in appendix e.2. ## conclusion hate speech detection systems are prone to overfitting to specific targets of hate speech and specific keywords in the input, complicating the detection of more implicit hatred and harming the generalisability to unseen demographics. yet, in addition to those known and interpretable vulnerabilities, systems may have less obvious weaknesses. the data splitting method we developed aims to highlight those. our splitting method is based on the clustering of internal representations of finetuned models, thus making the splits task-and dataset-specific. we proposed two variants (subset-sum-split and closest-split) that differ in how they assign clusters to the train and test set. the latter variant, in particular, led to consistent catastrophic drops in test set performance, when compared to a random split. moreover, while each split was developed using the hidden representations from a specific model, we identified that this result generalises when developing the split using one model, and evaluating it using another. the analyses of the resulting data splits showed that the properties of the train and test sets differ from dataset to dataset. since no property clearly correlates with decreased model performance for both datasets, closest-split cannot be easily replicated based on data splits' surface-level properties, and using latent representations is crucial to reveal the weaknesses we observed in the models. we encourage future work to consider evaluations using the closest-splits we release for hatexplain, in order to develop more robust systems, but also emphasise that even though our results were specific to hate speech detection, the methodology can be more widely applied. to challenge models beyond i.i.d. evaluation, we do not need costly data annotations. instead, we can start by relying on systems' latent features to simulate train-test distribution shifts.