Checking H ate C heck: a cross-functional analysis of behaviour-aware learning for hate speech detection (2022)

Abstract: Behavioural testing—verifying system capabilities by validating human-designed input-output pairs—is an alternative evaluation method of natural language processing systems proposed to address the shortcomings of the standard approach: computing metrics on held-out data. While behavioural tests capture human prior knowledge and insights, there has been little exploration on how to leverage them for model training and development. With this in mind, we explore behaviour-aware learning by examining several fine-tuning schemes using HateCheck, a suite of functional tests for hate speech detection systems. To address potential pitfalls of training on data originally intended for evaluation, we train and evaluate models on different configurations of HateCheck by holding out categories of test cases, which enables us to estimate performance on potentially overlooked system properties. The fine-tuning procedure led to improvements in the classification accuracy of held-out functionalities and identity groups, suggesting that models can potentially generalise to overlooked functionalities. However, performance on held-out functionality classes and i.i.d. hate speech detection data decreased, which indicates that generalisation occurs mostly across functionalities from the same class and that the procedure led to overfitting to the HateCheck data distribution.

## introduction the standard method for evaluating natural language processing (nlp) systems-computing metrics on held-out data-may be a good indicator of model correctness, but tends to overestimate performance in the wild @xcite , does not indicate possible sources of models failure @xcite and overlooks potential dataset biases @xcite @xcite . behavioural testing of nlp models @xcite has been proposed as an additional evaluation methodology, where system functionalities are validated by checking specific input-output behaviour of the system. this is done through challenge sets: expert-crafted inputoutput pairs that capture human prior knowledge and intuition about how an agent should perform the task (linzen, 2020) and enable systematic verification of system capabilities @xcite . for the purposes of this paper, we consider a behavioural test suite to be a collection of test cases, input-output pairs that describe an expected behaviour. each case assesses a specific functionality, which are grouped into functionality classes. for example, test cases in hatecheck @xcite , a test suite for hate speech detection, include (" @xcite belong in a zoo.", hateful), ("no @xcite deserves to die.", non-hateful) and ("i had this queer feeling we were being watched", non-hateful). these cases assess the functionalities: implicit derogation of a protected group or its members, non-hate expressed using negated hateful statement and non-hateful homonyms of slurs foot_0 . these functionalities are grouped into the derogation, slur usage and negation classes. a test suite may also contain aspects, relevant properties of test cases that are orthogonal to the functionalities. an example of aspect in hatecheck is the set of possible targeted identity groups. while behavioural testing has been designed as a diagnostics tool, whether and how to leverage it for model training and development has seen little exploration, even though the human insights encoded in the test cases could potentially lead to more robust and trustworthy models. however, naively using behavioural testing for both training and evaluation is a risky affair-giving models access to the test cases could clue them into spurious correlations and lead to overestimation of model performance @xcite . we view these risks as strong motivation to explore such settings, in order to gain insights into the vulnerability of behavioural tests to gaming and over-optimisation. we explore three questions regarding behaviouraware learning: q1: do models generalise across test cases from the same functionality? this is a sanity check: test cases from the same functionality share similar patterns-sometimes generated by the same template-so we expect that behaviour-aware learning leads to better performance on test cases from functionalities seen during training. q2: do models generalise from covered functionalities to held-out ones? by examining how behaviour-aware learning affects performance on held-out functionalities, we can estimate the robustness of the approach to potentially overlooked phenomena. equivalently, performance decrease is an indicator of overfitting to functionalities covered during training. q3: do models generalise from test cases to the target task? improvements in the target task performance, as measured by independent and identically distributed (i.i.d.) data, would indicate that a model was able to extract the knowledge encoded in the behavioural tests. conversely, a decrease in target task performance would signal overfitting to the behavioural test distribution. in this paper, we explore behaviour-aware learning by fine-tuning pre-trained bert @xcite models on hatecheck foot_1 . we experiment with several splitting methods and evaluate on different sets of held-out data: test cases for covered functionalities (q1), test cases for held-out functionalities (q2), and hate speech detection i.i.d. data (q3). in addition to hatecheck's functionalities, we consider performance on held-out functionality classes and identity groups. by investigating our research questions, we address potential pitfalls and identify promising approaches for behaviour-aware learning foot_2 . ## related work traditional nlp benchmarks are created from text corpora assembled to reflect the naturally-occurring data distribution, which may fail to sufficiently capture important phenomena. challenge sets were created as an additional evaluation framework, characterised by greater control over data that enables testing for specific linguistic phenomena @xcite . ribeiro et al. (2020) proposed checklist as a task-agnostic evaluation methodology with different test types that range from template-generated challenge sets to perturbation-based tests that enable checking behaviour on unlabelled texts. inspired by check-list, röttger et al. (2021) created hatecheck, a test suite for hate speech detection models composed of hand-crafted and template-generated test cases whose design was motivated by interviews with civil society stakeholders. using challenge data and behavioural tests to explicitly drive model development and training has largely gone unexplored. @xcite created hans, a challenge set for natural language inference (nli) designed to contradict classification heuristics that exploit spurious correlations in nli datasets. they used the hans templates to augment nli training data, which helped prevent models from adopting such heuristics, though the improvement on held-out cases was inconsistent. @xcite proposed inoculation by fine-tuning, where a model originally trained on a non-challenge dataset is fine-tuned on a few examples from a challenge set and then evaluated on both datasets. they do not assess generalisation from covered to held-out functionalities, as they use samples from the same functionality for training and testing. to the best of our knowledge, we are the first to examine cross-functional behaviour-aware learning by fine-tuning models on different configurations of test suite and task data and evaluating performance across multiple generalisation axes. test suites have limited coverage: the included functionalities, functionality classes and aspects are only subsets of the phenomena of interest. for example, hatecheck covers seven protected groups, which are particular samples of the full set of communities targeted by hate speech. therefore, naive evaluation of models fine-tuned using test suite data can lead to overestimating their performance: models can overfit to the covered phenomena and pass the tests, but fail cases from uncovered phenomena (e.g., hate targeted at an uncovered identity group). since we cannot directly evaluate performance on uncovered cases, we use performance on held-out sets of functionality, functionality classes and aspects as a proxy for generalisation across those three axes, as described in sections 3.2 and 3.4. ## results and discussion covered functionalities performance (q1) figure 2 exhibits performance on hatecheck all split. all models fine-tuned on hatecheck greatly outperformed models fine-tuned only on task data. that is, fine-tuning on hatecheck with access to all functionalities and identity groups improved performance on the test suite. prior finetuning on task data did not make a relevant dif-accuracy 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% davidson-funcout founta-funcout covered held-out (a) funcout. accuracy 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% davidson-identout founta-identout covered held-out (b) identout. accuracy 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% davidson-classout founta-classout covered held-out (c) classout. figure 3: performance comparison between covered and held-out phenomena on funcout, identout and heldout test sets: accuracy for covered phenomena is consistently better, though discrepancy magnitude varies across phenomena of interest. accuracy 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% davidson davidson-funcout founta founta-funcout (a) funcout. accuracy 50.00% 60.00% 70.00% 80.00% 90.00% 100.00% davidson davidson-identout founta founta-identout (b) identout. accuracy 50.00% 55.00% 60.00% 65.00% 70.00% 75.00% davidson davidson-classout founta founta-classout (c) classout. figure 4: held-out performance change after fine-tuning on hatecheck: accuracy improves for held-out functionalities and identity groups, but decreases for held-out functionality classes. macro f1 score 40.00% 50.00% 60.00% 70.00% 80.00% davidson davidson-all founta founta-all all davidson et al. founta et al. figure 5 : performance on the task test sets: macro f 1 score decreases after fine-tuning on hatecheck. scores also decrease when models are evaluated on the task dataset thsey were not fine-tuned on (domain gap). figure 4 shows the impact of fine-tuning on hatecheck by contrasting performance before and after the procedure. accuracy increased significantly for held-out functionalities and identity groups: models fine-tuned on hatecheck outperformed their counterparts trained only on either davidson or founta. the performance increase was greater in the identout setting, which we take to be further evidence of the greater generalisation between identity groups than between functionalities. while the fine-tuning procedure contributed positively to performance in the funcout setting, the same did not happen in the classout scenario. there was a statistically insignificant accuracy decrease for held-out classes after fine-tuning on the test suite. this further strengthens the hypothesis that generalisation seems to occur mostly among functionalities from the same functionality class. task data performance (q3) figure 5 compares model performance on the task test sets 8 . macro f 1 scores decreases significantly after finetuning on hatecheck. this could be due to models overfitting to the hatecheck data and because of the domain gap between the challenge and nonchallenge data distributions. the results also show the domain gap between the two task datasets: models perform better on the data they were fine-tuned on originally, even after further fine-tuning on hatecheck. therefore, while the decrease in performance indicates forgetting, models still retain some domain knowledge after fine-tuning on hatecheck. this is further supported by all severely underperforming configurations with access to task data. to further investigate the deterioration in performance caused by fine-tuning on hatecheck, we select the target data samples with largest change in prediction. that is, given a sample s and the gold label probabilities p before (s) and p after (s) predicted before and after fine-tuning on hatecheck, we calculate for each sample the change in prediction: ∆ p (s) = p after (s)p before (s). then, for each hate speech detection dataset, we 8 our results are similar to the ones reported by @xcite : we got micro/macro f1 scores of 90.56%/69.70% and 93.19%/71.73% for @xcite reported 91.5%/70.8% and 92.9%/70.3% respectively. select the samples with: ## conclusion we have presented a cross-functional analysis of behaviour-aware learning for hate speech detection. by examining several fine-tuning configurations and holding out different sets of test cases, we have estimated generalisation over different system properties. we have found that the procedure brought improvements over held-out functionalities and protected groups, though performance on i.i.d. task data and held-out functionality classes decreased. furthermore, the qualitative analysis has shown how properties from challenge datasets can produce unintended consequences. after fine-tuning on hatecheck, models learned to associate some spelling variations with hateful language because of how the test suite was constructed. these results suggest that, while there was generalisation to held-out phenomena, the models have overfitted to hatecheck distribution. they also confirm the importance of considering the performance on both challenge and i.i.d. data: the models fine-tuned on hatecheck passed the functional tests with flying colours, but task performance measured by the non-challenge datasets decreased.