H ate C heck: Functional Tests for Hate Speech Detection Models (2021)

Abstract: Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HateCheck, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HateCheck’s utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.

## introduction hate speech detection models play an important role in online content moderation and enable scientific analyses of online hate more generally. this has motivated much research in nlp and the social sciences. however, even state-of-the-art models exhibit substantial weaknesses (see @xcite @xcite . so far, hate speech detection models have primarily been evaluated by measuring held-out performance on a small set of widely-used hate speech datasets (particularly @xcite @xcite , but recent work has highlighted the limitations of this evaluation paradigm. aggregate performance metrics offer limited insight into specific model weak-nesses @xcite . further, if there are systematic gaps and biases in training data, models may perform deceptively well on corresponding held-out test sets by learning simple decision rules rather than encoding a more generalisable understanding of the task (e.g. @xcite @xcite . the latter issue is particularly relevant to hate speech detection since current hate speech datasets vary in data source, sampling strategy and annotation process @xcite , and are known to exhibit annotator biases @xcite @xcite as well as topic and author biases @xcite . correspondingly, models trained on such datasets have been shown to be overly sensitive to lexical features such as group identifiers @xcite @xcite , and to generalise poorly to other datasets @xcite . therefore, held-out performance on current hate speech datasets is an incomplete and potentially misleading measure of model quality. to enable more targeted diagnostic insights, we introduce hatecheck, a suite of functional tests for hate speech detection models. functional testing, also known as black-box testing, is a testing framework from software engineering that assesses different functionalities of a given model by validating its output on sets of targeted test cases @xcite . @xcite show how such a framework can be used for structured model evaluation across diverse nlp tasks. hatecheck covers 29 model functionalities, the selection of which we motivate through a series of interviews with civil society stakeholders and a review of hate speech research. each functionality is tested by a separate functional test. we create 18 functional tests corresponding to distinct expressions of hate. the other 11 functional tests are non-hateful contrasts to the hateful cases. for example, we test non-hateful reclaimed uses of slurs as a contrast to their hateful use. such tests are particularly challenging to models relying on overly simplistic decision rules and thus enable more accurate evaluation of true model functionalities @xcite . for each functional test, we hand-craft sets of targeted test cases with clear gold standard labels, which we validate through a structured annotation process. 1 hatecheck is broadly applicable across english-language hate speech detection models. we demonstrate its utility as a diagnostic tool by evaluating two bert models @xcite , which have achieved near state-of-the-art performance on hate speech datasets @xcite , as well as two commercial models -google jigsaw's perspective and two hat's siftninja. 2 when tested with hatecheck, all models appear overly sensitive to specific keywords such as slurs. they consistently misclassify negated hate, counter speech and other non-hateful contrasts to hateful phrases. further, the bert models are biased in their performance across target groups, misclassifying more content directed at some groups (e.g. women) than at others. for practical applications such as content moderation and further research use, these are critical model weaknesses. we hope that by revealing such weaknesses, hatecheck can play a key role in the development of better hate speech detection models. definition of hate speech we draw on previous definitions of hate speech @xcite as well as recent typologies of abusive content @xcite to define hate speech as abuse that is targeted at a protected group or at its members for being a part of that group. we define protected groups based on age, disability, gender identity, familial status, pregnancy, race, national or ethnic origins, religion, sex or sexual orientation, which broadly reflects international legal consensus (particularly the uk's 2010 equality act, the us 1964 civil rights act and the eu's charter of fundamental rights). based on these definitions, we approach hate speech detection as the binary classification of content as either hateful or content warning this article contains examples of hateful and abusive language. all examples are taken from hatecheck to illustrate its composition. examples are quoted verbatim, except for hateful slurs and profanity, for which the first vowel is replaced with an asterisk. ## related work targeted diagnostic datasets like the sets of test cases in hatecheck have been used for model evaluation across a wide range of nlp tasks, such as natural language inference @xcite , machine translation @xcite and language modelling @xcite . for hate speech detection, however, they have seen very limited use. @xcite compile three datasets for evaluating model performance on what they call complex offensive language, specifically the use of reclaimed slurs, adjective nominalisation and linguistic distancing. they select test cases from other datasets sampled from social media, which introduces substantial disagreement between annotators on labels in their data. @xcite use templates to generate synthetic sets of toxic and non-toxic cases, which resembles our method for test case creation. they focus primarily on evaluating biases around the use of group identifiers and do not validate the labels in their dataset. compared to both approaches, hatecheck covers a much larger range of model functionalities, and all test cases, which we generated specifically to fit a given functionality, have clear gold standard labels, which are validated by near-perfect agreement between annotators. in its use of contrastive cases for model eval-uation, hatecheck builds on a long history of minimally-contrastive pairs in nlp (e.g. @xcite @xcite . most relevantly, @xcite and @xcite propose augmenting nlp datasets with contrastive cases for training more generalisable models and enabling more meaningful evaluation. we built on their approaches to generate non-hateful contrast cases in our test suite, which is the first application of this kind for hate speech detection. in terms of its structure, hatecheck is most directly influenced by the checklist framework proposed by @xcite . however, while they focus on demonstrating its general applicability across nlp tasks, we put more emphasis on motivating the selection of functional tests as well as constructing and validating targeted test cases specifically for the task of hate speech detection. ## conclusion in this article, we introduced hatecheck, a suite of functional tests for hate speech detection models. we motivated the selection of functional tests through interviews with civil society stakeholders and a review of previous hate speech research, which grounds our approach in both practical and academic applications of hate speech detection models. we designed the functional tests to offer contrasts between hateful and non-hateful content that are challenging to detection models, which enables more accurate evaluation of their true functionalities. for each functional test, we crafted sets of targeted test cases with clear gold standard labels, which we validated through a structured annotation process. we demonstrated the utility of hatecheck as a diagnostic tool by testing near-state-of-the-art transformer models as well as two commercial models for hate speech detection. hatecheck showed critical weaknesses for all models. specifically, models appeared overly sensitive to particular keywords and phrases, as evidenced by poor performance on tests for reclaimed slurs, counter speech and negated hate. the transformer models also exhibited strong biases in target coverage. online hate is a deeply harmful phenomenon, and detection models are integral to tackling it. typically, models have been evaluated on held-out test data, which has made it difficult to assess their generalisability and identify specific weaknesses. we hope that hatecheck's targeted diagnostic insights help address this issue by contributing to our understanding of models' limitations, thus aiding the development of better models in the future.