H atemoji: A Test Suite and Adversarially-Generated Dataset for Benchmarking and Detecting Emoji-Based Hate (2022)

Abstract: Detecting online hate is a complex task, and low-performing models have harmful consequences when used for sensitive applications such as content moderation. Emoji-based hate is an emerging challenge for automated detection. We present HatemojiCheck, a test suite of 3,930 short-form statements that allows us to evaluate performance on hateful language expressed with emoji. Using the test suite, we expose weaknesses in existing hate detection models. To address these weaknesses, we create the HatemojiBuild dataset using a human-and-model-in-the-loop approach. Models built with these 5,912 adversarial examples perform substantially better at detecting emoji-based hate, while retaining strong performance on text-only hate. Both HatemojiCheck and HatemojiBuild are made publicly available.

## introduction online hate harms its targets, disrupts online communities, pollutes civic discourse and reinforces social power imbalances @xcite . the sheer scale of hateful content online has led to widespread use of automated detection systems to find, monitor and stop it @xcite . however, hateful content is complex and diverse, which makes it challenging to detection systems. one particular challenge is the use of emoji for expressing hate. emoji are pictorial representations that can be embedded in text, allowing complex emotions, actions and intentions to be displayed concisely @xcite . over 95% of internet users have used an emoji and 10 million are sent every day @xcite . following england's defeat in the euro 2020 football final, there was widespread racist use of emoji such as , and @xcite . this paper focuses on emoji-based hate, answering two research questions: rq1 what are the weaknesses of current detection systems for hate expressed with emoji? rq2 to what extent can human-and-model-in-theloop training improve the performance of detection systems for emoji-based hate? to answer rq1, we present hatemojicheck, a suite of functional tests for emoji-based hate. we provide 3,930 test cases for seven functionalities, covering six identities. 2,126 original test cases are matched with three types of challenging perturbations to enable accurate evaluation of model decision boundaries @xcite . we use hatemojicheck to assess google jigsaw's perspective api as well as models trained on academic datasets, exposing critical model weaknesses. to answer rq2 and address the model weaknesses identified by hatemojicheck, we implement a human-and-model-in-the-loop dynamic training scheme. we build on the work of @xcite , who used this approach for textual hate. our work begins where their study ends. we conduct three rounds of adversarial data generation focused explicitly on emoji-based hate, tasking annotators to generate sentences that the model-inthe-loop misclassifies. this process yields a dataset of 5,912 entries, half of which are challenging contrasts. we call this dataset hatemojibuild. the dataset is evenly split between hate and non-hate, and each hateful entry has labels for the type and target. between each round, the model-in-the-loop is re-trained so that annotators are trying to trick a progressively stronger and more 'emoji-aware' model. relative to existing commercial and academic models, our models improve performance on the detection of emoji-based hate, without sacrificing performance on text-only hate. we make several contributions: (1) we construct hatemojicheck, which tests key types of emojibased hate as separate functionalities, (2) we evaluate the performance of existing academic and commercial models at detecting emoji-based hate, (3) we present hatemojibuild, a labeled emojibased hate speech dataset that is adversariallygenerated for model training and (4) we train models that can accurately detect emoji-based hate. these data-centric contributions demonstrate the benefits of systematic and granular evaluation, and the need to diversify how hate detection systems are trained. we make both hatemojicheck and hatemojibuild publicly available. definition of hate we use the united nations definition of hate speech: "any kind of communication in speech, writing or behavior, that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are, in other words, based on their religion, ethnicity, nationality, race, color, descent, gender or other identity factor" (united nations, 2019, p.2). 2 ## hatemojicheck: functional tests for emoji-based hate the concept of functional testing, an expected input-output behavior @xcite , has been adapted from software engineering for nlp tasks @xcite , including hate speech detection @xcite . these diagnostic tests identify model vulnerabilities to intentionally simple and non-ambiguous constructions of hate so comprise a minimal performance standard. ## building better models with hatemojibuild as reported in §4, we find existing models perform poorly on emoji-based hate as measured with hatemojicheck. we address those failings by 5 see annotator demographics in appendix b. in each round we tasked a team of 10 trained annotators with entering content that the modelin-the-loop would misclassify. 7 we refer to this model as the target model. annotators were instructed to generate linguistically diverse entries while ensuring each entry was (1) realistic, (2) clearly hateful or non-hateful and ( 3 ) contained at least one emoji. each entry was first given a binary label of hateful or non-hateful, and hateful content was assigned secondary labels for the type and target of hate @xcite . each entry was validated by two additional annotators, and an expert resolved disagreements. after validation, annotators created a perturbation for each entry that flipped the label. to maximize similarity between originals and perturbations, annotators could either make an emoji substitution while fixing the text or fix the emoji and minimally change the surrounding text. including a hateful and non-hateful version of each sentence in the dataset prevents overfitting to certain emoji or identity references. each perturbation received two additional label annotations, and disagreements were resolved by the expert. this weekly cadence of annotator tasks was repeated in three consecutive weeks. the dataset composition and inter-annotator agreement is described in appendix e. ## discussion hatemojicheck reveals critical model weaknesses in detecting emoji-based hate. existing commercial and academic models perform poorly at identifying hate where the identity term has been replaced with an emoji representation (f2 and f4), even though they perform well at identifying the equivalent textual statements. these models have better performance on verb swap (f1) but then struggle with the polarity perturbations (f1.2). this suggests that the models are overfitting to the identity term and ignore the sentiment from the emoji, leading to false positive predictions. our newly trained models substantially improve performance on original hateful statements from f2, f4 and f5, indicating they have a better semantic grasp of emoji substitutions and appends. they also make large performance gains on the polarity perturbations in f1.2, f3.2, f5.2 and f6.2, suggesting they better incorporate information on how different emoji condition the likelihood of hatefulness. despite improving on existing models, our models still perform relatively more poorly on f4, as well as the f1.2, f5.3 and f7.1 perturbations. these weaknesses could potentially be addressed in future work through additional data generation. training on just one round of adversarial data yields the biggest improvement on hatemoji-check. thereafter, performance plateaus. this aligns with the sharp increase and then fall in mer 9 selection rate is p[h(x)|@xmath0, i.e., the proportion of the population in subgroup a who have 1 (hate) as the given class label. the 'true' ratio of class labels is shown in fig. 2 across emoji-based rounds of data generation (in fig. 1 ). on the adversarial test sets, r8-t only marginally outperforms r6-t and r7-t on all r1-7 test sets. it slightly underperforms on the r5-7 test sets from hatemojibuild. this suggests that while training on a relatively small number of entries can substantially improve performance, the gains quickly saturate. future work could investigate performance differences in more detail using the labels for type and target of hate that we provide for all three rounds of data. due to practical constraints, hatemojicheck has several limitations, which could be addressed in future work. first, it includes relatively simplistic statements, and so offers negative predictive power: high performance only indicates the absence of specific weaknesses @xcite . second, it is inspired by realworld hate but synthetically generated. future work could evaluate against more complex and diverse forms of emoji-based hate. third, it is limited in scope. it only considers short english-language statements with one binary label. only six identities are included, none of which are intersectional, and the set of emoji covers a fraction of the full unicode standard. in-scope identities and emoji usage are culturally-dependent @xcite so future work could assess the generalizability of hatemojicheck to other cultures and languages. these limitations do not diminish hatemojicheck's utility as a tool for effectively identifying model weaknesses to emoji-based hate. by publicly releasing the test suite, practitioners and academics can scrutinize their models prior to deployment or publication. adversarial data generation is a powerful technique for creating diverse, complex and informative datasets. however, it also introduces challenges. first, the entries are 'synthetic' rather than sampled from 'real-world' examples. substantial training and time are required to ensure that annotators understand real online hate and can imitate it. second, annotators can exhaust their creativity and start producing unrealistic, simplistic or non-adversarial examples. third, because of the need for training, supervision and support, only a small pool of annotators is feasible, which can introduce additional idiosyncratic biases. given these issues, quality control is a key consideration throughout the data generation process. encouragingly, we find carefully-curated and adversarially-generated training datasets can significantly improve performance on emoji-based hate as a particular type of challenging content, and that this approach is effective with relatively few training examples. thus, substantial model improvements can be realized with minimal financial and computational cost. ## related work emoji-based hate has received limited attention in prior work on hate and abusive language detection. studies that attend to emoji often do so as a potential input feature to aid classifier performance. for example, @xcite improve offensive language classification with an 'emotion-aware' mechanism built on emoji embeddings. @xcite find that adding emoji features to continuous bag of words and word unigram models marginally improves performance for abusive language detection on indonesian twitter. @xcite identifies examples of subtle and non-direct hate speech in german-danish twitter conveyed through 'winking' or 'skeptical' emoji that flag irony or non-literal meaning in their accompanying text. @xcite train an emoji-based masked language model (mlm) for zero-shot abuse detection. they show this method improves performance on classifying abuse in german, italian and spanish tweets compared to an mlm which does not attend to emoji. @xcite use abusive emoji as a proxy for learning a lexicon of abusive words. their findings indicate that emoji can disambiguate abusive and profane usages of words such as f*ck and b*tch. by contrast, our work focuses on emoji-based hate as a challenge for hate detection models. with hatemojicheck, we enable a systematic evaluation of how well models handle different types of emoji-based hate. rather than adjusting the model architecture, we account for emoji-based hate in our iterative data generation process and show that models trained on such data perform better on emoji-based hate, while retaining strong performance on text-only hate. as a suite of functional tests for evaluation, hatemojicheck directly builds on previous work by @xcite and @xcite . @xcite introduced functional tests as a framework for nlp model evaluation with checklist, showing that their approach can identify granular model strengths and weaknesses that are obscured by high-level metrics like accuracy and f1-score. @xcite adapted this framework to hate detection with hatecheck, which covers 29 model functionalities motivated by interviews with civil society stakeholders and a review of previous hate speech literature. like hatecheck, we pair hateful test cases with contrasting perturbations that are particularly challenging to models relying on overly simplistic decision rules and thus reveal granular decision boundaries. hatecheck did not consider emoji-based hate, which is the main focus of our work. our approach to training better models for emojibased hate builds directly on work by @xcite , who apply iterative human-and-model-inthe-loop training to hate detection models. like us, they used the dynabench interface @xcite to implement their training system, which has also been used to improve model performance for other tasks such as reading comprehension @xcite and sentiment analysis @xcite . earlier work by @xcite introduced a similar 'build it, break it, fix it' system of repeated interactions between a hate classifier and crowdworkers to develop safe-by-design chatbots. unlike previous work, we focus data generation on a particular type of hateful content, emoji-based hate, and show that the training scheme can address specific model weaknesses on such content without sacrificing performance on text-only hate. ## conclusion online hate is a pervasive, harmful phenomenon, and hate detection models are a crucial tool for tackling it at scale. we showed that emoji pose a particular challenge for such models, and presented hatemojicheck, a first-of-its-kind evaluation suite for emoji-based hate. it covers seven functionalities with 3,930 test cases. using this test suite, we exposed clear weaknesses in the performance of commercial and academic models on emoji-based hate. to address these weaknesses, we created the hatemojibuild dataset using an innovative human-and-model-in-the-loop approach. we showed that models trained on this adversarial data are substantially better at detecting emojibased hate, while retaining strong performance on text-only hate. our approach of first identifying granular model weaknesses, and then generating an adversarial dataset to address them presents a promising direction for building models to detect other diverse and emerging forms of online harm.