Playing the Part of the Sharp Bully: Generating Adversarial Examples for Implicit Hate Speech Detection (2023)

Abstract: Research on abusive content detection on social media has primarily focused on explicit forms of hate speech (HS), that are often identifiable by recognizing hateful words and expressions. Messages containing linguistically subtle and implicit forms of hate speech still constitute an open challenge for automatic hate speech detection. In this paper, we propose a new framework for generating adversarial implicit HS short-text messages using Auto-regressive Language Models. Moreover, we propose a strategy to group the generated implicit messages in complexity levels (EASY, MEDIUM, and HARD categories) characterizing how challenging these messages are for supervised classifiers. Finally, relying on (Dinan et al., 2019; Vidgen et al., 2021), we propose a “build it, break it, fix it”, training scheme using HARD messages showing how iteratively retraining on HARD messages substantially leverages SOTA models’ performances on implicit HS benchmarks.

## introduction the spread of offensive content and hate speech (hs) is a severe and increasing problem in online social communities. while in the last years numerous studies in the natural language processing community have proposed computational methods to address the spread of malicious content, they tend to over-rely on overt and explicit forms of hs, neglecting more implicit and veiled ones (e.g., "i'm either in north florida or nigeria sometimes i can't tell the difference." from the white supremacy forum dataset (wsf) @xcite ). implicit hs contains expressions of coded or indirect language that does not immediately denote hate but still disparages a person or a group based on protected characteristics such as race, gender, cultural identity, or religion @xcite . implicitness goes beyond wordrelated meaning, implying figurative language such as irony and sarcasm, generally hiding the real sense, making it more challenging to grasp sometimes even for humans. from a computational perspective, current sota models fail to properly detect implicit and subtle hs messages, as peculiar features connected to sentiment, inference, context and irony, as well as complex syntactic structures, cannot be properly understood @xcite . in order to improve the automated detection of hs messages, a few recent studies focus on obtaining more targeted diagnostic insights for current nlp models by systematically providing means of creating hs adversarial examples @xcite @xcite and more guided training strategies aiming to identify veiled hs implications @xcite @xcite . however, most of these approaches obtain implicit adversarial instances i) scraping posts from the web, causing data disproportion and spurious hate correlations, ii) performing perturbations of input sentences neglecting text variety for training, and iii) manually creating such messages, which require high-annotation costs and experienced crowdsourcers. in this work, we propose a new framework for generating on-scale close-to-human-like adversarial implicit hs texts using the pre-trained language model (plm) gpt3 @xcite , which is known to output biased and hateful content @xcite . although such hateful messages pose real threats, we use this inadmissible behavior to improve existing hate classifiers, pushing forward the research to systematically neutralize implicit hateful messages. while the proposed approach follows the alice model @xcite , that combines demonstration-based prompting and already trained hs classifiers to generate adversarial messages, in our work we go beyond it further develop-ing a generation framework for implicit hs detection, that implements a variant of constrained beam search decoding through novel soft-constrains approaches. we rely on auto-regressive plms that play the role of a bully challenging a hs classifier on implicit messages. given an implicit hateful prompt, we encourage generations to be more implicit and adversarial by i) guiding generation with demonstration-based prompts and implicit messages, ii) soft-constraining the generation probabilities in such a way that the output text is "similar" to the demonstration examples occurring in the prompt, iii) minimizing classification scores of implicit hate detectors, iv) weighting generation if offensive or implicit words of an hs lexicon are used, and v) determining the optimal number of input sentences to generate instances that are hard to classify. additionally, we present a build it, break it, fix it approach inspired by @xcite , grouping implicit hs adversarial examples into three categories: easy, medium, and hard, according to their challenging level. then, we incrementally retrain sota models on implicit hs detection on these three groups showing how hard generated messages improve sota models' performances substantially on ishate, a collection of hs benchmarks annotated with implicit hs labels @xcite . 1 note: this paper contains examples of language which may be offensive to some readers. they do not represent the views of the authors. ## related work in the following, we first report on the most significant research work on abusive language and hate speech detection carried out in the natural language processing (nlp) community, and then on works describing the generation of adversarial examples to analyze and improve nlp model. ## proposed framework as introduced before, most of the existing methods to detect abusive language in short text messages rely on supervised approaches that strongly depend on labeled datasets for training. but as observed by @xcite , most of the available datasets mainly contain explicit forms of hs, ignoring abusive content expressed in more implicit or subtle ways. this results in the current methods' poor detection performance on the implicit hs class as the training datasets are highly imbalanced @xcite . to mitigate this issue, we propose a framework to generate on-scale close-to-human-like adversarial implicit hs texts using the pre-trained language model gpt3 @xcite . our generation framework is composed by four components (figure 1 ): i) a demonstration-based prompt, ii) a search method, iii) a goal function, and iv) a set of constraints. from a starting demonstration-based prompt, the plm completes the prompt with a possible next token at a time in such a way that the final output minimizes a goal function (i.e., indicating whether a message is challenging) and satisfies the constraints. each next token is obtained through a search method that determines which of those possible next tokens are the most suitable to produce a challenging message. except for the prompt, the other components depend on the classifier we aim to attack, to target its weaknesses. in the following, we describe each component of our framework, as illustrated in figure 1 . ## generation of implicit hs adversarial messages in this section, we report and analyze the experimental results to demonstrate the effectiveness of the proposed framework. first, we list the targeted research questions (section 4.1), then, we describe the dataset we use in our experiments (section 4.2), the experimental setting (section 4.3), and finally we discuss the obtained results (section 4.4). ## improving hs classifiers on implicit messages in this section, we report on how our generation framework can be used to improve implicit hate speech detection on the ishate benchmark, relying on a variant of the build it, break it, fix it approach @xcite . ## conclusions in this paper, we have presented a new framework for generating on-scale adversarial implicit texts for hs detection using auto-regressive language models. the proposed framework follows @xcite and guides a gpt3 plm to output messages that fool current hs classifiers on implicit messages. following @xcite , we have further developed a variant of constrained beam search decoding, providing a guided generation strategy through i) implicit hate demonstrationbased prompts, ii) scores of implicit hate detectors, iii) generation weights through implicit and explicit words in hs lexicons, and iv) text similarity constraints that compare the used prompt and the expected output. we show how the proposed framework can produce, from a batch of generated messages, 70% of implicit hs messages, where 90% of them result to be hard adversarial cases for a competitive sota model on the ishate benchmark (i.e., hatebert). furthermore, we have proposed a build it, break it, fix it, approach that uses the adversarial examples generated by the above described framework to incrementally retrain machine learning models and improve their classification performances. we showed how adversarial generation leverages the hatebert classification model on the ishate dataset by improving false negative classification. while this strategy may have potential issues, such as cyclical progress, it remains a valuable approach to improve model robustness, accelerate progress, define clear objectives @xcite @xcite , and gain a deeper understanding of models' errors as shown in our paper. as for future work, we plan to explore how to embed implicit and subtle statements properly in representation spaces @xcite , deciphering models for code language @xcite and provide bias mitigation strategies for social stereotypes @xcite .