Analyzing the Real Vulnerability of Hate Speech Detection Systems against Targeted Intentional Noise (2022)

Abstract: Hate speech detection systems have been shown to be vulnerable against obfuscation attacks, where a potential hater tries to circumvent detection by deliberately introducing noise in their posts. In previous work, noise is often introduced for all words (which is likely overestimating the impact) or single untargeted words (likely underestimating the vulnerability). We perform a user study asking people to select words they would obfuscate in a post. Using this realistic setting, we find that the real vulnerability of hate speech detection systems against deliberately introduced noise is almost as high as when using a whitebox attack and much more severe than when using a non-targeted dictionary. Our results are based on 4 different datasets, 12 different obfuscation strategies, and hate speech detection systems using different paradigms.

## introduction computer-mediated communication is plagued by toxic and hateful behavior that can cause serious harm @xcite . consequently, automatically detecting such behavior has become a major research area @xcite @xcite @xcite . whenever there is an automatic system in place to filter out hateful messages, people will try to circumvent it by obfuscating their message. however, there are limits, as the communicative intent has to stay intact. if the intended audience cannot relatively easily understand a message, obfuscation has gone too far. thus, people will usually only obfuscate a few terms they think will be problematic or could be responsible for filtering out a message (see in this paper, we analyze the real vulnerability of state-of-the-art hate speech detection systems against targeted obfuscation. we keep a post recognizable by obfuscating at most one target token per test example. we perform an annotation study to analyze the target selection strategies in real-world setting. we use 12 types of plausible obfuscation strategies and apply them to the targeted tokens. in order to generalize our analysis, we repeat our experiments on multiple hate speech datasets. for future benchmarking, we open-source our code base, trained models, and obfuscated test samples foot_0 . ## ethical considerations in our research, we are discussing and explaining obfuscation strategies. people could use those strategies to avoid detection and eventually cause even more harm @xcite . we consider this risk to be small, as people are creative and would have (and almost certainly already have) come up with all of the described strategies. we decided to release all our code (even the parts obfuscating single words), as we see a clear benefit in researchers reproducing our results and facilitate their own research. this outweighs the risk caused by people using that code to automatically obfuscate their messages. law enforcement agencies might use our research to build more robust detection systems. this can be positive, as marginalized groups might not have to deal with being targets of hate speech all the time and might dare again to use their free speech rights without being threatened into silence. however, social media platforms may use deobfuscation to ban words they consider offensive. this might have unintended consequences, e.g. a person called richard gaywood was not able to use his name as it include the word gay and consider to violate community standards @xcite . an even more serious harm are overreaching governments censoring non-hateful expression of opinions. over the last years, the web-based censorship as well as surveillance has significantly increased in some parts of the world @xcite . improved detection models will even more empower such authoritarian regimes and give them the opportunity to increase the severity of surveillance towards its citizens @xcite , as they can no longer circumvent censorship through obfuscation. ## obfuscation strategies obfuscation is the deliberate act of obscuring the intended meaning of communication by adding noise to the message. this can happen in many ways. for example, @xcite show that appending a positive word like love can already fool a classifier. @xcite demonstrate the vulnerability of hate detection models by simple replacements of certain tokens with emojis. another obfuscation strategy is to paraphrase the whole message or use a metaphorical expression to indirectly express the same point. for example, instead of all those researchers are stupid as hell one could write those einsteins are not the sharpest knife in the drawer. however, this also changes the meaning and presupposes that the intended audience is aware of the possible replacement and able to make the connection. such a replacement might also change the perceived severity of a toxic comment, e.g. it is possible that people would consider the sentence with einstein as more hateful, as it adds a potential connotation of jewish researcher. in this paper, we only focus on producing simple obfuscation strategies @xcite ) which are generally observed to be implemented in realistic settings. camel case while camel casing is usually used to improve the readability of a text (e.g. naming conventions in computer programming), it can also be used to add noise. in our camel-casing strategy, we capitalize every alternate letter (starting from the second letter). char drop while keeping first and last character untouched we either drop a randomly chosen character from the selected token or drop all vowels @xcite . we call the later obfuscation as vowel drop. we ensure preservation of token perception @xcite @xcite ). therefore, we only add the noise to tokens with 3 or more characters. char flip character shuffling within the boundaries of the word (excluding boundary characters) does not have much effect on word semantics. however, it would be quite easy to get this implemented during the message composition. to generate such noise, we only consider tokens having length more than 3. excluding first and last, we randomly select two characters and flip them. diacritics some non english languages use extra marks or glyph (such as ˆ) above or below (or sometimes next to) a letter for explicit enunciation. ## target selection we propose model independent token selection strategies that range from very broad (all tokens) to very specific (the words conveying the hateful intent). our intention is to provide every possible cases in order to analyse the model robustness in depth. all we obfuscate all tokens with more than 3 characters (obfuscation of shorter words cannot be reliably performed). this is the most aggressive obfuscation strategy that will probably make it unreadable to humans and machines alike. random any a single word (with more than 3 characters) is randomly selected from the message text. we do not obfuscate the first and last word of the text. ## experimental setup we experiment with all the obfuscation and target selection strategies outlined above. to make sure that our results are not specific to a dataset or detection method, we also use multiple dataset and methods as outlined next. ## results we analyze model vulnerability by looking at the decline in relative performance when they are tested on obfuscated posts. since we use unbalanced datasets, we estimate the performance using f1 macro score evaluate on hate-labels. target selection ## related work although most of the previous studies raise concern about model robustness against obfuscation attacks, the real vulnerability is understudied. among simple obfuscation strategies, studies @xcite @xcite introduce syntactic perturbations to validate the robustness of hate detection models. we find overlapping of some of the obfuscation strategies discussed in this paper. kirk et al. (2022) proposes test-suite contained emoji-based hateful statements and find high vulnerability of text based models. the study also proposes adversarial examples to strengthen the model robustness. complex obfuscation include viper @xcite which is a probabilistic visual perturber that keep the token recognizable. target selection has been studied in both model dependent and independent settings. model dependent targets are either selected by looking at model architecture and its parameters @xcite or solely depend upon model output @xcite @xcite . among model independent targets, studies consider all the tokens in the message @xcite @xcite . however, in real time settings, this type of target selection is not observed. in our work, we squeeze target selection to maximum of single token and perform an annotation study, that estimate real vulnerability of the models. ## conclusions previous work, simulating the obfuscation behavior of haters in a simplified way, is likely to misanalyze the real vulnerability of hate speech detection systems to obfuscation. we have shown that obfuscating all words in a post is a useful lower bound, but a very unrealistic strategy (as the communicative value of the message breaks down). note that in this work, we deliberately limited ourselves to quite simple lexical modifications. however, detection systems still show a surprising vulnerability against these simple strategies. while it might be possible (and fairly easy) to shield a system against particularly known obfuscation strategies (e.g. by detecting k-e-b-a-b or s_n_a_k_e obfuscation with a regular expression), we need to aim for systems that are also robust against unseen strategies. as the user study conducted in this paper shows, people have an intuitive understanding of which words are problematic. by obfuscating a single word, someone trying to obfuscate their message can impact the system performance on the same scale as when using a whitebox attack (that has the 'unfair' advantage of having access to the internal workings of the system). we also show that experiments relying on a fixed dictionary of problematic words for obfuscation are likely underestimating the impact of obfuscation on hate speech detection systems.