On the Robustness of Offensive Language Classifiers (2022)

Abstract: Social media platforms are deploying machine learning based offensive language classification systems to combat hateful, racist, and other forms of offensive speech at scale. However, despite their real-world deployment, we do not yet comprehensively understand the extent to which offensive language classifiers are robust against adversarial attacks. Prior work in this space is limited to studying robustness of offensive language classifiers against primitive attacks such as misspellings and extraneous spaces. To address this gap, we systematically analyze the robustness of state-of-the-art offensive language classifiers against more crafty adversarial attacks that leverage greedy- and attention-based word selection and context-aware embeddings for word replacement. Our results on multiple datasets show that these crafty adversarial attacks can degrade the accuracy of offensive language classifiers by more than 50% while also being able to preserve the readability and meaning of the modified text.

## introduction online social media platforms are dealing with an unprecedented scale of offensive (e.g., hateful, threatening, profane, racist, and xenophobic) language (twitter; @xcite . given the scale of the problem, online social media platforms now increasingly rely on machine learning based systems to proactively and automatically detect offensive language @xcite @xcite . the research community is actively working to improve the quality of offensive language classification @xcite @xcite @xcite @xcite . a variety of offensive language classifiers ranging from traditional shallow models (svm, random forest), deep learning models (cnn, lstm, gru), to transformerbased models (bert, gpt-2) have been proposed in prior literature @xcite @xcite . amongst these approaches, bert-based transformer models have achieved state-of-the-art performance while ensembles of deep learning models also generally perform well @xcite . it remains unclear whether the state-of-the-art offensive language classifiers are robust to adversarial attacks. while adversarial attacks are of broad interest in the ml/nlp community @xcite , they are of particular interest for offensive language classification because malicious users can make subtle perturbations such that the offensive text is still intelligible to humans but evades detection by machine learning classifiers. prior work on the robustness of text classification is limited to analyzing the impact on classifiers of primitive adversarial changes such as deliberate misspellings @xcite , adding extraneous spaces @xcite , or changing words with their synonyms @xcite @xcite . however, the primitive attacks can be easily defended againsta spell checker can fix misspellings and a word segmenter can correctly identify word boundaries even with extra spaces @xcite . additionally, a normal synonym substitution will not theoretically hold for offensive language as less offensive language will be substituted and thus meaning will be lost. crucially, we do not know how effective these text classifiers are against crafty adversarial attacks employing more advanced strategies for text modifications. to address this gap, we analyze the robustness of offensive language classifiers against an adversary who uses a novel word embedding to identify word replacements and a surrogate offense classifier in a black-box setting to guide modifications. this embedding is purpose-built to evade offensive language classifiers by leveraging an evasion collection that comprises of evasive offensive text gathered from online social media. using this embedding, the adversary modifies the offensive text while also being able to preserve text readability and semantics. we present a comprehensive evaluation of the state-of-the-art bert and cnn/lstm based offensive language classifiers, as well as an offensive lexicon and google's perspective api, on two datasets. we summarize our key contributions below. ## attack methods this section describes our adversarial attack method as well as a recent visual adversarial attack @xcite and a simpler attack @xcite for baseline comparison. ## results in this section, we first present the results of our proposed adversarial attack approach and then those of existing approaches from prior literature on the olid dataset. evaluation was also performed on the solid dataset and the results followed a similar trend. full results for all attacks are located in the appendix. ## analysis of embeddings as discussed in section 3.1, the adversary's strategy is to make crafty word replacements using a new embedding generated from an evasion collection (here made of deleted tweets not detected by an offense classifier). results show that these embeddings successfully support the adversary at evading offensive language classifiers while maintaining readability and semantics. for further insights, we compare the off-the-shelf pretrained (p re) embedding with the embedding fine-tuned on the evasion collection (f t ). we examine the embeddings using the 59 words as probes which are both in the offensive lexicon @xcite and in the olid test. for each word we get the 20 most similar words from p re and from f t for comparison. fine-tuned embeddings move evasive substitute words closer to offensive probe words. we calculate the average position of the first evasive word foot_7 amongst the 20 most similar words. p re has an average distance of 11, while f t has an average distance of 3. thus, on average, f t is more likely to find evasive replacements. for example, in p re dispicable appears as the 3rd most similar word to despicable, but it is the most similar in f t . since f t could contain some unintelligible words, we repeat the experiment to filter out substitute words used by less than 3 different users. the same overall trend still holds. updated embeddings learn creative replacements. we manually compare the entries in the two lists (f t and p re) of substitute words for each probe word. f t learns creative replacements absent in p re. examples include the word azz being the most similar word to ass in f t , but being absent within the most similar word list for p re. similarly, niggah appears as a replacement for bitch in f t , but not in p re. these examples, along with the previous distance analysis, illustrate the craftiness in our evasion dataset. original tweet f t very similar the amount of fake pages that watches my snap is insane i hate bitches so bad g the amount of phising pages that watches my snap is unbelievable i hate bitchs so bad g @user nigga is a rapist, he deserves all the shitty things that happen to him @user nigha is a peado, he deserves all the awful things that happen to him arab men are some of the most egotistical, disrespectful, pieces of shit you'll ever meet arab men are some of the most egotistical, inconsiderate, pieces of shxt you'will ever meet somewhat similar @user look at his lame ass with that shit eating smile... he is so vile. ## related work we first review related work on robustness of text classification in general and then closely related research on evading offensive language classifiers. evading text classifiers. prior work has explored ways to evade text classification in general. @xcite showed that character-level perturbations such as misspellings and word-level perturbations using off-the-shelf glove embeddings can evade text classifiers. @xcite proposed an approach to create portmanteaus, which could be extended to adversarial texts. @xcite added a sequence of words to any input to evade text classifiers. @xcite proposed a gan to generate adversarial attacks on text classification tasks. @xcite leverage bert to propose solutions for replacement words, @xcite leverage wordnet. in contrast to prior work evading text classifiers, our work includes approaches to leverage embeddings built from a special evasion text collection. robustness of text classifiers. our work is also relevant to prior studies of the robustness of text classifiers to adversarial inputs. rojas-galeano (2017) showed that primitive adversarial attacks (e.g., misspellings) can be detected and countered using edit distance. @xcite evaluated the robustness of self-attentive models in tasks of sentiment analysis, machine translation, and textual entailment. we examine robustness of similar models, however, we fine tune our embeddings to be task specific, while they do not, and we also test on the state-of-the-art offensive language classifiers. evading offensive language classifiers. @xcite examined robustness of hate speech classifiers against adding typos, whitespace, and non-hate words to text. as discussed earlier, prior work has shown that such primitive perturbations can be detected and reversed @xcite . in contrast, we focus on more crafty text perturbations in our work. @xcite surveyed the ways text has been encoded by humans to avoid censorship and explain challenges which automated systems would have to overcome. this work does not propose an automated approach for text perturbation. @xcite proposed viper for visual adversarial attacks. we implemented viper and @xcite as baseline attacks and showed that our approach is more successful overall. overall, our work advances the research in this space by investigating robustness of offensive language classifiers against crafty adversarial attacks. ## conclusion in this paper, we showed that state-of-the-art offensive language classifiers are vulnerable to crafty adversarial attacks. our proposed adversarial attacks that leverage greedy and attention-based word selection and context-aware embeddings for word replacement were able to evade offensive language classifiers while preserving readability and semantics much better than prior simpler adversarial attacks. we report accuracy drops of up to 46 points or 67% against state-of-the-art offensive language classifiers. furthermore, unlike viper and simpler attacks, our proposed attack cannot be easily prevented using pre-processing strategies. the user study showed that our adversarial attack was able to maintain similar readability with only a slight drop in semantic preservation. our work also suggests ways to improve the robustness of offensive language classifiers through adversarial training @xcite @xcite . more specifically, our attack relies on the evasion collection, which contains crafty adversarial examples that evade detection by offensive language classifiers but are flagged based on manual feedback by users or human moderators. thus, offensive language classifiers can be adversarially trained on the latest evasion collection from time to time to improve their robustness to the ever evolving adversarial attacks. in this context it is noteworthy that continuous availability of large-scale manual feedback is quite unique to the problem of offensive language classification, where popular online social media platforms employ thousands of human moderators @xcite .