Perturbations in the Wild: Leveraging Human-Written Text Perturbations for Realistic Adversarial Attack and Defense (2022)

Abstract: We proposes a novel algorithm, ANTHRO, that inductively extracts over 600K human-written text perturbations in the wild and leverages them for realistic adversarial attack. Unlike existing character-based attacks which often deductively hypothesize a set of manipulation strategies, our work is grounded on actual observations from real-world texts. We find that adversarial texts generated by ANTHRO achieve the best trade-off between (1) attack success rate, (2) semantic preservation of the original text, and (3) stealthiness–i.e. indistinguishable from human writings hence harder to be flagged as suspicious. Specifically, our attacks accomplished around 83% and 91% attack success rates on BERT and RoBERTa, respectively. Moreover, it outperformed the TextBugger baseline with an increase of 50% and 40% in terms of semantic preservation and stealthiness when evaluated by both layperson and professional human workers. ANTHRO can further enhance a BERT classifier’s performance in understanding different variations of human-written toxic texts via adversarial training when compared to the Perspective API.

## introduction machine learning (ml) models trained to optimize only the prediction performance are often vulnerable to adversarial attacks @xcite . in the text domain, especially, a character-based adversarial attacker aims to fool a target ml model by generating an adversarial text x * from an original text x by manipulating characters of different words in x, such that some properties of x are preserved @xcite @xcite . we characterize strong and practical adversarial attacks as three criteria: (1) attack performance, as measured by the ability to flip a target model's predictions, (2) semantic preservation, as measured by the ability to preserve the meaning of an original text, and (3) stealthiness, as measured by how unlikely it is detected as machine-manipulation and removed by defense systems or human examiners (figure 1 ). while the first two criteria are natural derivation from adversarial literature @xcite , stealthiness is also important to be a practical attack under a mass-manipulation scenario. in fact, adversarial text generation remains a challenging task under practical settings. previously proposed character-based attacks follow a deductive approach where the researchers hypothesize a set of text manipulation strategies that exploit some vulnerabilities of textual ml models (figure 1 ). although these deductively derived techniques can demonstrate superior attack performance, there is no guarantee that they also perform well with regard to semantic preservation and stealthiness. we first analyze why enforcing these properties are challenging especially for character-based attacks. to preserve the semantic meanings, an attacker can minimize the distance between representative vectors learned from a large pre-trained modele.g., universal sentence encoder @xcite of the two sentences. however, this is only applicable in word-or sentence-based attacks, not in character-based attacks. it is because characterbased manipulated tokens are more prone to become out-of-distribution-e.g., @xmath0, from what is observed in a typical training corpus where the correct use of english is often assumed. in fact, existing character-based attacks such as textbugger @xcite , viper @xcite and deepwordbug @xcite generally assume that the meaning of the original sentence is preserved without further evaluations. in addition, a robust ml pipeline is often equipped to detect and remove potential adversarial perturbations either via automatic software @xcite , trapdoors @xcite or human-in-theloop @xcite . such detection is feasible especially when the perturbed texts are curated using a set of fixed rules that can be easily re-purposed for defense. thus, attackers such as viper and deepwordbug, which map each latin-based character to either non-english accents (e.g., ė, ā, d), or homoglyphs (characters of similar shape), fall into this category and can be easily detected under simple normalization techniques (sec. 4.1). textbugger circumvents this weakness by utilizing a set of more general character-editing strategiese.g., replacing and swapping nearby characters to synthesize human-written typos and misspellings. although texts perturbed by such strategies become less likely to be detected, many of them may distort the meaning of the original text (e.g., "garbage"→"gabrage", "dumb"→"dub") and can be easily flagged as machine-generated by human examiners. therefore, we argue that generating perturbations that both preserve original meanings and are indistinguishable from human-written texts be a critically important yet challenging task. to overcome these challenges, we introduce anthro, a novel algorithm that inductively finds and extracts text perturbations in the wild. as shown in figure 1 , our method relies on humanwritten sentences in the web in their raw form. we then use them to develop a character-based adversarial attack that is not only effective and realistic but is also helpful in training ml models that are more robust against a wide variety of humanwritten perturbations. distinguished from previous research, our work considers both spellings and phonetic features (how a word sounds), to characterize text perturbations. furthermore, we conducted user studies to quantitatively evaluate semantic preservation and stealthiness of adversarial texts. our contributions are as follows. ## a realistic adversarial attack given the above analysis, we now derive our proposed anthro adversarial attack. we first share how to systematically encode the sound-i.e., phonetic feature, of any given words and use it to search for their human-written perturbations that satisfy the sms property. then, we introduce an iterative algorithm that utilizes the extracted perturbations to attack textual ml models. ## evaluation we evaluate anthro by: (1) attack performance, (2) semantic preservation, and (3) humanlikeness-i.e., how likely an attack message is spotted as machine-generated by human examiners. ## anthro β attack anthro β . we examine if perturbations inductively extracted from the wild help improve the deductive textbugger attack. hence, we introduce anthro β , which considers the perturbation candidates from both anthro and textbugger in ln. 10 of alg. 1. alg. 1 still selects the perturbation that best flip the target model's prediction. attack performance. even though anthro comes second after textbugger when attacking bert model, ## defend anthro, anthro β attack we suggest two countermeasures against anthro attack. they are (i) sound-invariant model (soundcnn): when the defender do not have access to {h} k 0 learned by the attacker, the defender trains a generic model that encodes not the spellings but the phonetic features of a text for prediction. here we train a cnn model @xcite on top of a embeddings layer for discrete soundex++ encodings of each token in a sentence; (ii) adversarial training (adv.train): to overcome the lack of access to {h} k 0 , the defender extracts his/her perturbations in the wild from a separate corpus d * where d * ∩@xmath1.e., via self-attack with ratio 1:1, to fine-tune a more robust bert model. we use d * as a corpus of 34m general comments from online news. we compare the two defenses against bert and bert combined with 3 layers of normalization a+h+p. bert is selected as it is better than roberta at defending against anthro ( results. ## discussion and analysis evaluation with perspective api. we evaluate if anthro and anthro β can successfully attack the popular perspective api foot_1 , which has been task sentiment analysis categorization anthro 0.80 0.93 anthro β 0.86 1.00 generalization beyond offensive texts. although anthro extracts perturbations from abusive data, the majority of them are non-abusive texts. thus, anthro learns perturbations for non-abusive english words-e.g., hilarious->hi-larious, shot->sht. we also make no assumption on the task domains that anthro can attack. evidently, anthro and anthro β achieves 80%, 86% atk% and 90%, 100% atk% on fooling the sentiment analysis and text categorization api from google cloud ( limitation of misspelling correctors. similar to other spell-checkers such as pyspellchecker and symspell, the sota neuspell depends on a fixed dictionary of common misspellings, or synthetic misspellings generated by random permutation of characters @xcite . these checkers often assume perturbations are within an editdistance threshold from the original words. this makes them exclusive since one can easily generate new perturbations by repeating a specific character-e.g., "porn"→"pooorn". also, due to the iterative attack mechanism (alg. 1) where each token in a sentence is replaced by many candidates until the correct label's prediction probability drops, anthro only needs a single good perturbation that is not detected by neuspell for a successful replacement. thus, by formulating perturbations by not only their spellings but also their sounds, anthro is able to mine perturbations that can circumvent neuspell. ## conclusion we propose anthro, a character-based attack algorithm that extracts human-written perturbations in the wild and then utilizes them for adversarial text generation. our approach yields the best tradeoff between attack performance, semantic preservation and stealthiness under both empirical experiments and human studies. a bert classifier trained with examples augmented by anthro can also better understand human-written texts.