How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks (2023)

Abstract: Natural Language Processing (NLP) models based on Machine Learning (ML) are susceptible to adversarial attacks – malicious algorithms that imperceptibly modify input text to force models into making incorrect predictions. However, evaluations of these attacks ignore the property of imperceptibility or study it under limited settings. This entails that adversarial perturbations would not pass any human quality gate and do not represent real threats to human-checked NLP systems. To bypass this limitation and enable proper assessment (and later, improvement) of NLP model robustness, we have surveyed 378 human participants about the perceptibility of text adversarial examples produced by state-of-the-art methods. Our results underline that existing text attacks are impractical in real-world scenarios where humans are involved. This contrasts with previous smaller-scale human studies, which reported overly optimistic conclusions regarding attack success. Through our work, we hope to position human perceptibility as a first-class success criterion for text attacks, and provide guidance for research to build effective attack algorithms and, in turn, design appropriate defence mechanisms.

## introduction like many other machine learning models, natural language processing (nlp) models are susceptible to adversarial attacks. in nlp, these attacks aim to cause failures (e.g. incorrect decisions) in the model by slightly perturbing the input text in such a way that its original meaning is preserved. research has reported on the potential of adversarial attacks to affect real-world models interacting with human users, such as google's perspective and facebook's fasttext @xcite ) more generally, these attacks cover various learning tasks including classification and seq2seq (fake news @xcite , toxic content @xcite , spam messages @xcite ), style transfer @xcite and machine translation @xcite ). it is critical to properly assess model robustness against adversarial attacks to design relevant defence mechanisms. this is why research has investigated different attack algorithms based on paraphrasing @xcite , character-level @xcite and word-level @xcite perturbations, and made these algorithms available in standardized libraries @xcite . for the many nlp systems that interact with humans, we argue that effective adversarial attacks should produce text that is both valid and natural. validity refers to the property that humans perceive the same semantic properties of interest foot_0 for an adversarial text as for the original text it was produced from. naturalness refers to the perception that an adversarial text was produced by humans. adversarial texts that are invalid and/or unnatural can still cause failed nlp model decisions, however, their ultimate effect on humans is negligible because they would fail to convey the intended meaning (e.g. hate speech that is not perceived as hateful) or they would be suspected to be computer-generated (e.g., a phishing email using awkward vocabulary and grammar). unfortunately, the scientific literature on adversarial text attacks has neglected (and sometimes ignored) the inclusion of human perception as an essential evaluation criterion -see the absence of systematic analysis of adversarial texts as perceived by humans risks leading to overestimation of their semantic quality and, in turn, to fallacious model robustness assessment and misguidance during the design of defences. this was hinted in the seminal work from @xcite , where a 10-participant survey on one dataset and two attacks revealed a discrepancy between the human-perceived naturalness of adversarial examples. therefore, in this paper, we present the first extensive study that evaluates the human-perceived validity and naturalness of adversarial texts. we surveyed 378 participants in assessing, based on five criteria, over 3000 texts (original and adversarial) coming from three datasets and produced by nine state-of-the-art attacks. our investigations first reveal that the participants would classify 28.14% of adversarial examples into a different class than the original example. this means that the adversarial perturbations change human understanding of the modified text and, thus, fail to achieve their purpose. irrespective of the classification task, participants detect 60.3% of adversarial examples as computer-altered; they can even identify 52.38% of the exact altered word. these findings contrast the overly optimistic conclusions regarding attack success rates from previous small-scale human studies. our results underline that existing attacks are not effective in real-world scenarios where humans interact with nlp systems. through our work, we hope to position human perception as a first-class success criterion for text attacks, and provide guidance for research to build effective attack algorithms and, in turn, design appropriate defence mechanisms. ## motivation consider the example of fake news shown in figure 1b . ("original"). @xcite have shown that this example is detected by existing fake news detectors based on nlp machine learning models. however, the same authors have also revealed that, if one changes specific words to produce a new sentence ("adversarial"), the same detector would fail to recognize the modified sentence as fake news. this means that fake news could ultimately reach human eyes and propagate. fortunately, fake news -like hate speech, spam, phishing, and many other malicious text contents -ultimately targets human eyes and has not only to bypass automated quality gates (such as detectors) but also fool human understanding and judgment. indeed, to achieve their goal of propagating erroneous information, adversarial fake news should still relay wrong information -they should be "valid" fake news -and be perceived as a text seemingly written by humans -i.e. they should be "natural". the fake news example from figure 1 is unnatural because it uses irrelevant proper nouns like "slut tower" or "donald hobo" that do not exist in reality, and this makes the fake news ineffective. we, therefore, argue that invalid and/or unnatural examples do not constitute relevant threats. thus, the goal of adversarial text attacks becomes to produce examples that change model decision and are perceived by humans as valid and natural. our study aims to assess, using human evaluators, whether state-of-the-art text adversarial attacks meet this goal. the answer to this question remains unknown today because, as revealed by our survey of existing attacks (see for our study, the validity and naturalness requirements led us to consider word-based attacks. indeed, character-based attacks are easily attack name/paper type evaluation participants attacks studied validity naturalness s. d. g. m. hotflip (ebrahimi et al., 2018) word based x x x x 3 1 alzantot(alzantot et al., 2018) x x x x 20 1 input-reduction(feng et al., 2018) x x x x n/a 1 kuleshov(kuleshov et al., 2018) x x x x 5 1 bae(garg and ramakrishnan, 2020) x x 3 2 pwws(ren et al., 2019) x x x 6 1 textfooler (jin et al., 2020) x x 2 1 bert-attack(li et al., 2020) x x x 3 1 clare (li et al., 2021) x x x x 5 2 pso (zang et al., 2020) x x x 3 1 fast-alzantot (jia et al., 2019) x x x x x 0 0 iga (wang et al., 2019) x x x x x 0 0 textbugger (li et al., 2019) character based x x x 297 1 pruthi (pruthi et al., 2019) x x x x n/a 1 deepwordbug (gao et al., 2018) x x x x x 0 0 morris et al. (2020a) independent x x 10 2 our study 378 9 ## study design 4.1 adversarial texts to generate the adversarial texts presented to participants, we used the textattack library @xcite , which is regularly kept up to date with state-of-the-art attacks, including word-based ones. ## misc. results we conducted an analysis to check whether human perception of naturalness and validity is related to their language proficiency. we found out that language proficiency only affects some aspects of naturalness and not validity results. people with professional proficiency are more suspicious, they achieve a higher accuracy at detecting adversarial text compared to the other two groups(64.6% vs 54.8% and 57.0%). regarding grammaticality, people with higher proficiency level report more added errors to the original examples by adversarial attacks. lastly, for the meaning preservation there is a statistical difference only between two proficiencies, natives give a lower score compared to limited working proficiency. for detailed results, refer to ## discussion and conclusion our study unveils that a significant portion of adversarial examples produced by state-of-the-art text attacks would not pass human quality gates. these examples are either invalid (labelled differently from intended) or unnatural (perceived as computer altered). this means that the practical success rate of these attacks in systems interacting with humans would be lower than reported in purely model-focused evaluations. through our investigations, we discovered that validity is related to the meaning preservation of the original text by adversarial perturbations. as for naturalness, it appears that the detectability of (at least one) altered words, as well as meaning clarity are strong factors determining the suspiciousness of a text to have been computer-altered. the (perceived) presence of grammar errors is not a relevant criterion to determine naturalness. however, grammaticality may still make sense in contexts where exchanged texts rarely contain grammar mistakes (e.g. in professional or formal environments). more generally, the relevant criteria to evaluate the quality of adversarial examples depend on the considered use case and threat model. our goal, therefore, is not to qualify an existing attack as "worse than claimed", but rather to raise awareness that different threat scenarios may require different evaluation criteria. we, therefore, encourage researchers in adversarial attacks to precisely specify which systems and assumptions their study targets, and to justify the choice of evaluation criteria accordingly. in particular, we corroborate previous studies that discourage the use of automated checks to replace human validation @xcite . our study has revealed that human perception of grammaticality does not match the results of grammarchecking tools. we thus argue that humans play an essential role in the evaluation of adversarial text attacks unless these attacks target specific systems that do not involve or impact humans at all. interestingly, none of the existing attacks dominate on all criteria. a careful observation of tables 2 and 3 reveals that six attacks (over nine) lie on the pareto front (considering our evaluation criteria as objectives). this implies that different attacks fit better in different threat models. ultimately, we believe that our results shape relevant directions for future research on designing adversarial text. these directions include further understanding the human factors that impact the (im)perceptibility of adversarial examples, and the elaboration of new attacks optimizing these factors (in addition to model failure). the design of relevant attacks constitutes a critical step towards safer nlp models, because understanding systems' security threats paves the way for building appropriate defence mechanisms.